1.A perfect halftone / index test pattern with % points and gradients, fine lines, text etc... various angles / LPI / DPI FM patterns etc... This is the starting point to measure the following "Dot-Shift Quality Control" --- just as you asked this first point is from the Digitally generated RIP, to the FILM. What changed or "shifted" from the RIP to the Film version?
2. Next "QC" point to be measured is what shifts from the FILM to the "PREPARED SCREEN". (before you put the ink in and it's on press.. there can be issues in the way it's been cured from the film. In fact this is one of the worst areas for small shops who are sold the low-end exposure units that use diffuse lighting that cannot hold halftones and details, the cure times are too long, etc.) -- Screen mesh used and emulsion, film dot density, exposure type and time, all this will affect what is transferred from the FILM to the SCREEN.
3. The final "QC" point for the dot-gain/shift/loss... is what changes from the "PREPARED SCREEN" to the "PRINT" with the ink, pressures, etc all dialed in accordingly... Squeegees, screen tension, ink variables like viscosity and opacity, hue/sat/brightness of the result etc... this all plays into certain aspects of the print... but for the most part just holding the right halftones on each screen is key to getting the mixes to work out reliably.
Measuring those 3 points and seeing what you end up with, you can go backwards and create your overall custom compensation curves with any amount of clipping, pushing, pulling in whatever regions you need to try and do what is the only job of a "color separator" which is to do the pre-press technician's role in most offset shops. Account for the press variables using the simulations of your ink-mix mode.
It's just that in screenprint yes we have a lot of other complex variables like substrate colors and ink opacities and whites etc... but this is all math.
Art is what happens built up as an RGB-Alpha document over your substrate color (should not really be created in the art) - but you can have your alpha fade any colors to your shirt colors and have them based or not, mixed with the shirt color or not, etc.
But this should all be in full RGB gamuts with alpha transparency over chosen shirt colors so you can easily mock up on various products. But even if a shirt color is stuck in a document there are easy ways to pull it back to trans.
Art is the creative process that no machine can ever do, of course.
Printing is the attempt to make exact copies of that original design or pattern.
Separation? It's a mathematical process based on your input/output variables. It has already been completely automated in the world of digital off-set printing. Take a vacation to Xerox PARC, they are lightyears beyond screenprinting's technology.
http://en.wikipedia.org/wiki/PARC_(company)http://www.xerox.com/digital-printing/digital-printing-press/color-printing/xerox-igen4/spec-enus.html#yscroll_300