It's my experience that you are not going to get an equal comparison. You may be able to get the dot down to equal size but you'd have to use two different line screens for the same job...or a higher line count than you are using on your imagesttter.
As an example, A 3% dot at 65line screen may be similar to the micron count (size) of a 3% dot in an 80 line screen on a Epson printer. You have to use a high line screen because the epson printers dot is not going to be as well formed...(smooth clean edges) without residual spray/splatter around it...and The epson density of the dot is not as dark. So smaller dots (from an epson) are not as good as from a true imagesetter. They are two difference transfer processes.
As far as your issue with the epson being set up to clip at 5% There must be something set up in it's RIP to do this. In Photoshop, you can assign a transfer function setting so that whenever you print films from photoshop, it clips at whatever you assigned it to do, such as 5%. That is inside Photoshops print settings...but most people are not printing straight out of Photoshop these days due to Photshop ( think version CS5) taking out the option to print halftones in newer version.
This sounds similar to your topic but has to be done in your RIP somewhere. ( a setting) of some sort. Look around inside. I'm not familiar with where that is in your RIP software. Hopefully someone who knows can chime in.
Keep in mind, that when you do get the dots to be of a similar size) you also need to be concerned with the dots actual shape, dot density and the level of residual splatter around it that may play a small part in your outcome.
