I don't think you're expecting too much as you're doing what needs to be done, it seems. When we've printed on spandex before, and used a stretch additive, it's worked well, plenty of stretch. I'd speak with the manufacturer of your additive to see if they can give a suggestion. Though I don't recall doing overlay colors, I would think they will need the additive as well.
Steve