I'd try an impedance matching transformer. Something like a Tamura
MET-17 (available from Digikey for about $20 each and other places).
If they do what you want, put heat shrink tubing around them and call
it a done deal.

Does your stereo or intercomm have a way of controlling the pre-amp
output volume of the music? You might be a little "hot" compared your
comm radio.