I fly a Mooney so speed restrictions are not uncommon when approaching
busy GA fields.
Strangely about a month ago I was coming into SMO VFR and about the
time I crossed the fence tower said "93V, you're too fast".

Just a thought, but was it the same voice you heard while talking to the tower?

I was thinking it could have been an onlooker (like mxsmanic) standing by the
fence with a handheld.

Could it be that the tower got a speed from a radar sweep that was taken just
before you slowed down?
--
Jim in NC