At the aiport I fly out of (KPAO) after I readback my clearance to
ground they always tell me to "advise the tower you're IFR". So there's
no question about this issue there. I'm not sure if this has to do with
how busy our airport always is, or how busy the departure airspace is.
I'd guess both. I'll make a not to ask the controller next time I visit
the tower.

Max