It depends.

They tell you who to call by phone when they approve you to switch
frequencies.

If the last facility you were talking to was a civilian TRACON, they
usually want you to call them. If it was military or an ARTCC, they
usually want you to call FSS.

.... but that can change, even at a given airport.

If I were on the ground and wasn't sure who call, I'd call FSS first.
They'd either cancel IFR for me or tell me who to call.