I thought I had to follow what the tower says in the clearance,
until handed off to Departure...

You should be skeptical about following instructions from a non-radar
tower in poor visibility conditions. Tower guys have told me that
they're not going to issue instructions contrary to the DP unless good
visibility conditions prevail or the pilot assures them that he can
maintain his own obstacle clearance.