I agree they had you confused with an IFR clearance.

In Class B airspace, when you are VFR, you only have to say "unable
VFR in that direction/altitude" to any assigned altitudes or headings,
at which point the controller has a problem. You can suggest a
heading or altitude that would work to maintain VFR, or the controller
may even ask you what would work. The controller doesn't know the
weather details where you are. It is still your responsibility to
maintain VFR although approach should keep you separated from IFR
traffic.

For that matter, when enroute with flight following center should
maybe only suggest altitudes. You should announce any altitude
deviations to the controller. Again the controller doesn't know your
weather. You don't have to fly your VFR flight plan altitude.

Part of the problem is possibly the "plan your flight & fly your plan"
jingo that was taught years ago. It sounds nice in a classroom, but
in reality, you have to deal safely with local weather and other
detail factors that you can't possibly anticipate when you are flight
planning. It sounds like you did.

Keep the eyeballs outside. Comments anyone?