Reamed out by Approach

How about this one....
I'm headed direct from Lawrenceville, GA, to Rockford, IL, get clost to
RFD and amend destination to Adams, WI, which is about 335 degrees from RFD.
I get "Direct RFD, then Direct Adams"
2 minutes later I get Direct JVL, Janesville, WI (SE of Madison) then Direct
Adams
So I turn about 20 degrees east and head towards JVL
2 minutes later I get Direct current position to Adams
I thank them and turn about 30 degrees left and head for Adams
Now I'm perfectly lined up with Adams with Madison dead in my sites.
The controller asks me if I"m going to pass east or west of Madison.
I say, neither, I'll be right on top of them.
(he's wondering which freq to hand me off to)
He calls again, N6205Y will you be east or west of Madison?
I say Madison is directly enroute from my present position
Silence.
Then he says N6205Y are you on this frequency? How do you read? will you be
east or west of Madison?
I turn 10 degrees left and call back "N6205Y will now be passing west of
Madison"
He turns me over to MSN West approach.

I don't know why he couldn't have turned me over to MSN East ( I was on the
south east side of MSN) and let them worry about me.

Jim