No SID in clearance, fly it anyway?

Had a strange thing happen to me today. I filed IFR HPN - HFD (White
Plains NY to Hartford CT). The route part of my clearance came back
"radar vectors Carmel[VOR], V1, Hartford[VOR], Direct".

Usually out of HPN I would expect to get the Westchester-1 departure,
but it wasn't in my clearance, and I when I read back the clearance
without it, I got "readback correct". Seemed a little strange, but I
went with the flow (I should have asked for clarification, I'm not
sure why I didn't). The turn in the SID took us away from our first
fix, so I guess I figured they were just doing us a favor or something
with a shorter routing.

We took off and I started climbing straight out. The HPN-1 has an
almost 180-degree turn almost immediately. The guy I was with asked
me why I wasn't making the turn and I said we weren't on the SID. I
asked tower and they said I should be on the SID.

So, what went wrong? Did I goof? Am I supposed to fly the SID (not a
DP) even though it's not in the clearance? Is it more likely that it
was in the clearance but I just didn't hear it and the controller
didn't notice that it wasn't in the readback?