If you use the VOR for IFR navigation, 91.205(d)(2) makes 91.171 apply
to you.

.... but (contrary to popular belief) there is no requirement under Part
91 (or Part 135) that says you have to have a VOR receiver to fly IFR.

So, assuming you don't use it, you're right.