My take is direct gives you the same horizontal and vertical tolerances
that an airway does. Which I think, is 100' of altitude variation
(actually they don't usually see it, unless it is 150' or more), and 2
miles to the right or 2 miles to the left.

However it ATC sees you are veering off course, the controller can ask,
as a way of getting your attention to the deviation (even though it may
still be withing tolerances). He sees you are veering off course and
wants you to correct. But that doesn't mean you have busted your
clearance.

Take ATC statements for what they say. He wants you to verify that you
are direct.

Also, don't get in the habit of hitting the direct to button, you
should actually fly back to your course and get back on your original
direct track, not keep making new direct to tracks.

If you do a lot of IFR flying, get the Howie Keefe (www.aircharts.com),
text updates. You can use these to see if any of the waypoints have
changed if you update your GPS when the Keefe system begins (March I
think). This way you will know if your waypoints are up to date, even
though the database may not be.