I've noticed it as well. I attribute it to (not necessarily in order):

1) higher fuel prices;

2) most GA pilots fly VFR and don't use ATC and are counfused about the
TFRs all over the place that can get them busted;

3) purchase, maintenance, rental, and instruction costs have outpaced
growth in discretionary income (I don't have enough data to back this
up);

4) people now work longer hours and so have less free time (e.g., my
job used to permit me to skip a few hours during the day to fly - I can
no longer reasonably expect to do that so I put off flying until
weekends; however, I have other commitments on weekends because of
family and so flying loses); and

5) the post-9/11 Homeland Security rules make it less likely that
immigrants will get flight training.