What about those of us flying under Special Issue Class III? Can I do
sport pilot if I just let it lapse?

I've never actually had a denial: address the problem up front and got
my special issue before meeting the examiner.