Steven,
FAR 91.155(c) explicitly refers to "controlled airspace" i.e. airspace
classified as E,D,C,B, and A. Since it is possible to be in E airspace
without being near an airport, it is possible to operate under Special
VFR as long as the requirements in FAR 91.157 are met. Moreover,
ground based weather reporting is NOT required for Special VFR and FAR
91.157 (c)(2) explains what to do if "..ground visibility is not
reported..".