I think it's unfortunate that you were not able to tour the Governor's Mansion. I attended a reception there once just before Christmas (George Allen was in office) and it was beautiful inside and out. Part of the problem may be that it -is- continuously occupied. As you may be able to see from the poster's photo, it is fairly small as mansions go. The family actually has their living quarters upstairs.
Unique Suggestions Maybe someday the governor will have other accommodations and the mansion will become more of a museum. Til then, calling ahead may be the best option. You wouldn't want to walk in on the governor in his jammies!
Fun Alternatives A good alternative (and you wouldn't even need to move your car) is a tour of the state capitol right there on the same grounds. Designed by Mr. Jefferson with a beautiful freestanding bell tower (the Lt. Governor's office), it is a lovely place, and during good weather, a great place to bring a picnic lunch.
I wanted to visit the oldest continuously occupied executive mansion in the United States of America that was the Governor's Mansion in Richmond where the Governor of Virginia with his family lives now. But it was impossible. Tours were available only at some days.
Hmm... I found it rather unique in the USA where public offices usually stay wide open for visitors. Why does it work that way there? Later on, during my visits to a few governor's mansions and state capitols I could easy meet state representatives and other state VIPs. I was talking a few minutes with one of them in Little Rock, Arkansas.
Unique Suggestions Call +1 (804) 371-2642 and ask for tours in advance. Look at the entrance and the fountain in front of the Mansion on my picture.
Fun Alternatives Wait till Virginians choose next governor :-). When?