As a member of the tourist industry I couldn't help but reply to this thread. I have lived here over 12 years, walked the streets at all times and never had a problem. I agree, like any city there is crime but based on my past residence of Philadelphia & Baltimore both of which I was mugged in, Annapolis is like Mayberry. Don't wander off the main streets and pay attention to your surroundings and you'll be fine. Parking is plentiful and if your doing a day trip you should use a garage. If your staying, most accommodations have parking on-site. Pick your hotel wisely so you don't have to move your car. A lot of hotels claim to be in the 'historic district' but they're not so do your research. The hardest thing to navigate when driving are the circles and pedestrians. Other than that its pretty easy.
Here's a link for info about parking at the US Naval Academy Stadium ($5 per day) and using free shuttle to get downtown. This might be easier than finding the downtown parking garages and hoping for space. If you don't want to take the shuttle, the Stadium is also an easy and pretty pleasant walk across the Rowe Blvd Bridge to downtown . . . less than a mile to the statehouse. Some traffic noise walking along Rowe Blvd but the views from bridge are nice.
Before working at the courthouse in the downtown area (where the statehouse is), I did not realize it is a place you need to keep your guard up. It's a very charming area, and the local historical society strives to keep it that way. However, this area cannot escape modern day crime. For example, not long ago, one of the prosecutors from the courthouse was held up at gunpoint at a parking garage. Also, I learned from co-workers that despite being right next to the courthouse (therefore lots of police around), drug dealers hang out in the area, too. Just keep your guard up when you round a corner that is not well-traveled and full of people.
Not a danger unless you're an impatient driver ... but be warned that there's plenty of traffic here. Park in the garage and then walk. You can see more when you walk anyways!