Not JUST a Navy town...
"This city is what you make of it"
I spent 3 months living in Yokosuka while working at a small US Navy housing complex in Yokohoma. I've read a lot of negative things about Yokosuka on this site and unfortunately some of it is true. The truth of it is however, 1% of the sailors can ruin the experience for everyone visiting Yokosuka. If you can somehow schedule your trip when there isn't a Navy carrier in port I think you will see that Yokosuka is a very different city.
"Located close to some must-see spots"
There might not be much to do and see in Yokosuka, but try hopping on a train and visit the following destinations:
Yokohama - about 30 min train ride from Yokosuka-Chuo station on the Keikyu Line.
Tokyo - depending on where your final destination is time will vary, but expect roughly a 1.5-2 hour train ride. Best way to get to Tokyo is to use the Keikyu Line all the way to Shinagawa and transfer from there.
I took a bus trip out to Mt. Fuji which took roughly 4 hours. I'm not sure the best route to take by train, but it's not too terribly far away.