Right at this location the first Europeans settled themselves in Australia. Therefore you will see a lot of old and historical houses, churches, etc. Also the oldest building in Sydney, Cadman's Cottage (1816) is located in The Rocks. It's also the entrance to the Sydney Harbour Bridge. You can find this at Dawes Point Park, which is the first fortification of Australia. The Rocks is now a leading visitor destination that combines the best of the past and the present. It's a must see.
Unique Suggestions: Within 4 years of Sydney Cove becoming the forthest outpost of the British Empire in 1788, rows of convict-built dwelling lined the western slopes of the Cove. The first dwellings were built along natural rock ledges and were described as being "on the rocks"; giving rise to The Rocks name.
Fun Alternatives: For years it used to be the home of prostitution, gambling and street fights. You won't find anything like like this these days in The Rocks. Nowadays it's a very touristic neighbourhood with nice shops in a very historical surrounding. Today, the area bears testimony to its chequered history, with responsibility for the preservation of The Rocks' heritage and character a top priority for SHFA.
Take a train to Circular Quay and walk along George Street towards the Bridge.
The Rocks is an incredibly expensive section of Sydney (FYI, the priciest city in Australia) that specializes in haute and faux couture art galleries, overpriced souvenier shops, the usual run of McRestaurants and, on Saturday and Sunday, a market. From what I could tell, there is nothing here that can't be had, better, cheaper, and easier in other parts of the city. You can look around, but its boring.
Unique Suggestions: Proceed up the steps to the Glemore Hotel and have a beer on the rooftop, soak up the view and some suds and have a laugh with the friendly folks who seem to frequent it.
Fun Alternatives: The entry to the bridge isn't far from the Glenmore, you could walk along that, see some of my travelogues for the view.