Check out the penal colony ruins on Sarah Island
I put this in the 'off the beaten path' category because it is only accessible by taking one of the cruises that stop here or some other rented boat... See my Things to Do tip for more info on the cruise that brought us here. I wanted to go here just because the island shares my name. :) It proved to be an island with an interesting history.
The island was a penal colony from 1822 to 1833 where prisoners were sent from other jails when they committed additional crimes. They even had another smaller island right next to Sarah Island called Grummet Island where the worst of the worst were sent. About 1200 prisoners were sent there over the eleven years it operated. They were supervised by military personnel. The island became an industrial village over the years. In the end - there were even ships being built there.
There were still some remains there, although most of the buildings were built of wood and are completely gone. There is still ruins of the last living quarters built, the bread oven, the solitary confinement building, and a couple of other things.
At the end, the tour guide even put on her wet suit and jumped out in the water to show us where the boat ramps (used when building the ships) were still several feet under water.