Just outside of Geoje city, you can find Oksangeum-seong Fortress. This is possible as a long hike from Geoje or from the Geojeyeon Gwona.
Otherwise, if you have your own transportation, there are a few other small fortresses on the island. While they may not be the most spectacular remains, the fresh air hikes to get there, the panoramic views they offer and the colorful locals you meet on the way are worth the effort.
There are also several smoke signal stations around the coast.
Pick up a tourist map and explore.
At Koje don't forget the inlands. A busy spot are the Mundong Waterfalls. You can also look for the Nature Art Park (Bonzai trees and bigger stuff) or explore the Oryang fortress remains. This last one is hard to find (just keep on drivin') but has a nice view over the West-side of the island.
This location is just behind the Haegeumgang Theme Museum on the southern end of Geoje, near Sinseondae.
If you have your own transportation, it is a nice short walk down a path to the rocky beach and cliffs. When you get down there, you can walk along the pebble beach or climb the boulders along the cliffs. Many locals will be here fishing from the rocky shores.
The area around the 2 entrance bridges to the island is the one with most small local businesses. It's fun to drive around and discover the miscellaneous activities.