While this looks like the ruins of an old building, it is actually a natural formation. Borgarbrík, or "Castle Rock," is actually an example of a volcanic dike. An impressive group of basalt columns along with a spectacular backdrop.
Hiking around the southern part of the island is a delight. There are three marked trails: green, yellow, and red. The "red" trail is the longest and most scenic. It will take you around the perimeter of the accessible part of the island (the northern part is privately owned and off-limits to tourists). I'll let the photos speak for themselves as for the views you get.
Along the hiking trail, you can see the remnants of an ancient farm. While no record exists of when this farm was actually active, a record from 1712 indicates the farm had been abandoned "a long time ago..." Obviously, erosion has taken its toll over the centuries, and what's left of the foundation is partially submerged. It's still amazing to take a look at, though.
The Hrísey Church is quite interesting to look at from the outside. Enjoy the wooden structure along with its striking backdrop. Unfortunately, the church was locked when we visited. Perhaps you'll have better luck and be able to view the interior. Given the fine weather, we didn't mind all that much staying outside...
The best purpose, why to visit Hrisey is watching birds. Birds on this island are protected, so the are not afraid of people. You can see a moorfowl, whitch is popular like a Christmas dinner, a longbill or a sea swallow.
And it is very easy to get to Hrisey, because ferry leaves Arskogssandur's harbour every hour.