Arugam Bay, about 7 kms from Pottuvil, is a really good surfing spot. Still not crowded because it's quiet far from everything, but defenitly a nice chilled spot with lots of cool people around. And the locals are very good surfers too. Once in Arugam Bay, ask around, there is plenty of nice spots accessible either by walk, and some a bit more far but easy to go to with a tuktuk, watch for the elephants that are running wild in the region.
Lots of small to medium guesthouses ranging from really cheap and crappy to surprisingly nice for the middle of nowhere ! A place for all size of wallet and taste.
A walk to crocodile rock is a must. Along the way you will see a beautful landscape, lots of sea shells and interesting sand formations on the beach. The walk takes about 1 hour and it is advised you leave the rock before it gets dark.
You will have to change bus in Badulla (also if coming from Matara in the south), and probably stay there overnight, then go until Potuvil and take from the bus terminal a tricycle (2-3 km).