Staying in Koh Samet is wonderful and the small island surrounds their is like a heaven too. Koh Kudi is about 30 mins from Koh Samet. You can take a day tour with local fisherman. It's great!!
Ao Cho pier is pretty quiet. Some of the local youths will take turns to dive off the pier into the clear blue water. Try it!
Ko Samet is very peaceful with not a lot going on. Just lie back and relax if you can stand the sun torching your skin!
Ao Phai is (unless you go on the weekends when all the BKK people go there) a place with not a lot of people. You can easily find yourself 100 m to yourself...
While the western coast is mainly rocky, the eastern coast has many beaches, separated by rocky promontoirs. There is a track running from north to south along the eastern coast.