If you arrive to the island early, you'll have to wait till 10:30 when most of the museums open. A good place (I think the only one) to have breakfast then is the Hostellerie du Chevallier de Boufflers, the red house at the right as you go down the ferry.
There you get a huge breakfast with tea/coffee and as much bread as you can eat. The decoration is very colonial, with old B/W pics on the walls.
The food at the restaurant was not bad. They had an assortment of fish, chicken and beef that were all prepared in the local fashion... simple but decent. I will say one thing... they jack up the prices more than at usual tourist spots!!! A bottle of water cost almost 6 dollars USD!!! So remember to bring it with you!!!
By the beach, right where the ferry is, you will find many terraces serving fresh fish and other local specialities.