Favorite thing: There is no ATM in Paje so you need to take all the money you need with you otherwise you'll be taking a long trip back to Stone Town to get more out. I don't believe there are any any the neighbouring villages either (Jambiani or Bweju) but I'm not 100%... Better to be prepared though.
It gets quite windy in the late afternoon and evening so it would be wise to take a cardigan or some long sleeve tops with you as the wind can be quite chilly. Usually the wind starts to pick up when the tide begins to come in whenever that may be.
It also means that the cool breeze decieves you and you could be left red faced at the end of the day...
A lot of places close during the rainy season, April-May. We arrived at the end of May and only Cristal Resort was open in Paje. Many places will start to open their doors in the first week of June. This goes for restaurants too. On the positive side, it was absolutely deserted there and would not have been the same if it had been swarming with people.
This is the same for Stone Town, when we tried to book a room for a couple of nights, none of the hotels we wanted were not open yet. It's worth calling the places before you book if you are visiting during off seasons. Africa Travel Resource is really helpful and directed us to a hotel that was open.