There is not much to do in Les Baux except eat lunch in the 'lower town' and tour the castel ruins. If you are the type of person that likes history, wepons, and a good view, the castel tour will take quite a few hours from your afternoon. Even though the audio tour is kind of boring it is necessary as not a lot of detail is written out. Once you are in the gates it is not hard to find your way around so I won't bother talking about it here, but I do recommend you pack a lunch and eat it in the ruins. The views are better and it is a lot cheaper then the resturants in the lower town.
The climb into the town is strictly uphill. On this particular day it was rather hot so the walk from the parking area into town was quite exhausting. Once in town the streets are rather narrow with uneven stones. Be careful where you walk and pay attention. It would be very easy to take a tumble onto the very hard stone streets.