Walk around the beach in the morning, and you will be approached by local fishermen to see if you want to hire their boat for a fishing trip. I didn't try so I can't really comment. A century ago, the Sea of Cortez used to be fishing paradise full of Totoaba. But that's no more (see my "off the beaten path" tip for Totoaba details).
Surrounded by desert, San Felipe is a popular spot for off-road sports. Around town you'll see many dune buggies for rent. Every year it hosts San Felipe 250 race. Also the famous Baja 500 occasionally runs their course through town.
Off road racing with buggys, motorcycles, trucks etc is very popular in San Felipe. If you are active in this sport yourself or do it as a hobby, you might wanna come to San Felipe one day to either take part or watch popular races like the Baja1000 or Baja 250 who have San Felipe on their course map. The Baja Score 250 race starts and ends in San Felipe 10-12 March.