To get to Floresta da Tijuca, take the metro to Saens Pena and then take a bus going to Barra da Tijuca. It's best to let the driver know that you wish to get off at Tijuca. I said "Tijuca" to some of the passengers and they let me know when to get off.
I didn't take this as I chose to walk up, but there are jeeps that take you up to the top of the forest mountain and around the small twisty roads. They leave from the car park near the entrance.