PATH (Port Authority Trans-Hudson) operates trains across the river to the state of Jersey. They operate out of 33rd Street station near Penn Station, and from the World Trade Center. It makes staying out of town on the cheaper Jersey side of the Hudson feasible as it only takes 10-15 minutes to travel from somewhere like Hoboken or Newport.
The trains are reasonably modern, but the tracks must be from the 18th century as the carriages try to shake your limbs from your body, especially on the New York City side of the river. But they are often so crammed with people you are unlikely to fall over. The prices are reasonable, and the ticket system dovetails neatly into the New York City metro system.
If like me you happen to be staying in Jersey city it is only a 15min ride into town on the Path train so perhaps something to think about when you are arranging accomodation as Manhatten prices can be sky high, the taxes in New Jersey are also slightly less.
The train line runs from Newark Penn station 2 stops to Journal square where you have the option of a direct route to either WTC or else to 9th, 14th, 23, 33rd street @ 6th Avenue.
The cost is only $1.75 each way but if you buy a smart card (possibly online) you can reduce this to $1.30 each trip. I was advised at night if you are travelling to Jounal Square its preferable to go to Grove street & get a $7 cab ride as the side entrance closes and outside the station is not well lit or served by too many cabs.
During the weekend trains run direct to Hoboken & then onto 9th, 14th, 23 & 33rd street. The train runs fairly regularly up until around 11.30 then it runs every 20 / 30 min. You can use your metro card to travel on the path but you can't use a subway unlimited ride card the subway & Path are not connected