Plane is the only way! There is a very handy train from the Airport (A$9) right into the city or there are buses (the yellow ones) to take you into the city for $A9 and there are a number of specialised buses to take you direct to hotels.
Walking will take you far in the city but the much maligned train system is efficient and mostly reliable. These will take you all around the Sydney and surrounds. you can buy day tripper tickets.
Also buses in the city and inner suburbs eg Glebe, Balmain and parts of the North Shore.
The Airport Express bus, operated by State Transit, costs $7 single, $12 return (within 2 months). It allows luggage. It stops at many stops including the railway station, George Street, Circular Quay, then Kings Cross. The travelpass is not valid on this bus.