Why not biking/cycling? The region around Bautzen is hilly but not with too much elevation difference. There are plenty of biking trails, one of them leads along the Spree river, from the spring through picturesque villages/small towns via Bautzen and further to Berlin.
Bring your bike/equipment or rent - you can return it at a different place. Trails are well marked, inexpensive accommodation (Zimmer frei) is to find in each town along the way. The 'other contact' link below directs you to an interactive map of the region.
Bautzen is right in the centre of the public transportation network "ZVON" which covers the Saxon parts of Lausitz and Lower Silesia. Aside from the train routes between the major towns Bautzen, Görlitz and Zittau most transportation is done by bus. It is a rural regiona, meaning that buses don't run as frequently as around bigger cities like Dresden or Leipzig. So please plan accordingly and use the excellent website of the ZVON (see below) which is available in German, Sorbian, Czech and Polish languages.
A day ticket for the whole region costs 10 Euro for a single traveller and 20 Euro for a group up to five people. The same prices apply for the Euro-Neisse-Day ticket which covers the neighbouring regions in Poland as far as Boleslawiec in Poland, Liberec in the Czech Republic and even parts of the Krkonose/Karkonosze mountains in both countries. YAY!
To get around in Bautzen itself you can take city buses but I'd recommend to walk - there's so much to see in the town that it would be a waste to just rush by in a bus. Distances are walkable.
The most convenient way to get to Bautzen is by train. The town is located on the semi-major train route Dresden - Görlitz - Wroclaw (Poland) [Train connections with Dresden since 1846]. Regional trains run between Dresden and Görlitz at least hourly or more often and until around midnight. International trains between Dresden and Wroclaw run a couple of times daily. All of those stop in Bautzen.
Since these are regional trains you can make use of the so called 'Länderticket' - the Saxony ticket, which is actually valid in the three 'Länder' (states) Saxony, Thuringia and Saxony-Anhalt costs 20 Euro for singles, 28 Euro for groups up to five people travelling together. Pleae note: these tickets are valid from 9 am on on weekdays, all day on weekends. Validate them (fill in your name) before you board the train.
The train station in Bautzen is located a ten-minutes walk south from the old town. The building dates from 1921 and was renovated after damages in 1945. Worth a look are the sgraffiti by Alfred Herzog from that era, depicting traditional branches of the local economy (pic #4).