Uplistsikhe comes from the words uplis (meaning ruler) and tsikhe (meaning fortress), so Uplistsikhe basically means "fortress of the ruler". It may have been lived in earlier than 1000 BC. The complex of Uplistsikhe covers a slope and the top of a sandstone massif at the edge of the Mtkvari river. It's hard to tell if caves naturally existed in this stone because you can tell that rooms were carved into the stone, but I suspect the caves were there and they were modified extensively over time. Some of the caves have really ornately carved ceilings resembling three-dimentional plaster work. Some of the fronts of the caves have been carved into house-like shapes with triangular roof peaks. In some areas you can tell there were columns standing from floor to ceiling, but they are no where to be seen anymore. Walking along ancient stone walkways and stairwells made me feel like I was walking in a city like Jerusalem before Jesus was ever born. It was absolutely fascinating.
There are basically two ways to get to Uplistsikhe if you are driving there on your own. There is the E60/S1 highway to the north, and the Zahesi-Mtskheta-Kavtiskhevi-Gori road to the south. The E60/S1 highway is probably your most efficient route. From the E60/S1 highway, go south through Gori. Uplistsikhe is approximately 14 km from the center of Gori. Take Stalin Avenue south out of Gori and across the Mtkvari river. After you cross the river take a left onto the Zahesi-Mtskheta-Kavtiskhevi-Gori road. Follow this road through the next village of Khidistavi, keeping to the left when the road forks. About 2 km outside of Khidistavi there should be two brown signs announcing Uplistsikhe; take a left. Follow this road the remaining 7 km, over railroad tracks and over the Mtkvari river to the parking lot. Parking is free. The entrance to the Uplistsikhe complex is modern and there are restrooms, a souvenir shop and a limited cafe just inside.
I've been there two times.
1) first time I went there by train (elektrichka). In Tbilisi central train station there is platform, wich is separately from main platform. They call it Borjomi platform. The trains which go to Borjomi are taking their start from there. So, you have to buy ticket for this very train and went till station "Kva-Khvreli". from this station you need to walk a kilometre or maximum two to get to the place. The train ticket costs approx. 2 Laris.
2) second time I was there by minibus. Me and another Vt member (julie - travelmad478) we took minibus (marshutka) from Didube bus station (in Tbilisi) to Gori. (that costed 4 Laris). Form Gori we found taxi wich took us to Uplistsikhe for 20 Laris. He waited for us and took back in Gori.
I must say that both transports - the train and the bus are dirty and not much comfortable.
Also, the train is taking start at 7 AM and the last one back you have at 5 PM. So you will have some limit in time.
As for minibus you can take it from Tbilisi no matter when, but as for coming back there must be limit also.
Uplistsikhe is somewhere between Kaspi and Gori - somebody might suggest you to go there from Kaspi. But I was told that it is much closer from Gori and the roads are better.
Btw, it's in 15 kilometres from Gori.
one more note: if you go there by train there is the station called Uplistsikhe. One local told me long time ago that better to come down from train on station Kvakhvreli. Cuz it's nearer to the place then from the station Uplistsikhe.
Just so you know... the Uplistsikhe caves were carved out of a sandstone massif, which means there is a lot of sand around that gets whipped up if there is wind. If you wear contacts bring eye drops and contact solution, or glasses in case sand gets in your eyes. I've also heard that if it rains the stone and the wooden stairs that you walk on could become slippery. Lastly, there are a lot of crevaces, slopes, and pits. Wear good walking shoes and keep an eye on where small children are going. There are safety rails but not everywhere they probably should be.