Behind the Palais de Papes is the Rocher des Domes. These are gardens laid out in the 19th Century. From here, there are spectacular views across the Rhone. Across the river, one can see not only the Pont St. Benezet, but also the Tour Phillippe-le-Bel, which is where Benezet originally connected to. And, Fort St. Andre, a defensive building constructed during the same period.
The old bridge doesn't go all the way across the river as it was wiped out by a flood.
You can, however, walk out far enough to enjoy the front of the Palace.