My best swim in Croatia was at Plat.
A pebble beach, but safe to enter, without any remarkable rock and wide enough to swim in any direction without disturbing the other bathers. It's some kilometers south of town, taking a few minutes by car (or by boat), but fortunately, adjacent to our hotel.
Banje beach is OK, nice turquoise water with a view of the Old Town walls at one end, but it is quite crowded and noisy - at least it was when we were there. It is free but there are no changing rooms. People try to change in the bathroom in the cafe, which gets the cafe people very upset. So you need to buy a drink or something in order to use the bathroom.
Another option, if you don't mind laying out some money, is renting one of the white canopy beds I saw on the beach (I've never seen such a thing before but maybe I'm not enough of a beach-goer). These beds come with a roof and curtains all around, so you can hide from the crowds (and the sun).
To reach Banje beach, you exit the walled City of Dubrovnik through the east gate (Ploce Gate). Walk 100 meters (300 feet), walk down the steps and you are there. If you are arriving at the port, which is in Gruz, you walk past the port to the intersection and get on a bus to the Old Town. You can also take a taxi. Some cruise companies provide a free shuttle that drops you off right outside the Old Town, at Pile Gate.