Just a couple of precautions that I think every sensible thinking tourist takes anyhow when going to an unknown new place. But in Paldiski especially it is better to gather some information beforehand. After you know it, you'll be fine like anywhere else. Common sense and cautiousness is enough!
1) Be aware of bumpy side roads. The roads might be in very bad condition, and especially if you visit Paldiski winter time, you cannot even see under the snow. There might be holes, items, debris, whatever. Don't break your car.
2) Don't touch any unknown objects on the ground. There shouldn't be anymore any explosives or such, but one never knows. Just think that it has been a Soviet military and nuclear submarine base for decades...
3) Likewise, be careful when walking, wondering and photographing the buildings and surroundings. It is OK and totally legal to take pictures (it is Estonia, not Russia, and the base is not active any more). But take care - the collapsed buildings and partly ruined houses might have some nasty iron bars and stuff. Also, watch your step to avoid falling down after having stepped to something you didn't expect.
4) Nightlife and moving around at night: Better be careful. There is actually no nightlife in Paldiski, at least not official one. I heard there is one local bar or pub, but I didn't want to go there. I had heard that sometimes there are some fights or clashes among the locals. Many of the inhabitants are Russians, and not necessarily other languages are spoken. The unemployment rate is very high in the small town.
It is safe to walk and drive daytime, safe to visit the shop and accommodation etc. If you like to have drink, eat or spend evening, I suggest to stay/go at the (only) hotel Valge Laev on the main street Rae. It is a safe place.
There are lots of sharp and nasty things lying around, and if you don't speak Estonian or Russian you will find it hard to explain you've been bitten/cut by something. I could only find two people who spoke any English.
Do not touch things You do not know what they do on the territory of former soviet military objects. There may be some things that can harm You. Bombs and explosives are almost all removed but some things may be still dangerous like power units still under the current, chemical pollution, metal on floors of building ruins, nails, open cellars, ruins etc.
ads are quite bumpy and full of big holes around Pakri bank.
Drive very slowly.
But the view is worthwhile after some hard time on the roads!