During the day light hours it's perfectly safe to walk around the city of Newburgh, however once the sun goes down, I don't recommend walking around the city of Newburgh. Newburgh has an extrememly high murder, drug and prostitution rate not to mention robberies, muggings, car break in etc. Stick to the water front if you insist on walking around, there tend to be more people and more cops to halt of criminal behavior.
Unless you're in the actually city of Newburgh, chances are you are going to have to drive from place to place. Most of the tourist attractions are pretty spaced out and the restaurants are scattered about.
The best way to get to and from Newburgh is by car or motorcycle. If you live in the east, just cross the Hudson via Newburgh-Beacon, Mid Hudson or Bear Mtn Bridges and you can get to Newburgh. You can even rid eyour bike into Newburgh on the Newburgh Beacon Bridge. The east bound span has a pedestrian walk to get across. You save a dollar too getting back to the east side :) .