Just avoid it after dark, especially in front of the sandwich shop/bar entrance on the eastbound side (you'll know you've found the spot I'm talking about when you see the clusters of people just standing there on the sidewalk). This area is notorious for drug activity and other antisocial behavior.
It's not too bad in the daytime, though... Don't let the nighttime shenanigans keep you from visiting the grounds of historic and beautiful Park Place, or the Taber Museum, which are at the same intersection but across the street from the sidewalk gathering place.
If you rent a car, be sure to get thee to Brandon Park at some point (and Brandon Perk for coffee!). However, be warned that directly in front of the park entrance is the most ridiculous intersection in town, dubbed "Confusion Corner" by the locals. Three separate roads (Rural Ave., Market Street, Hepburn Street), plus the one that leads into the park, converge here, and the only thing preventing daily accidents is a healthy dose of caution. Proceed with it!
Here's an example of a local custom, smack in the middle of the downtown residential area.
Rather grim to see, especially while watching children playing right next door.