Yes, the east side is where the poor people live. And the highest percentage of drug dealers. Stay off streets named after fruit trees (Cherry), numbers (4th), and with "wood" in the name (Robinwood) at most times.
Otherwise, you'll be fine. I volunteer in the area and have delivered meals on wheels, secret Santa, etc. and I'm as white as they come. You're actually more likely to be robbed in the parking lots of the Fashion Square or Prime Outlet malls, or major grocery chains - it's where the money is, and they're close to I-75 for a quick getaway.
Don't go to Greenpoint Nature Center alone. I've seen lots of needles and syringes and condoms in the parking lot.
That's about it. Yes, abandoned houses and derelict lots are sad and scary looking, but they don't attack you.
The East Side can be a nice place to visit if you go to the right spots. A lot of it is rundown though, and most of the crime in Saginaw occurs on the East Side. So, be smart, lock your stuff up, etc. like you would in any other city.
Saginaw is an ultra-poor city with high unemployment. This may explain the condition the east-side is in. I drove through while heading home on 75 from up north. I got the heck out of there promptly. I never imagined I would be that frightened. I was afraid to even stop at red lights. I am not exaggerating. There is absolutely no reason to Vacation in Saginaw. If its because of the touristy town of Frankenmuth, then stay in Frankenmuth. Or Bridgeport.