Overall,Vancouver was a very safe city. Everyone is warned about the downtown eastside and it should probably be avoided. When in Gastown, don't go too far in either direction and you should be fine.
I found the homeless people on the streets to be intimidating at first, but they're harmless. Giving them money is a personal choice, but if you refuse, they usually just leave you alone and move on.