I took this trip with two of my college track teammates...three younger girls. We went out every night, walked back to the hotel in the wee hours of the morning, and drove through the run-down part of town to get to Graceland. And had no problems AT ALL. Were nervous for maybe the first five minutes we were there. Just have some sense about it. Memphis is like any other city. Don't go out alone, don't go wandering in unfamiliar places, and treat the people you see with respect.
Although we found that many attractions in the downtown area are relatively close together and that it is possible to walk to most of them if you are staying downtown, it is most likely smarter and safer to take the trolley or a cab if going any further than a block or two. This is especially true at night. Much of the downtown area is rife with abandoned builings and questionable neighborhoods, especially so if you are from out of town and unsure exactly where you are going.
Do be careful in Memphis. Tourists are rarely targeted and the downtown area routinely has the least crime of any part of the City. Downtown is very heavilly patrolled, and has cameras mounted on every corner. Still, keep your guard up. Dont park on dark streets, dont walk in downtown Alleys at night. Walk with a friend. (except to go to Rendevous). Be aware of your surroundings. Do not carry all of your cash or all of your credit cards, leave some in the hotel safe.