Generally, if you are not from Azerbaijan and are away from some of the main areas (i.e. Momine, Museums, Parks, etc.) and have a camera, expect for the police to stop you. Even if you are in the main areas of Nakhchivan, expect for many people to ask you what you are doing. You might even have a few people yell at you to stop taking photos.
The reason behind this as it was explained to me is that a few years ago some tourists took photos of Nakhchivan. Only later to have those photos used in a very unflattering way against the Republic. More specifically, they were used to claim that Nakhchivan was Armenian land. Armenia is a very touching subject there. Since that time all foreigners with cameras are suspected of possible wrongdoing.
I was stopped from taking pictures of Mount Alinja and anything near the mountain. I was using a 35 year old Pentax K1000 SLR film camera, I had my Azerbaijani ID to prove I was allowed to be there, but the police still claimed that I might immediately send any of my photos to Armenia and refused for me to take any photos. Later I was using a Sony a100 DSLR in the hills where the old citadel once stood and I was detained for taking photos. Luckily, the police chief knew who I was and I was released. Still, the thought of staying in an Azerbaijani jail was something I did not want to think about.
My best advise is make sure that you have someone who speaks both English and Azerbaijani to assist you if you should decide to see more of the area. Second, carry your passport. Third, if they say do not take photos, don't. The police will confiscate your camera.