I don't think it's explicitly dangerous. But on our stay downtown we detoured around some homeless guy laid out on the street and police standing around him, some other dude just passed out on the sidewalk, a group of guys just hanging around a corner that just didn't give me a good vibe, some crazy guy early on in the morning wandering in the middle of the street screaming expletives, and some men and women sitting at a bar yelling at tourist looking people. Like me. Anyway, this was two years ago. I was significantly more on edge than I was in NYC, though I haven't been to NYC since the Bloomberg mayorship. We won't stay downtown ever again.
I would go downtown again for Powell's books, that's pretty cool. But they had to have an employee at the door just to turn the homeless back, who kept trying to get it. Never violent, they were intercepted and turned right around. But I saw them a couple times talking to someone, and I've never seen an employee with such an explict job as homeless control.
Note- Screaming Guy was out of our hotel window, so not threatened, just annoyed. We stayed at what is now the Royal Sonesta.