So, this happened to a guy at my level at the comp I work for. It's our policy that if you do this you're not eligible for re-hire. Anyway, he didn't like where he ended up and had to start looking again. Now every time someone calls for his reference, our HR group has to tell them he's not eligible for rehire. Sucks, but that's the game.
Maybe you think you're not worried about burning bridges, but unless you won the lottery or know you'll never need a job, don't do it. It will come up.
When I do interviews, if I like a candidate, I will check references. I call all previous employers on their application. I don't even bother with the listed references. I ask one question: is Mr. X eligible for re-hire. If they say no, when I have the second interview with the candidate I ask them about it.
It's not a deal killer, but of this was the reason, I wouldn't be terribly excited.
Sure there's reasons. You were being treated poorly. Asked to do something unethical, etc. But generally, it's not a good idea.