I'll weigh in as my wife and I have been there several times. We always go just off season, so mid November to early December, or late April to early May. There are enough people to keep it interesting, but it is nowhere near as crowded, or expensive, as the high season during winter. We get villas for roughly half the price of the high season. The other poster mentioned Christmas, it is twice as expensive as the rest of the high season and packed (so we've been told).
If you go, I would highly recommend flying from St. Martin over the ferry. It certainly isn't cheap for such a short flight, but so much easier and faster. You also get to land on the second shortest commercial runway in the world, usually the approach is over a hill, the runway starts at the lower part of the hill. Only STOL aircraft can take off and land, so the planes are 8 to 14 or so passengers. Not for the faint of heart, but I love stuff like this and think it's pretty cool. This and the lack of cruise ships actually help keep things less crowded than they probably would be otherwise.
I haven't been to Anguilla to compare, but two of my favorite beaches are there, Governeur and Saline. Lightly used the time of year we go and stunning. Lots of good restaurants and as the other poster mentioned, driving around the island is pretty fun, if not a bit harrowing at times. It is definitely less urban than the Dutch side of St. Martin and less, I would call it rustic, than the French side. The two beaches I mentioned are definitely better than any we went to on St. Martin. We liked it there too though (stayed mainly on the French side a few years back). I have no idea where things stand on St. Barts with regard to hurricane recovery though.
If you want any specific recommendations, let me know.