We stayed at Koh Samui a few days, it was enjoyable but still a pretty busy island in places.
We ended up staying here:
Baan TalayNice beach side place that was clean and enjoyable. We would walk down the beach in the evening and just pop in some where to eat dinner.
If you are planning on diving, I would recommend to go ahead and go to Koh Tao. That's where you will end up anyway if you dive out of Koh Samui anyway. You might be able to swing a dive out of Koh Samui and just have them drop you off at Tao afterwards, so you avoid the ferry.
At Tau we dove at "Master Divers." They were a good shop and friendly. We were a little put off because our "daily dive master" was very green and kind of bossy. In fact she had just gotten her DM the week before and had never dove anywhere else besides Tao. They also made me do a refresher course since i had not dove in 6 months, which was fairly lame.
Diving quality was decent, we were disappointed honestly. Vis wasn't great, maybe around 40-50 ft daily. Sea life was good though. We dove Cozumel a few months later and it blew Tao away.
But overall the experience was fun, the island is very chill.
Let me know if you want more info.....