I was kind of in this situation a couple years ago with a lakehouse and proximity to the coast. It's really tough finding a "jack of all trades" boat, cause you seem to sacrifice something either way you go.
Not sure what your budget is, but I wouldn't go anything smaller than 21-22' to fit your family comfortably on. If you've got little kids, higher gunnels may be something to look into. The most power you can afford would be ideal.
I paid cash for a used 21' deep v center console, with high gunnels (I've got little kids), and minimal frills. I keep thinking about upgrading to a bigger, nicer boat, but I kind of like the fact that I can do what I want with this boat and not worry about scratching, scuffing, staining anything like I would on a new boat. My thought was to let the family grow with this one and possibly get a nicer boat in the future. I'm limited in the bays because of draft, but I take comfort in a drier ride in less than ideal conditions.
If I were you, I'd go check out as many boats as you have access to. Take the family and crawl over every inch, open every hatch, sit in the head, and explore. Check out different makes because they all seem to have little features unique to them. Make sure there's enough seating and you can move fore and aft comfortably. If you want to ski/wake board take that into consideration as well, they make brackets for outboards.
Hope you find one that fits you and your family and you guys can make some great memories. My kids LOVE going out on the boat every chance we get.