Does it matter if I do palm up or palm down for single arm tricep cable overhead extension? Or should I be doing some of each to hit different areas? I find palms down is a little more comfortable for me but I get limited a bit more in weight / reps because grip becomes more of a limiting factor. My goal is mainly hypertrophy / general strength.

I always do palms down, but I also always do that movement bilateral

I really don't think the placement of palms matters much, so if it's more comfortable for you to do it palms down, I'd go with that.

My goal is also hypertrophy, and I'd say my triceps are decently sized

Allign the work to your goal - dips, dips and more dips. You can add close grip bench press, jm presses, skull crushers and board presses. All of these will do more for you than an isolation exercise like tricep extensions.

Is that the only tricep exercise you're doing? OH tricep extension is a great exercise to focus on the long head of the tricep, especially if you stretch deep at the bottom. I would combine that exercise with 1-2 others that focus on the other heads of the triceps.

If your goal is to promote tricep strength for increased stretch in bench or another pressing movement, JM presses are great and also hit long head. Close grip works well too for overall tricep strength.

Really boils down to what your goal is…