It's tough to say "in this temperature wear X but wear Y when it's colder" because humidity/moisture and moreso wind can have a huge impact. I spend most of my summers running in the mountains where we fight high winds, cooler temperatures, and afternoon showers.
It was 19 degrees at the start of the Bandera 50K back in January . I wore a pair of running tights with windstopper paneling on the front while my friend wore shorts. To each their own, I guess.
My wife and I are big fans of arm sleeves and short sleeve shirts. With arm sleeves, you have them out and then slowly lower them down to your wrist as you warm up. Unlike a second layer, you don't have two layers covering your armpits and torso where you are probably already warm. And unlike a long sleeve shirt, you aren't having to fight sleeves falling down. If I know that I'll be in long sleeves the whole time, I'll probably wear a half-zip so I can zip down to regulate temperature.
For windy days with no rain, I'll throw in a Patagonia Houdini jacket. It's small enough to fit in a waist band or larger pocket. For days that I know I'll be getting wet, I have a North Face HyperAir GTX (Goretex Active jacket) which is designed to be waterproof but, relatively speaking for anything waterproof, more breathable. Both of these are available in mens and womens fits.
Also, some type of gloves but it's a balance of keeping your hands warm and dry vs having hot and sweaty hands from too much warmth and lack of breathability. Same goes for a hat or beanie. I'll also carry a buff in a pocket as you can never have too many buffs during a winter run.