As "Idiocracy" suggests, assortive mating is the villain.
Up until the 1950s or so, it was common to marry people from your hometown or your family's small social circle ... someone you met at school or church.
That type of pairing ensured that there was a mixing of intelligence. A smart person (sometimes the man, sometimes the woman) would marry someone of lesser intelligence but their children had a chance at inheriting "smart genes."
That's why you saw so many inventions and advancements come from people with backgrounds we would look down on today.
Today, smart men marry smart women and because they are committed to work, planning for the future and pouring resources into their offspring, they have one child.
There aren't a lot of "diamonds in the rough" any more because assortive mating severely reduces the odds of it happening.
Not to say there aren't any, but it's not the 1800s any more. And our modern social and economic structures make it almost impossible for some poor kid to rise above his background and contribute a new idea or invention.