I can only comment on International air travel since the 1970s.
In the 1970s, my family flew around to many islands in the Caribbean without passports. My father was a private pilot so I'm talking about doing this in a 4 seater Cessna. I have a copy of my birth certificate which has a stamp on it from traveling to Curacao.
Now, how long could you stay? Was this the same for citizens of all countries? I don't know.
By the mid 1970s, we went to Saudi Arabia for the first time and had to have Passports, Entry and Exit Visas.
During the 1970s, we traveled around Europe including train travel to West Germany, France, Belgium, Netherlands, etc. I distinctly remember the train stopping at the border between West Germany and France and the French border guards checking everybody's passports.
During the 1980s, we had to have passports, visas, etc. to travel abroad except for Mexico and Canada where you only needed a driver's license.
In most cases, as an American, you can usually get an "on-arrival" tourist visa which allows you to stay for up to 90 days or sometimes 180 days.
In the modern EU for example, you can travel between countries in the EU without getting your passport checked.
So, in 2017, I was able to fly into Frankfurt, Germany, rent a car and drive to Belgium and Luxembourg without being stopped.
In 2019, I returned to the EU and they of course checked my previous comings and goings from the EU. But, the passport officers in Madrid were not as meticulous as the officers in Frankfurt.