It goes way, way back.
Baylor is the oldest college in Texas.
A&M is the oldest state college.
We've been playing football since 1899.
Geographic proximity means competition for the same students, especially back when travel was not so easy. This also led to a lot of families with graduates of each school and family feuds.
As said above, Aggie cadets have a long history of poaching Baylor women from the less manly Baylor guys, especially for the first 90-100 years of our history when the female population was low to non-existent in College Station.
Baylor, TCU, SMU, Rice, all used to be competitive with A&M and Texas in football back in the first half of the last century. They still long for those days.
Probably some class warfare, it costs more to go to Baylor, and A&M was historically, well, an A&M college. Under the Morill act, A&M colleges were established for the sons of working men to get a practical education, while Baylor was the place you went to learn Latin and Shakespeare.
In the 1925 A&M/Baylor game, Baylor brought a jalopy Model T onto the field as some sort of "spirit" vehicle, and almost ran over some Aggie football players.
At the 1926 game in Waco, after an agreement that the car would not return, some cars were driven onto the field. Thinking it was the 1925 incident again, Ags rushed the field, and a coed was knocked off the car. Ags said they thought the women on the cars were Baylor men in drag.
Wonder why they would think that

A fight broke out, Aggies said that Baylor had trunks full of wooden planks brought for the occasion and started beating cadets. Baylor claimed the trunks were football equipment.
Colonel Dunn, the Aggie Band director, had the BQs play the National Anthem, and the cadets came to attention, saluted, and the riot stopped.
Senior Cadet Charles Sessums of Dallas was hit in the back of the head with a chair. Some Aggies claim it was after he had come to attention. He walked off the field, but died the next day, probably of a blood clot.
Aggie legend claims that the cadets stopped a northbound train, loaded one of their training cannons on board, and headed to Waco to level the campus. Supposedly only the Texas Rangers stopped them.
Great story, pure bullcrap. Besides, the Ags would have taken infantry, artillery, signal corps, etc. etc. not just one cannon.
It was not a Corps trip, so the entire student body was not at Waco. Aggies said Baylor never would have tried anything with the whole Corps in attendance.
The Ags sent an apology letter to the Baylor newspaper, saying that an Aggie would never intentionally harm a woman. The Baylor paper edited it to read that "an Aggie would never intentionally
touch a woman".
Investigations were made, but the killer was never identified, in spite of tons of witnesses. The father of Sessums suspected that the son of a prominent family was involved, which was why it never came to light. Some Baylor guy probably went to his grave in the last 30 years knowing he was a killer, and never telling a soul.
All athletic contests between the schools were suspended for the next four years, until all involved had a chance to graduate.
As an aside, after the famous Rice MOB incident of 1973, things were getting ugly in the stands. We BQs were told by our director to be ready to play the National Anthem, just like 47 years before. It never came to that.
Now, what was the question again?