I'm not much of a pot banger myself, but I think the potential downsides are very easy to see.
1. He has a 17-9 record as a head coach. Most are willing to excuse that given where he made that record, but it is no guarantee that he will end up with a better record in the SEC just because Texas A&M is better than Duke. The is definitely a world where the increased competition of the SEC cancels out the advantages Texas A&M had relative to Duke, and he just maintains a 8-4 type of record like Sumlin did (there is a world where he does a lot worse than this too).
2. He needs to prove he can land a top 5 recruiting class without Jimbo, or we are going to start losing our talent advantage quick. Talk all you want about bringing in guys that "want to play for him", and I will probably agree with you, but if we can't keep up with Bama, Georgia, tu, and LSU in recruiting, we are not going to be able to take that next step.
3. I am in a wait-and-see mode with a lot of his coaching hires. S&C and OC look solid, but it is a bit disconcerting to see how many Florida guys we brought on considering how much of a train wreck they were last season. Also, I say S&C looks good, but there is definitely a possibility that old-school techniques there don't work in the modern portal/NIL world. Personally, I don't care and want him even if that is the case, but it could be a potential issue.
4. He doesn't have the lengthy experience at top tier programs that several coaches do. This is a minor point, as he still has a decade in P5 and half of that time spent at Tier II programs Notre Dame and Texas A&M, but in a league littered with head coaches that were part of national championship teams, it is a slight knock.