We have thought about going to Tokyo for the Olympics, as well. We've considered other Olympic destination, but for us, it quickly gets too expensive with too much uncertainty.
Getting tickets to the events you want can be difficult. Many of the tickets are available in packages. For Rio, it was a hotel+flight+<certain options on tickets>.
You will want to make sure you have tickets to events and a hotel... and the ability to get to the event.
We were in Japan, last summer. I posted about it on the Travel Forum. Several people gave great advice and it was very beneficial. One example is, you can get a
JRail Pass, that allows you access to the rail without buying individual passes. In doing so, you MUST buy it from the US, by a US agency. You take what you buy here, and then redeem it in Tokyo/Narita. This helps limit it to tourists. Highly recommended.
For Japan, MOST hotels are limited to a 2Adults and an infant. They are 2 twin beds in the room that push together for a King. Most don't have rollaway or pullouts. You pay hotel based on beds... and occupancy. So, for us, we stayed in Odaiba at the Hilton. That is one of the few places you can get a 3rd bed. Having 3 people in the room was more expensive than 2. Additionally, if you wanted a 3rd bed in the room, it was an additional charge. We also stayed in Odawara at another Hilton property, because it provided for more than 2 beds. Again, cost per person and per bed.
We spent about $500/night per room. If you look at Aug 2019, Tokyo Hilton Odaiba shows about $367 for the small room (430sq'), plus additional beds, if necessary... the larger room (861sq') is $510.
I would put Tokyo as more expensive than NYCity and much more sprawled (if you are used to Manhatten).
Wonderful place. We went all over (Nagoya, Odawara, Hakone, Tokyo) and could have spent months there without doing every thing.
We even let the kids go wandering around Odaiba without us, because well, I can only take so much Gundham. They had a blast.
All in all, we spent about $5K/person including airfare ($1750 RT), hotel, rail ($400/person for JRail) etc for 11 days. If you are going to an Olympic event, it wouldn't surprise me if you had to double that.