Did they explain why they would keep the deposit if you walked away?

I would tell them that you're having someone else do the floors for you and that you want your deposit back.

A couple of thoughts here.
1) Good on the company you hired for at least trying to make it right by redoing their work. I think that a lot of companies in that position would have ghosted you and stopped answering their phone. It sounds like they are bad at what they do, but at least initially tried to make it right.
2) That being said, I absolutely would not use them moving forward. They clearly aren't competent on what they are doing.
3) Approach them and tell them that you would like to part ways, and have your deposit returned. In return, allow them to pick up any materials that they provided (within reason obviously). If need be, present the documentation that you have stating that the current installation is sub par.
4) If they decline to return your deposit, contact an attorney, and send them a certified demand letter requesting your deposit back. Will probably cost you around $200 unless you have an attorney friend.
5) If that fails, absolutely file a claim in small claims court. You can do all of that yourself, and don't need an attorney. it is easy. Obviously, before you do that you will want to read through all of the fine print of anything you signed. It sounds like you have a good case which would be perfect for small claims court. Just make sure you over the top prepared. Document everything. Timelines of all conversations, pictures, etc... If it goes that far, I would not only ask for my deposit back, but ask for money to remove everything they have done so that you are stating from scratch.

It sounds to me like the installer knows they did a bad job, but would prefer to keep the deposit so that they don't lose money on the materials. My guess is that they just threw option 2 out there to see what happens, knowing full well that you might reject it, but maybe they would get lucky and you would take it. Hopefully when you push back, they will cave, and you wont have to go the court option.

I would claim the full amount of hour deposit. The flooring is not reusable, but they are welcome to come remove it and take it with them.

They are super upside down on the job and they don't want to lose even more on the job so they're trying to keep the deposit. I don't think a judge is going to let them do that, especially without a signed contract.

On a side note, DO NOT EVER GIVE ANY CONTRACTOR ANY MONEY WITHOUT A CONTRACT. EVER EVER.

You may come out on top, but without a contract, you have nothing saying that they will install it per any standard.