If the deal falls through for a reason the offer does not cover, or a condition was already waived in writing, the seller can have a legitimate claim to keep some or all of the deposit, and that dispute is worked out between the lawyers rather than informally between the two parties.
The deposit sits in the brokerage's trust account the entire time, so neither side can simply release it without the other's written agreement or a court order, which is exactly why the condition wording in the offer matters so much before either side signs.
Joel Dyck makes sure every condition a buyer relies on is worded properly before an offer goes in, so a deposit is never at risk on a technicality. Get your hand-reviewed valuation from Joel Dyck.