If you back out for a reason the contract does not cover, or after already waiving the relevant condition in writing, the seller can have a legitimate claim to keep some or all of the deposit, and that gets resolved between the lawyers rather than informally.
The deposit sits in the brokerage's trust account the entire time specifically so neither side can release it without the other's written agreement or a court order, which is why the exact wording of each condition matters before you sign anything.
Getting that wording right before an offer goes in is worth more than most buyers expect. Get your hand-reviewed valuation from Joel Dyck.