What is the contract you signed with the agent? If the contract stipulates that the intermediary is responsible for helping you find a satisfactory house and complete the transaction, you can continue to ask the intermediary to recommend the house to you and complete the transaction without paying the intermediary fee. If you don't want to deal with this intermediary again, you can also refund all or part of the agency fee according to the contract.
As for the deposit, it mainly depends on how the intermediary contract signed by you and the homeowner and the intermediary is stipulated. If there is a similar explanation that the intermediary meets the relevant requirements due to the loan procedures, the deposit can be refunded in full. If negotiation fails, it is estimated that the best result is to get back part of the deposit.