First of all, if he is the lawyer invited by the other party, he will not receive you again, and neither the lawyer nor his law firm can accept your entrustment.
Secondly, according to your hypothesis, he went to the other party's lawyer, and you haven't signed the contract, so the information he got during the conversation with you can't be directly used as evidence.
So you can rest assured!