Since the down payment is made by the man and his family, and the property was bought before you got married, the down payment and five-year repayment of the property all belong to the man and his family.
The rest of the loan and the appreciation of the house are legally recognized as the property of your husband and wife, no matter who pays the loan.