However, this attitude may also make you feel uneasy and confused. If you want a divorce, you can try to communicate frankly with him and express your thoughts and feelings. If you are not sure or don't want a divorce, you can also tell him and discuss how to solve the problem.
Anyway, I suggest you consult a professional marriage counselor before making any decision. They can help you clear your mind and give valuable advice.