Colleagues often ask you to treat, but you refuse without considering his feelings. He has the cheek to ask you to treat without considering your feelings. It's that every time you treat a guest readily, it encourages him to deserve it. Friends and colleagues should be polite, and he should be polite when you invite him to dinner. This is a respectful attitude.
Since he is so rude, you don't have to be polite. If such a colleague has nowhere to go, he will be rejected, and he will not be invited to dinner again in the future, leaving it to grateful people.