Can I reply if the overdue withdrawal fails?

You can't.

If the withdrawal fails within the time limit, the other party will give up the patent right for failing to pay the annual fee, that is to say, the open technical content is regarded as a free contribution to society. In this case, using his technology again is not infringing. If it is invalid, you can't reply.