The first use right is after the application date but before publication. Considering that the first user has made preparations for production, equipment, workshop, personnel and materials, it is still quite a loss to withdraw at once. Therefore, due to human factors, he can continue to operate within the original scope. Because your patent is not made public, he has no subjective malice of plagiarism.
Of course, your doubts are also justified. Infringement is infringement, and the principle is not enough. And how to judge whether he is in the original range is not easy to control, such as output and sales area. Even if you stare at him, you may not get these data, which is really difficult to implement and define in reality.