Suggestion 1: physical defense, please ask the boss to install an electric door that can be opened by swiping the card, haha, others will naturally come to you with friendly requests; You can't drive if you have a bad attitude, and you can't drive if your superiors or regulations don't let you in.
Recommendation 2: intellectual defense, setting up signs+changing communication methods; Because you are a buyer, you must read the book Advantage Negotiation by Roger Dawson. When we meet in this way, we speak with "special initiative". For example, can you say that you need my help? Ok, please wait in XXX, or call XXX to get permission before you go in. The sign is "information for visitors", because you are a personnel officer, you can get one yourself and then reimburse it.
Suggestion 2: The best thing to do is not to worry about the simple problem of others' progress or not, but not to be unfavorable. Visitors can be divided into two categories: those who mix poorly and those who mix well. Those who mix poorly are generally well-behaved and easy to deal with. It's usually the so-called awesome people, not the burden of your work. This is an opportunity to make more friends and make more valuable contacts.
I hope you are satisfied. If the answer is ok, praise it.