Then according to your situation, you should want to sign an exclusive authorization!
If you go to the patent office for filing, he won't care how much you authorized, nor will he investigate whether you authorized it before, just for filing!
Then the relationship between a factory and you in the future is a civil situation, which needs to be solved separately!
Of course, you can also sign an exclusive authorization that will not be authorized to others except factory A, so you can only sign an exclusive authorization to tell you two, and you can't authorize others!
I hope I can help you, and I hope to adopt it!