(1) What is the patent?
It's not that you said that the magnet RbFeB material has such a patent. In fact, there may be thousands or even tens of thousands of patents for this material. So, I don't know which patent you are talking about.
(2) What is the technology used in the product?
You should judge the infringement and compare your product with the patent of the other party. What kind of technology does your product use? I don't know what your skill is. What do you compare with patents?
Combined with your question,
(1) As long as it is not a loudspeaker product sold in Japan, it will not be affected by the magnet patent. Is that so?
What technology is used for the speaker single product? Which patent is the magnet patent?
(2) In this case, if my products are sold to Japan, will there be any problem?
That depends on whether there are patents in Japan and what technology is used in the products you export.
Generally speaking, it is impossible to judge.