Sure, but agents almost always have different background information, in some cases radically different background information.

In that case, it's the correct situation that they come up with different prior.

he wanted to claim that there exists some ideal prior that can be derived directly from the problem formulation.

Yes, but he also made a point to always include all background information in the problem formulation. He explicitly wrote so, and his formulas had a trailing term P(...|...,X) to account for this.
It might be interesting to explore what happens to models if you change part of the background information, but I think it's undeniable that with the same information you are bound to come up with the same prior.
This is why I think objective Bayesian probability is a better framework than subjective Bayesian: objectivity accounts for and explains subjectivity.