All you can possible do is to use your intuition to decide if you should use your intuition.

Yes, you have to start with something more basic than expected utility calculation, or you run into an infinite regress. Expected utility calculations are tools and you use them to achieve your goals more effectively. If you want to shoot yourself in the foot, nobody can prevent you from doing so.

Consider someone told you that their intuition says that they should not act on the information they have about risks from AI and that the value of seeking more information is too low because they don't expect to find any information that would change their mind at this point. Then how could you possible come to an agreement with them about risks from AI if you both rely on your intuitions?

You can't reach an agreement. Humans (or minds in general) with widely divergent intuitions or epistemological standards have very different beliefs and it can be impossible for them to come to an agreement. There are no universally compelling arguments that convince all possible minds.

I don't see how it's impossible to assign probabilities by using your intuitions. "Go ahead and pick a number out of the air, but then be very willing to revise it upon the slightest evidence that it doesn’t fit will with your other numbers."