If people in charge follow the epistemology that you propose I think there's a good chance that humanity won't survive this century because someone thinks taking a 1% chance to destroy humanity isn't a problem.

You did not understand the proposal. Let's analyze a situation like that. Suppose there is a physics experiment which has a 99% chance to be safe, and a 1% chance to destroy humanity. The people in charge ask, "Should we accept it as a fact that the experiment will be safe?"

According to our previous stipulations, the utility of believing that it is safe will be 0.99, minus the disutility of believing that it is safe when it is not, so a total of 0.98, considering only the elements of truth and falsehood.

But presumably people care about not destroying humanity as well. Let's say the utility of destroying humanity is negative 1,000,000. Then the total utility of treating it as a fact that the experiment will be safe will be 0.98 - (0.01 * 1,000,000), or in other words, -9,999.02. Very bad. So they will choose not to believe it. Nor does that mean that they will believe the opposite: this would a utility of -0.98, which is still negative. So they will choose to believe neither, and simply say, "It would probably be safe, but there would be too much risk of destroying humanity, so we will not do it." This is presumably the result that you want, and it is also the result from following my proposal.

Any single job application I sent out has a less a more than 80% chance of being rejected. If I would follow your practical advice I wouldn't sent out any applications. That's pretty stupid "practical advice".

This should be analyzed in the same way. You do not choose to say, "This will definitely not be accepted," because you will not maximize your utility that way. Instead, you say, "This will probably not be accepted, but it might be."

In other words, you seem to think that I was proposing a threshold where if something has a certain probability, you suddenly decide to accept it as a fact. There is no such threshold, and I did not propose one. Depending on the case, you will choose to treat something as a fact, when it will maximize your utility to do so. Thus for example when you look out the window and see rain, you say, "It is raining," rather than "It is probably raining," because the cost of adding the qualification in every instance is greater than the benefit of just being right about the rain, given the small risk of being wrong and the harmlessness of that in most cases.

While that's true there's a lot of value of having norms of discussion on LW that uphold the ideal of truth. I don't think it's good to try to act against truthseeking norms like you do here.

I am in favor of truth and truthseeking norms, and I most definitely did not "try to act against truthseeking norms" as you suggest that I did. I am against the falsehood of asserting that you care only about truth. No truthseeker would claim such a thing; but a status seeker might claim it.

But you don't know which algorithm to follow to make yourself believe some arbitrary thing.

Actually, I do. I already said that believing something is basically the same as treating it as a fact, and I know how to treat something as a fact. Again, I might not want to treat it as a fact, but that is no different from not wanting to go to the store: the algorithm is equally clear.

I see absolutely no practical problems in labeling my beliefs as "pretty sure it's true", "likely true", "more likely than not", etc. I do NOT prefer to 'say and think, "this is how it is."'

Your comment history contains many flat out factual claims without any such qualification. Thus your revealed preferences show that you agree with me.

I can easily set up a gradient with something like Amicus Plato, sed magis amica veritas at one end and somebody completely unprincipled on the other.

I agree that there is such a gradient, but that is quite different from a black and white division into people who care about truth and people who don't, as you suggested before. This is practically parallel to the discussion of the binary belief idea: if you don't like the binary beliefs, you should also admit that there is no binary division of people who care about truth and people who don't.

You explicitly said:

   Consequently the total expected utility of believing the theory is -0.192. Therefore I am not going to believe it.

which actually gives zero utility to believing what is true. That puts you in a rather extreme position on that gradient.

First of all, that was a toy model and not a representation of my personal opinions, which is why it started out, "But suppose you also think there is an 80% chance..." If you are asking about my real position on that gradient, I am pretty far into the extreme end of caring about truth. Far enough that I refuse to pronounce the falsehood that I don't care about anything else.

Second, it is unfair even to the toy model to say that it gives zero utility to believing what is true. It assigns a utility of 1 to believing a truth, and therefore 0.8 to a probability of 80% of believing a truth. But the total utility of believing something with a probability of 80% is less, because that probability implies a 20% chance of believing something false, which has negative utility. Finally, in the model, the person adds in utility or disutility from other factors, and ends up with a overall negative utility for believing something that has an 80% chance of being true. I.e. not "truth" and not "zero utility." In particular, to the degree that it is true or probably true, that adds utility. Believing a falsehood with the same consequences, in this model, would have even lower utility.