An update to this post

It appears that this issue has been discussed before in the thread Naturalism versus unbounded (or unmaximisable) utility options. The discussion there didn't end up drawing the conclusion that perfect rationality doesn't exist, so I believe this current thread adds something new.

Instead, the earlier thread considers the Heaven and Hell scenario where you can spend X days in Hell to get the opportunity to spend 2X days in Heaven. Most of the discussion on that thread was related to the limit of how many days an agent count so as to exit at some point. Stuart Armstrong also comes up with the same solution for demonstrating that this problem isn't related to unbounded utility.

Qiaochu Yaun summarises one of the key takeaways: "This isn't a paradox about unbounded utility functions but a paradox about how to do decision theory if you expect to have to make infinitely many decisions. Because of the possible failure of the ability to exchange limits and integrals, the expected utility of a sequence of infinitely many decisions can't in general be computed by summing up the expected utility of each decision separately."

Cudos to Andreas Giger for noticing what most of the commentators seemed to miss: "How can utility be maximised when there is no maximum utility? The answer of course is that it can't." This is incredibly close to stating that perfect rationality doesn't exist, but it wasn't explicitly stated, only implied.

Further, Wei Dai's comment on a randomised strategy that obtains infinite expected utility is an interesting problem that will be addressed in my next post.

Unfortunately the only opinions you're gonna get on what should be instituted as a norm are subjective ones. So... Take the average? What if not everyone thinks that's a good idea? Etc, etc, it's basically the same problem as all of ethics.

Drawing that distinction between normative and subjective offensiveness still seems useful.

Just encountered an interesting one:

Eradication of the Parasitoid Wasp is genocide!

This seems to be the nature of internet groups in general... most people just don't have the ability to commit to something for long term with no external incentives and weak social ties. I've tried numerous colearning, masterminds, etc. and it's always a struggle.

Even my coaching clients who I give twenty bucks to every time they make a meeting often take 2-3 MONTHS before they can start to do this with anything resembling consistetency.

That being said, here's what I've found helps:

1. Text reminders 15 minutes before every meeting to every person who should show up.
2. Create carrots and sticks using beeminder, habitrpg, or lift as a group.
3. Ping participants through facebook or text one or two times throughout the week with something interesting related to the topic of the group to keep the group top of mind.
4. Create real life bonds otuside of the "let's talk about what we learned" dynamic.

META (anything other than a specific topic to learn)

My guess is that deduction, along with bayesian updating, are being considered part of our rules of inference, rather than axioms.

To amplify on Qiaochu's answer, the part where you promote the Solomonoff prior is Bayesian deduction, a matter of logic - Bayes's Theorem follows from the axioms of probability theory. It doesn't proceed by saying "induction worked, and my priors say that if induction worked it should go on working" - that part is actually implicit in the Solomonoff prior itself, and the rest is pure Bayesian deduction.

Nobody answers - bystander effect. (Not necessarily applicable here since nobody's asking for help, but still.)

I'm actually grateful for having heard about that Basilisk story, because it helped me see Eliezer Yudkowsky is actually human. This may seem stupid, but for quite a while, I idealized him to an unhealthy degree. Now he's still my favorite writer in the history of ever and I trust his judgement way over my own, but I'm able (with some System 2 effort) to disagree with him on specific points.

I can't think I'm entirely alone in this, either. With the plethora of saints and gurus who are about, it does seem evident that human (especially male) psychology has a "mindless follower switch" that just suspends all doubt about the judgement of agents who are beyond some threshold of perceived competence.

Of course such a switch makes a lot of sense from an evolutionary perspective, but it is still a fallible heuristic, and I'm glad to have become aware of it - and the Basilisk helped me get there. So thanks Roko!