definitely doesn't converge to any limit

I'm confused about what you want when you say this. I think an example or two would help me.

My best guess: you want the assigned probability to equal the "ideal" probability (as laid out in the probabilistic reflection paper) when we take the limit of computing power going to infinity. It's difficult to take this limit for Abram's original proposal, but if you make it doable by restricting it to S, now there can be things that are true but not provable using statements in S, so infinite computing power doesn't get us there.

Abram Demski's system does exactly this if you take his probability distribution and update on the statements "3 is odd", "5 is odd", etc. in a Bayesian manner. That's because his distribution assigns a reasonable probability to statements like "odd numbers are odd". Updating gives you reasonable updates on evidence.

I have, although formal logic is not my field so please excuse me if I have misunderstood it.

Eliezer does not demonstrate that overcoming the Löbian obstacle is necessary in the construction of tiling agents, he rather assumes it. No form of program verification is actually required, if you do not use the structure of a logical agent. Consider, for example, the GOLUM architecture[1] which is a form of tiling agent that proceeds by direct experimentation (simulation). It does not require an ability to prove logical facts about the soundness and behavior of its offspring, just an ability to run them in simulation. Of course logical program analysis helps in focusing in on the situations which give rise to differing behavior between the two programs, but there are no Gödelian difficulties there (even if there were you could fall back on probabilistic sampling of environments, searching for setups which trigger different results).

The MIRI argument, as I understand it is: “a program which tried to predict the result of modifying itself runs into a Löbian obstacle; we need to overcome the Löbian obstacle to create self-modifying programs with steadfast goal systems.” (I hope I am not constructing a strawman in simplifying it as such.) The problem comes from the implicit assumption that the self-modifying agent will use methods of formal logic to reason about the future actions of its modified self. This need not be the case! There are other methods which work well in practice, converge on stable solutions under the right circumstances, and have been well explored in theory and in practice.

I'm reminded of the apocryphal story of two space-age engineers that meet after the fall of the Soviet Union. The American, who oversaw a $1.5 million programme to develop the “Astronaut Pen” which would write in hard vacuum and microgravity environments, was curious to know how his Russian counterpart solved the same problem. “Simple,” he replied, “we used a pencil.”

You could expend significant time, energy, and donations to solve the problem of Löbian obstacles in the context of tiling agents for self-modifying AI. Or you could use an existing off-the-shelf solution that solves the problem in a different way.

[1] http://goertzel.org/GOLEM.pdf

I agree that it is challenging to assign forecasting power to a study, as we're uncertain about lots of background conditions. There is forecasting power to the degree that the set A of all variables involved with previous subjects allow for predictions about the set A' of variables involved in our case. Though when we deal with Omega who is defined to make true predictions, then we need to take this forecasting power into account, no matter what the underlying mechanism is. I mean, what if Omega in Newcomb's Problem was defined to make true predictions and you don't know anything about the underlying mechanism? Wouldn't you one-box after all? Let's call Omega's prediction P and the future event F. Once Omega's prediction are defined to be true, we can denote the following logical equivalences: P(1 boxing) <--> F(1 boxing) and P(2 boxing) <--> P(2 boxing). Given this conditions, it impossible to 2-box when box B is filled with a million dollars (you could also formulate it in terms of probabilities where such an impossible event would have the probability of 0). I admit that we have to be cautious when we deal with instances that are not defined to make true predictions.

Suppose it is well-known that the wealthy in your country are more likely to adopt a certain distinctive manner of speaking due to the mysterious HavingRichParents gene. If you desire money, could you choose to have this gene by training yourself to speak in this way?

My answer depends on the specific set-up. What exactly do we mean with "It is well-known"? It doesn't seem to be a study that would describe the set A of all factors involved which we then could use to derive A' that applied to our own case. Unless we define "It is well-known" as a instance that allows for predictions in the direction A --> A', I see little reason to assume a forecasting power. Without forecasting power, screening off applies and it would be foolish to train the distinctive manner of speaking. If we specified the game in a way that there is forecasting power at work (or at least we had reason to believe so), depending on your definition of choice (I prefer one that is devoid of free will) you can or cannot choose the gene. These kind of thoughts are listed here or in the section "Newcomb’s Problem’s Problem of Free Will" in the post.

Thanks for the comment!

However, in the A,B-Game we assume that a specific gene makes people presented with two options choose the worse one -- please note that I have not mentioned Omega in this sentence yet! So the claim is not that Omega is able to predict something, but that the gene can determine something, even in absence of the Omega. It's no longer about Omega's superior human-predicting powers; the Omega is there merely to explain the powers of the gene.

I think there might be a misunderstanding. Although I don't believe it to be impossible that a gene causes you to think in specific ways, in the setting of the game such a mechanism is not required. You can also imagine a game where Omega predicts that those who pick a carrot out of a basket of vegetables are the ones that will die shortly of a heart attack. As long as we believe in Omega's forecasting power, its statements are relevant even if we cannot point at any underlying causal mechanisms. As long as the predicted situation is logically possible (here, all agents that pick the carrot die), we don't need to reject Omega's prediction just because such a compilation of events would be unlikely. Though we might call Omega's predictions into question. Still, as long as we believe in its forecasting power (after such a update), we have to take the prediction into account. Hence, the A,B-Game holds even if you don't know of any causal connection between the genes and the behaviour, we only need a credible Omega.

Arguably trying for apostasy, failing due to motivated cognition, and producing only nudging is a good strategy that should be applied more broadly.

A good strategy for what ends?

Incidentally, I don't actually consider being thoughtful about social dynamics a comparative advantage. I think we need more, like, sociologists or something--people who are actually familiar with the pitfalls of being a movement.

That deflates that criticism. For the object-level social dynamics problem, I think that people will not actually care about those problems unless they are incentivised to care about those problems, and it's not clear to me that is possible to do.

What does the person who EA is easy for look like? My first guess is a person who gets warm fuzzies from rigor. But then that suggests they'll overconsume rigor and underconsume altruism.

I'm less concerned about one of the principles failing than I am that the principles won't be enough--that people won't apply them properly because of failures of epistemology.

Is epistemology the real failing, here? This may just be the communism analogy, but I'm not seeing how the incentive structure of EA is lined up with actually getting things done rather than pretending to actually get things done. Do you have a good model of the incentive structure of EA?

I see now that it's not obvious from the finished product, but this was actually the prompt I started with. I removed most of the doom-mongering (of the form "these problems are so bad that they are going to sink EA as a movement") because I found it less plausible than the actual criticisms and wanted to maximize the chance that this post would be taken seriously by effective altruists.

Interesting. The critique you've written strikes me as more "nudging" than "apostasy," and while nudging is probably more effective at improving EA, keeping those concepts separate seems useful. (The rest of this comment is mostly meta-level discussion of nudging vs. apostasy, and can be ignored by anyone interested in just the object-level discussion.)

I interpreted the idea of apostasy along the lines of Avoiding Your Belief's Real Weak Points. Suppose you knew that EA being a good idea was conditional on there being a workable population ethics, and you were uncertain if a workable population ethics existed. Then you would say "well, the real weak spot of EA is population ethics, because if that fails, then the whole edifice comes crashing down." This way, everyone who isn't on board with EA because they're pessimistic about population ethics says "aha, Ben gets it," and possibly people in EA say "hm, maybe we should take the population ethics problem more seriously." This also fits Bostrom's idea- you could tell your past self "look, past Ben, you're not taking this population ethics problem seriously, and if you do, you'll realize that it's impossible and EA is wasted effort." (And maybe another EAer reads your argument and is motivated to find that workable population ethics.)

I think there's a moderately strong argument for sorting beliefs by badness-if-true rather than badness-if-true times plausibility because it's far easier to subconsciously nudge your estimate of plausibility than your estimate of badness-if-true. I want to say there's an article by Yvain or Kaj Sotala somewhere about "I hear criticisms of utilitarianism and think 'oh, that's just uninteresting engineering, someone else will solve that problem' but when I look at other moral theories I think 'but they don't have an answer for X!' and think that sinks their theory, even though its proponents see X as just uninteresting engineering," which seems to me a good example of what differing plausibility assumptions look like in practice. Part of the benefit of this exercise seems to be listing out all of the questions whose answers could actually kill your theory/plan/etc., and then looking at them together and saying "what is the probability that none of these answers go against my theory?"

Now, it probably is the case that the total probability is small. (This is a belief you picked because you hold it strongly and you've thought about it a long time, not one picked at random!) But the probably may be much higher than it seems at first, because you may have dismissed an unpleasant possibility without fully considering it. (It also may be that by seriously considering one of these questions, you're able to adjust EA so that the question no longer has the chance of killing EA.)

As an example, let's switch causes to cryonics. My example of cryonics apostasy is "actually, freezing dead people is probably worthless; we should put all of our effort into making it legal to freeze live people once they get a diagnosis of a terminal condition or a degenerative neurological condition" and my example of cryonics nudging is "we probably ought to have higher fees / do more advertising and outreach." The first is much more painful to hear, and that pain is both what makes it apostasy and what makes it useful to actually consider. If it's true, the sooner you know the better.

A friend lent me this exact book recently, and while I followed the first chapter, I quickly gave up upon encountering the exercises, because I could not even do the first one. At all.

I don't think I am stupid, so I hope you're just underestimating the required mathematical thinking experience (if not actual background knowledge, none of which is assumed) required to get through this book. Mathematicians do so as a rule.

Christian apologist William Lane Craig claims the skeptical slogan "extraordinary claims require extraordinary evidence" is contradicted by probability theory, because it actually wouldn't take all that much evidence to convince us that, for example, "the numbers chosen in last night's lottery were 4, 2, 9, 7, 8 and 3." The correct response to this argument is to say that the prior probability of a miracle occurring is orders of magnitude smaller than mere one in a million odds.

This only talks about the probability of the evidence given the truth of the hypothesis, but ignores the probability of the evidence given its falsity. For a variety of reasons, fake claims of miracles are far more common than fake TV announcements of the lottery numbers, which drastically reduces the likelihood ratio you get from the miracle claim relative to the lotto announcement.

The specific miracle also has lower prior probability (miracles are possible+this specific miracle's details), but that's not the only issue.

3 and 4 are generalizations to sets of sentences. But you're right, the generalization is pretty simple.

Arbitrarily large models are allowed in the first-order theory of arithmetic, and no first-order theory of arithmetic can restrict models to only the integers. This is one of the surprising results of compactness.