Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Leaving LessWrong for a more rational life

33 [deleted] 21 May 2015 07:24PM

You are unlikely to see me posting here again, after today. There is a saying here that politics is the mind-killer. My heretical realization lately is that philosophy, as generally practiced, can also be mind-killing.

As many of you know I am, or was running a twice-monthly Rationality: AI to Zombies reading group. One of the bits I desired to include in each reading group post was a collection of contrasting views. To research such views I've found myself listening during my commute to talks given by other thinkers in the field, e.g. Nick Bostrom, Anders Sandberg, and Ray Kurzweil, and people I feel are doing “ideologically aligned” work, like Aubrey de Grey, Christine Peterson, and Robert Freitas. Some of these were talks I had seen before, or generally views I had been exposed to in the past. But looking through the lens of learning and applying rationality, I came to a surprising (to me) conclusion: it was philosophical thinkers that demonstrated the largest and most costly mistakes. On the other hand, de Grey and others who are primarily working on the scientific and/or engineering challenges of singularity and transhumanist technologies were far less likely to subject themselves to epistematic mistakes of significant consequences.

Philosophy as the anti-science...

What sort of mistakes? Most often reasoning by analogy. To cite a specific example, one of the core underlying assumption of singularity interpretation of super-intelligence is that just as a chimpanzee would be unable to predict what a human intelligence would do or how we would make decisions (aside: how would we know? Were any chimps consulted?), we would be equally inept in the face of a super-intelligence. This argument is, however, nonsense. The human capacity for abstract reasoning over mathematical models is in principle a fully general intelligent behaviour, as the scientific revolution has shown: there is no aspect of the natural world which has remained beyond the reach of human understanding, once a sufficient amount of evidence is available. The wave-particle duality of quantum physics, or the 11-dimensional space of string theory may defy human intuition, i.e. our built-in intelligence. But we have proven ourselves perfectly capable of understanding the logical implications of models which employ them. We may not be able to build intuition for how a super-intelligence thinks. Maybe—that's not proven either. But even if that is so, we will be able to reason about its intelligent behaviour in advance, just like string theorists are able to reason about 11-dimensional space-time without using their evolutionarily derived intuitions at all.

This post is not about the singularity nature of super-intelligence—that was merely my choice of an illustrative example of a category of mistakes that are too often made by those with a philosophical background rather than the empirical sciences: the reasoning by analogy instead of the building and analyzing of predictive models. The fundamental mistake here is that reasoning by analogy is not in itself a sufficient explanation for a natural phenomenon, because it says nothing about the context sensitivity or insensitivity of the original example and under what conditions it may or may not hold true in a different situation.

A successful physicist or biologist or computer engineer would have approached the problem differently. A core part of being successful in these areas is knowing when it is that you have insufficient information to draw conclusions. If you don't know what you don't know, then you can't know when you might be wrong. To be an effective rationalist, it is often not important to answer “what is the calculated probability of that outcome?” The better first question is “what is the uncertainty in my calculated probability of that outcome?” If the uncertainty is too high, then the data supports no conclusions. And the way you reduce uncertainty is that you build models for the domain in question and empirically test them.

The lens that sees its own flaws...

Coming back to LessWrong and the sequences. In the preface to Rationality, Eliezer Yudkowsky says his biggest regret is that he did not make the material in the sequences more practical. The problem is in fact deeper than that. The art of rationality is the art of truth seeking, and empiricism is part and parcel essential to truth seeking. There's lip service done to empiricism throughout, but in all the “applied” sequences relating to quantum physics and artificial intelligence it appears to be forgotten. We get instead definitive conclusions drawn from thought experiments only. It is perhaps not surprising that these sequences seem the most controversial.

I have for a long time been concerned that those sequences in particular promote some ungrounded conclusions. I had thought that while annoying this was perhaps a one-off mistake that was fixable. Recently I have realized that the underlying cause runs much deeper: what is taught by the sequences is a form of flawed truth-seeking (thought experiments favored over real world experiments) which inevitably results in errors, and the errors I take issue with in the sequences are merely examples of this phenomenon.

And these errors have consequences. Every single day, 100,000 people die of preventable causes, and every day we continue to risk extinction of the human race at unacceptably high odds. There is work that could be done now to alleviate both of these issues. But within the LessWrong community there is actually outright hostility to work that has a reasonable chance of alleviating suffering (e.g. artificial general intelligence applied to molecular manufacturing and life-science research) due to concerns arrived at by flawed reasoning.

I now regard the sequences as a memetic hazard, one which may at the end of the day be doing more harm than good. One should work to develop one's own rationality, but I now fear that the approach taken by the LessWrong community as a continuation of the sequences may result in more harm than good. The anti-humanitarian behaviors I observe in this community are not the result of initial conditions but the process itself.

What next?

How do we fix this? I don't know. On a personal level, I am no longer sure engagement with such a community is a net benefit. I expect this to be my last post to LessWrong. It may happen that I check back in from time to time, but for the most part I intend to try not to. I wish you all the best.

A note about effective altruism…

One shining light of goodness in this community is the focus on effective altruism—doing the most good to the most people as measured by some objective means. This is a noble goal, and the correct goal for a rationalist who wants to contribute to charity. Unfortunately it too has been poisoned by incorrect modes of thought.

Existential risk reduction, the argument goes, trumps all forms of charitable work because reducing the chance of extinction by even a small amount has far more expected utility than would accomplishing all other charitable works combined. The problem lies in the likelihood of extinction, and the actions selected in reducing existential risk. There is so much uncertainty regarding what we know, and so much uncertainty regarding what we don't know that it is impossible to determine with any accuracy the expected risk of, say, unfriendly artificial intelligence creating perpetual suboptimal outcomes, or what effect charitable work in the area (e.g. MIRI) is have to reduce that risk, if any.

This is best explored by an example of existential risk done right. Asteroid and cometary impacts is perhaps the category of external (not-human-caused) existential risk which we know the most about, and have done the most to mitigate. When it was recognized that impactors were a risk to be taken seriously, we recognized what we did not know about the phenomenon: what were the orbits and masses of Earth-crossing asteroids? We built telescopes to find out. What is the material composition of these objects? We built space probes and collected meteorite samples to find out. How damaging an impact would there be for various material properties, speeds, and incidence angles? We built high-speed projectile test ranges to find out. What could be done to change the course of an asteroid found to be on collision course? We have executed at least one impact probe and will monitor the effect that had on the comet's orbit, and have on the drawing board probes that will use gravitational mechanisms to move their target. In short, we identified what it is that we don't know and sought to resolve those uncertainties.

How then might one approach an existential risk like unfriendly artificial intelligence? By identifying what it is we don't know about the phenomenon, and seeking to experimentally resolve that uncertainty. What relevant facts do we not know about (unfriendly) artificial intelligence? Well, much of our uncertainty about the actions of an unfriendly AI could be resolved if we were to know more about how such agents construct their thought models, and relatedly what language were used to construct their goal systems. We could also stand to benefit from knowing more practical information (experimental data) about in what ways AI boxing works and in what ways it does not, and how much that is dependent on the structure of the AI itself. Thankfully there is an institution that is doing that kind of work: the Future of Life institute (not MIRI).

Where should I send my charitable donations?

Aubrey de Grey's SENS Research Foundation.

100% of my charitable donations are going to SENS. Why they do not get more play in the effective altruism community is beyond me.

If you feel you want to spread your money around, here are some non-profits which have I have vetted for doing reliable, evidence-based work on singularity technologies and existential risk:

  • Robert Freitas and Ralph Merkle's Institute for Molecular Manufacturing does research on molecular nanotechnology. They are the only group that work on the long-term Drexlarian vision of molecular machines, and publish their research online.
  • Future of Life Institute is the only existential-risk AI organization which is actually doing meaningful evidence-based research into artificial intelligence.
  • B612 Foundation is a non-profit seeking to launch a spacecraft with the capability to detect, to the extent possible, ALL Earth-crossing asteroids.

I wish I could recommend a skepticism, empiricism, and rationality promoting institute. Unfortunately I am not aware of an organization which does not suffer from the flaws I identified above.

Addendum regarding unfinished business

I will no longer be running the Rationality: From AI to Zombies reading group as I am no longer in good conscience able or willing to host it, or participate in this site, even from my typically contrarian point of view. Nevertheless, I am enough of a libertarian that I feel it is not my role to put up roadblocks to others who wish to delve into the material as it is presented. So if someone wants to take over the role of organizing these reading groups, I would be happy to hand over the reigns to that person. If you think that person should be you, please leave a reply in another thread, not here.

EDIT: Obviously I'll stick around long enough to answer questions below :)

Comments (273)

Comment author: someonewrongonthenet 22 May 2015 04:12:13AM 15 points [-]

Recently I have realized that the underlying cause runs much deeper: what is taught by the sequences is a form of flawed truth-seeking (thought experiments favored over real world experiments) which inevitably results in errors, and the errors I take issue with in the sequences are merely examples of this phenomenon.

I guess I'm not sure how these concerns could possibly be addressed by any platform meant for promoting ideas. You cannot run a lab in your pocket. You can have citations to evidence found by people who do run labs...but that's really all you can do. Everything else must necessarily be a thought experiment.

So my question is, can you envision a better version, and what would be some of the ways that it would be different? (Because if you can, it aught to be created.)

Comment author: John_Maxwell_IV 22 May 2015 04:54:38AM *  38 points [-]

Thanks for sharing your contrarian views, both with this post and with your previous posts. Part of me is disappointed that you didn't write more... it feels like you have several posts' worth of objections to Less Wrong here, and at times you are just vaguely gesturing towards a larger body of objections you have towards some popular LW position. I wouldn't mind seeing those objections fleshed out in to long, well-researched posts. Of course you aren't obliged to put in the time & effort to write more posts, but it might be worth your time to fix specific flaws you see in the LW community given that it consists of many smart people interested in maximizing their positive impact on the far future.

I'll preface this by stating some points of general agreement:

  • I haven't bothered to read the quantum physics sequence (I figure if I want to take the time to learn that topic, I'll learn from someone who researches it full-time).

  • I'm annoyed by the fact that the sequences in practice seem to constitute a relatively static document that doesn't get updated in response to critiques people have written up. I think it's worth reading them with a grain of salt for that reason. (I'm also annoyed by the fact that they are extremely wordy and mostly without citation. Given the choice of getting LWers to either read the sequences or read Thinking Fast and Slow, I would prefer they read the latter; it's a fantastic book, and thoroughly backed up by citations. No intellectually serious person should go without reading it IMO, and it's definitely a better return on time. Caveat: I personally haven't read the sequences through and through, although I've read lots of individual posts, some of which were quite insightful. Also, there is surprisingly little overlap between the two works and it's likely worthwhile to read both.)

And here are some points of disagreement :P

You talk about how Less Wrong encourages the mistake of reasoning by analogy. I searched for "site:lesswrong.com reasoning by analogy" on Google and came up with these 4 posts: 1, 2, 3, 4. Posts 1, 2, and 4 argue against reasoning by analogy, while post 3 claims the situation is a bit more nuanced. In this comment here, I argue that reasoning by analogy is a bit like taking the outside view: analogous phenomena can be considered part of the same (weak) reference class. So...

  • Insofar as there is an explicit "LW consensus" about whether reasoning by analogy is a good idea, it seems like you've diagnosed it incorrectly (although maybe there are implicit cultural norms that go against professed best practices).

  • It seems useful to know the answer to questions like "how valuable are analogies", and the discussions I linked to above seem like discussions that might help you answer that question. These discussions are on LW.

  • Finally, it seems you've been unable to escape a certain amount of reasoning by analogy in your post. You state that experimental investigation of asteroid impacts was useful, so by analogy, experimental investigation of AI risks should be useful.

The steelman of this argument would be something like "experimentally, we find that investigators who take experimental approaches tend to do better than those who take theoretical approaches". But first, this isn't obviously true... mathematicians, for instance, have found theoretical approaches to be more powerful. (I'd guess that the developer of Bitcoin took a theoretical rather than an empirical approach to creating a secure cryptocurrency.) And second, I'd say that even this argument is analogy-like in its structure, since the reference class of "people investigating things" seems sufficiently weak to start pushing in to analogy territory. See my above point about how reasoning by analogy at its best is reasoning from a weak reference class. (Do people think this is worth a toplevel post?)

This brings me to what I think is my most fundamental point of disagreement with you. Viewed from a distance, your argument goes something like "Philosophy is a waste of time! Resolve your disagreements experimentally! There's no need for all this theorizing!" And my rejoinder would be: Resolving disagreements experimentally is great... when it's possible. We'd love to do a randomized controlled trial of whether universes with a Machine Intelligence Research Institute are more likely to have a positive singularity, but that unfortunately we don't currently know how to do that.

There are a few issues with too much emphasis of experimentation over theory. The first issue is that you may be tempted to prefer experimentation over theory even for problems that theory is better suited for (e.g. empirically testing prime number conjectures). The second issue is that you may fall prey to the streetlight effect and prioritize areas of investigation that look tractable from an experimental point of view, ignoring questions that are both very important and not very tractable experimentally.

You write:

Well, much of our uncertainty about the actions of an unfriendly AI could be resolved if we were to know more about how such agents construct their thought models, and relatedly what language were used to construct their goal systems.

This would seem to depend on the specifics of the agent in question. This seems like a potentially interesting line of inquiry. My impression is that MIRI thinks most possible AGI architectures wouldn't meet its standards for safety, so given that their ideal architecture is so safety-constrained, they're focused on developing the safety stuff first before working on constructing thought models etc. This seems like a pretty reasonable approach for an organization with limited resources, if it is in fact MIRI's approach. But I could believe that value could be added by looking at lots of budding AGI architectures and trying to figure out how one might make them safer on the margin.

We could also stand to benefit from knowing more practical information (experimental data) about in what ways AI boxing works and in what ways it does not, and how much that is dependent on the structure of the AI itself.

Sure... but note that Eliezer Yudkowsky from MIRI was the one who invented the AI box experiment and ran the first few experiments, and FHI wrote this paper consisting of a bunch of ideas for what AI boxes consist of. (The other thing I didn't mention as a weakness of empiricism is that empiricism doesn't tell you what hypotheses might be useful to test. Knowing what hypotheses to test is especially nice to know when testing hypotheses is expensive.)

I could believe that there are fruitful lines of experimental inquiry that are neglected in the AI safety space. Overall it looks kinda like crypto to me in the sense that theoretical investigation seems more likely to pan out. But I'm supportive of people thinking hard about specific useful experiments that someone could run. (You could survey all the claims in Bostrom's Superintelligence and try to estimate what fraction could be cheaply tested experimentally. Remember that just because a claim can't be tested experimentally doesn't mean it's not an important claim worth thinking about...)

Comment author: TheAncientGeek 22 May 2015 10:21:43AM *  3 points [-]

I'm annoyed by the fact that the sequences in practice seem to constitute a relatively static document that doesn't get updated in response to critiques people have written up

For a start.... there's also a lack of discernible point in a lot of places. But too much good stuff to justify rejecting the whole thing.

Comment author: [deleted] 23 May 2015 01:33:08PM *  5 points [-]

it seems you've been unable to escape a certain amount of reasoning by analogy in your post. You state that experimental investigation of asteroid impacts was useful, so by analogy, experimental investigation of AI risks should be useful.

It seems I should have picked a different phrase to convey my intended target of ire. The problem isn't concept formation by means of comparing similar reference classes, but rather using thought experiments as evidence and updating on them.

To be sure, thought experiments are useful for noticing when you are confused. They can also be semi-dark art in providing intuition pumps. Einstein did well in introducing special relativity by means of a series of thought experiments, by getting the reader to notice their confusion over classical electromagnetism in moving reference frames, then providing an intuition pump for how his own relativity worked in contrast. It makes his paper one of the most beautiful works in all of physics. However it was the experimental evidence which proved Einstein right, not the gedankenexperimenten.

If a thought experiment shows something to not feel right, that should raise your uncertainty about whether your model of what is going on is correct or not (notice your confusion), to whit the correct response should be “how can I test my beliefs here?” Do NOT update on thought experiments, as thought experiments are not evidence. The thought experiment triggers an actual experiment—even if that experiment is simply looking up data that is already collected—and the actual experimental results is what updates beliefs.

My impression is that MIRI thinks most possible AGI architectures wouldn't meet its standards for safety.

MIRI has not to my knowledge released any review of existing AGI architectures. If that is their belief, the onus is on them to support it.

but note that Eliezer Yudkowsky from MIRI was the one who invented the AI box experiment

He invented the AI box game. If it's an experiment, I don't know what it is testing. It is a setup totatly divorced from any sane reality for how AGI might actually develop and what sort of controls might be in place, with built-in rules that favor the AI.

Yet nevertheless, time and time again people such as yourself point me to the AI box games as if it demonstrated anything of note, anything which should cause me to update my beliefs.

It is, I think, the examples of the sequences and the character of many of the philosophical discussions which happen here that drive people to feel justified in making such ungrounded inferences. And it is that tendency which possibly makes the sequences and/or less wrong a memetic hazard.

Comment author: Valentine 23 May 2015 04:19:22PM 9 points [-]

If a thought experiment shows something to not feel right, that should raise your uncertainty about whether your model of what is going on is correct or not (notice your confusion), to whit the correct response should be “how can I test my beliefs here?”

I have such very strong agreement with you here.

The problem isn't concept formation by means of comparing similar reference classes, but rather using thought experiments as evidence and updating on them.

…but I disagree with you here.

Thought experiments and reasoning by analogy and the like are ways to explore hypothesis space. Elevating hypotheses for consideration is updating. Someone with excellent Bayesian calibration would update much much less on thought experiments etc. than on empirical tests, but you run into really serious problems of reasoning if you pretend that the type of updating is fundamentally different in the two cases.

I want to emphasize that I think you're highlighting a strength this community would do well to honor and internalize. I strongly agree with a core point I see you making.

But I think you might be condemning screwdrivers because you've noticed that hammers are really super-important.

Comment author: [deleted] 23 May 2015 06:50:03PM *  -1 points [-]

Selecting a likely hypothesis for consideration does not alter that hypothesis' likelihood. Do we agree on that?

Comment author: Valentine 23 May 2015 08:17:04PM 4 points [-]

Hmm. Maybe. It depends on what you mean by "likelihood", and by "selecting".

Trivially, noticing a hypothesis and that it's likely enough to justify being tested absolutely is making it subjectively more likely than it was before. I consider that tautological.

If someone is looking at n hypotheses and then decided to pick the kth one to test (maybe at random, or maybe because they all need to be tested at some point so why not start with the kth one), then I quite agree, that doesn't change the likelihood of hypothesis #k.

But in my mind, it's vividly clear that the process of plucking a likely hypothesis out of hypothesis space depends critically on moving probability mass around in said space. Any process that doesn't do that is literally picking a hypothesis at random. (Frankly, I'm not sure a human mind even can do that.)

The core problem here is that most default human ways of moving probability mass around in hypothesis space (e.g. clever arguments) violate the laws of probability, whereas empirical tests aren't nearly as prone to that.

So, if you mean to suggest that figuring out which hypothesis is worthy of testing does not involve altering our subjective likelihood that said hypothesis will turn out to be true, then I quite strongly disagree.

But if you mean that clever arguments can't change what's true even by a little bit, then of course I agree with you.

Perhaps you're using a Frequentist definition of "likelihood" whereas I'm using a Bayesian one?

Comment author: [deleted] 24 May 2015 04:05:35AM *  0 points [-]

Perhaps you're using a Frequentist definition of "likelihood" whereas I'm using a Bayesian one?

There's a difference? Probability is probability.

So, if you mean to suggest that figuring out which hypothesis is worthy of testing does not involve altering our subjective likelihood that said hypothesis will turn out to be true, then I quite strongly disagree.

But if you mean that clever arguments can't change what's true even by a little bit, then of course I agree with you.

If you go about selecting a hypothesis by evaluating a space of hypotheses to see how they rate against your model of the world (whether you think they are true) and against each other (how much you stand to learn by testing them), you are essentially coming to reflective equilibrium regarding these hypothesis and your current beliefs. What I'm saying is that this shouldn't change your actual beliefs -- it will flush out some stale caching, or at best identify an inconsistent belief, including empirical data that you haven't fully updated on. But it does not, by itself, constitute evidence.

So a clever argument might reveal an inconsistency in your priors, which in turn might make you want seek out new evidence. But the argument itself is insufficient for drawing conclusions. Even if the hypothesis is itself hard to test.

Comment author: Valentine 24 May 2015 05:28:14PM 8 points [-]

Perhaps you're using a Frequentist definition of "likelihood" whereas I'm using a Bayesian one?

There's a difference? Probability is probability.

There very much is a difference.

Probability is a mathematical construct. Specifically, it's a special kind of measure p on a measure space M such that p(M) = 1 and p obeys a set of axioms that we refer to as the axioms of probability (where an "event" from the Wikipedia page is to be taken as any measurable subset of M).

This is a bit like highlighting that Euclidean geometry is a mathematical construct based on following thus-and-such axioms for relating thus-and-such undefined terms. Of course, in normal ways of thinking we point at lines and dots and so on, pretend those are the things that the undefined terms refer to, and proceed to show pictures of what the axioms imply. Formally, mathematicians refer to this as building a model of an axiomatic system. (Another example of this is elliptic geometry, which is a type of non-Euclidean geometry, which you can model as doing geometry on a sphere.)

The Frequentist and Bayesian models of probability theory are relevantly different. They both think of M as the space of possible results (usually called the "sample space" but not always) and a measurable subset EM as an "event". But they use different models of p:

  • Frequentists suggest that were you to look at how often all of the events in M occur, the one we're looking at (i.e., E) would occur at a certain frequency, and that's how we should interpret p(E). E.g., if M is the set of results from flipping a fair coin and E is "heads", then it is a property of the setup that p(E) = 0.5. A different way of saying this is that Frequentists model p as describing a property of that which they are observing - i.e., that probability is a property of the world.
  • Bayesians, on the other hand, model p as describing their current state of confidence about the true state of the observed phenomenon. In other words, Bayesians model p as being a property of mental models, not of the world. So if M is again the results from flipping a fair coin and E is "heads", then to a Bayesian the statement p(E) = 0.5 is equivalent to saying "I equally expect getting a heads to not getting a heads from this coin flip." To a Bayesian, it doesn't make sense to ask what the "true" probability is that their subjective probability is estimating; the very question violates the model of p by trying to sneak in a Frequentist presumption.

Now let's suppose that M is a hypothesis space, including some sector for hypotheses that haven't yet been considered. When we say that a given hypothesis H is "likely", we're working within a partial model, but we haven't yet said what "likely" means. The formalism is easy: we require that HM is measurable, and the statement that "it's likely" means that p(H) is larger than most other measurable subsets of M (and often we mean something stronger, like p(H) > 0.5). But we haven't yet specified in our model what p(H) means. This is where the difference between Frequentism and Bayesianism matters. A Frequentist would say that the probability is a property of the hypothesis space, and noticing H doesn't change that. (I'm honestly not sure how a Frequentist thinks about iterating over a hypothesis space to suggest that H in fact would occur at a frequency of p(H) in the limit - maybe by considering the frequency in counterfactual worlds?) A Bayesian, by contrast, will say that p(H) is their current confidence that H is the right hypothesis.

What I'm suggesting, in essence, is that figuring out which hypothesis HM is worth testing is equivalent to moving from p to p' in the space of probability measures on M in a way that causes p'(H) > p(H). This is coming from using a Bayesian model of what p is.

Of course, if you're using a Frequentist model of p, then "most likely hypothesis" actually refers to a property of the hypothesis space - though I'm not sure how you would find out the frequency at which hypotheses turn out to be true the way you figure out the frequency at which a coin comes up heads. But that could just be my not being as familiar thinking in terms of the Frequentist model.

I'll briefly note that although I find the Bayesian model more coherent with my sense of how the world works on a day-by-day basis, I think the Frequentist model makes more sense when thinking about quantum physics. The type of randomness we find there isn't just about confidence, but is in fact a property of the quantum phenomena in question. In this case a well-calibrated Bayesian has to give a lot of probability mass to the hypothesis that there is a "true probability" in some quantum phenomena, which makes sense if we switch the model of p to be Frequentist.

But in short:

Yes, there's a difference.

And things like "probability" and "belief" and "evidence" mean different things depending on what model you use.

What I'm saying is that this shouldn't change your actual beliefs -- it will flush out some stale caching, or at best identify an inconsistent belief, including empirical data that you haven't fully updated on. But it does not, by itself, constitute evidence.

Yep, we disagree.

I think the disagreement is on two fronts. One is based on using different models of probability, which is basically not an interesting disagreement. (Arguing over which definition to use isn't going to make either of us smarter.) But I think the other is substantive. I'll focus on that.

In short, I think you underestimate the power of noticing implications of known facts. I think that if you look at a few common or well-known examples of incomplete deduction, it becomes pretty clear that figuring out how to finish thinking would be intensely powerful:

  • Many people make resolutions to exercise, be nicer, eat more vegetables, etc. And while making those resolutions, they often really think they mean it this time. And yet, there's often a voice of doubt in the back of the mind, as though saying "Come on. You know this won't work." But people still quite often spend a bunch of time and money trying to follow through on their new resolution - often failing for reasons that they kind of already knew would happen (and yet often feeling guilty for not sticking to their plan!).
  • Religious or ideological deconversion often comes from letting in facts that are already known. E.g., I used to believe that the results of parapsychological research suggested some really important things about how to survive after physical death. I knew all the pieces of info that finally changed my mind months before my mind actually changed. I had even done experiments to test my hypotheses and it still took months. I'm under the impression that this is normal.
  • Most people reading this already know that if they put a ton of work into emptying their email inbox, they'll feel good for a little while, and then it'll fill up again, complete with the sense of guilt for not keeping up with it. And yet, somehow, it always feels like the right thing to do to go on an inbox-emptying flurry, and then get around to addressing the root cause "later" or maybe try things that will fail after a month or two. This is an agonizingly predictable cycle. (Of course, this isn't how it goes for everyone, but it's common enough that well over half the people who attend CFAR workshops seem to relate to it.)
  • Most of Einstein's work in raising special relativity to consideration consisted of saying "Let's take the Michelson-Morley result at face value and see where it goes." Note that he is now considered the archetypal example of a brilliant person primarily for his ability to highlight worthy hypotheses via running with the implications of what is already known or supposed.
  • Ignaz Semmelweis found that hand-washing dramatically reduced mortality in important cases in hospitals. He was ignored, criticized, and committed to an insane asylum where guards beat him to death. At a cultural level, the fact that whether Semmelweis was right was (a) testable and (b) independent of opinion failed to propagate until after Louis Pasteur gave the medical community justification to believe that hand-washing could matter. This is a horrendous embarrassment, and thousands of people died unnecessarily because of a cultural inability to finish thinking. (Note that this also honors the need for empiricism - but the point here is that the ability to finish thinking was a prerequisite for empiricism mattering in this case.)

I could keep going. Hopefully you could too.

But my point is this:

Please note that there's a baby in that bathwater you're condemning as dirty.

Comment author: ESRogs 25 May 2015 03:22:51AM *  4 points [-]

But the argument itself is insufficient for drawing conclusions.

This seems like it would be true only if you'd already propagated all logical consequences of all observations you've made. But an argument can help me to propagate. Which means it can make me update my beliefs.

For example, is 3339799 a prime number?

One ought to assign some prior probability to it being a prime. A naive estimate might say, well, there are two options, so let's assign it 50% probability.

You could also make a more sophisticated argument about the distribution of prime numbers spreading out as you go towards infinity, and given that only 25 of the first 100 numbers are prime, the chance that a randomly selected number in the millions should be prime is less than 25% and probably much lower.

I claim that in a case like this it is totally valid to update your beliefs on the basis of an argument. No additional empirical test required before updating.

Do you agree?

Comment author: Vladimir_Nesov 24 May 2015 10:36:34AM *  4 points [-]

[...] this shouldn't change your actual beliefs [...] it does not, by itself, constitute evidence [...] the argument itself is insufficient for drawing conclusions. Even if the hypothesis is itself hard to test.

Is that a conclusion or a hypothesis? I don't believe there is a fundamental distinction between "actual beliefs", "conclusions" and "hypotheses". What should it take to change my beliefs about this?

Comment author: John_Maxwell_IV 24 May 2015 05:29:16AM 5 points [-]

People select hypotheses for testing because they have previously weakly updated in the direction of them being true. Seeing empirical data produces a later, stronger update.

Comment author: RobbBB 24 May 2015 08:58:00PM *  4 points [-]

Except that when the hypothesis space is large, people test hypotheses because they strongly updated in the direction of them being true, and seeing empirical data produces a later, weaker update. Where an example of 'strongly updating' could be going from 9,999,999:1 odds against a hypothesis to 99:1 odds, and an example of 'weakly updating' could be going from 99:1 odds against the hypothesis to 1:99. The former update requires about 20 bits of evidence, while the latter update requires about 10 bits of evidence.

Comment author: John_Maxwell_IV 25 May 2015 12:07:52PM 1 point [-]

Interesting point. I guess my intuitive notion of a "strong update" has to do with absolute probability mass allocation rather than bits of evidence (probability mass is what affects behavior?), but that's probably not a disagreement worth hashing out.

Comment author: Valentine 24 May 2015 07:36:08PM 1 point [-]

I like your way of saying it. It's much more efficient than mine!

Comment author: John_Maxwell_IV 25 May 2015 11:59:27AM *  2 points [-]

Thanks! Paul Graham is my hero when it comes to writing and I try to pack ideas as tightly as possible. (I recently reread this essay of his and got amazed by how many ideas it contains; I think it has more intellectual content than most published nonfiction books, in just 10 pages or so. I guess the downside of this style is that readers may not go slow enough to fully absorb all the ideas. Anyway, I'm convinced that Paul Graham is the Ben Franklin of our era.)

Comment author: John_Maxwell_IV 23 May 2015 05:17:58PM 7 points [-]

Thanks for the response.

I have a feeling your true rejection runs deeper than you're describing. You cite a thought experiment of Einstein's as being useful and correct. You explain that Less Wrong relies on thought experiments too heavily. You suggest that Less Wrong should lean heavier on data from the real world. But the single data point you cite on the question of thought experiments indicates that they are useful and correct. It seems like your argument fails by its own standard.

I think the reliability of thought experiments is a tricky question to resolve. I think we might as well expand the category of thought experiments to "any reasoning about the world that isn't reasoning directly from data". When I think about the reliability of this reasoning, my immediate thought is that I expect some people to be much better at it than others. In fact, I think being good at this sort of reasoning is almost exactly the same as being intelligent. Reasoning directly from data is like looking up the answers in the back of the book.

This leaves us with two broad positions: the “humans are dumb/the world is tricky” position that the only way we can ever get anywhere is through constant experimentation, and the “humans are smart/the world is understandable” position that we can usefully make predictions based on limited data.

I think these positions are too broad to be useful. It depends a lot on the humans, and it depends a lot on the aspect of the world being studied. Reasoning from first principles works better in physics than in medicine; in that sense, medicine is a trickier subject to study.

If the tricky world hypothesis is true for the questions MIRI is investigating, or the MIRI team is too dumb, I could see the sort of empirical investigations you propose as being the right approach: they don’t really answer the most important questions we want answered, but there probably isn’t a way for MIRI to answer those questions anyway, so might as well answer the questions that are answerable and see if the results lead anywhere.

Anyway, I think a lot of the value of many LW posts is in finding useful ideas that are also very general. (Paul Graham's description of what philosophy done right looks like.) Very general ideas are harder to test, because they cut across domains. The reason I like the many citations in Thinking Fast and Slow is that I expect the general ideas it presents to be more reliably true because they're informed by at least some experimental data. But general, useful ideas can be so useful that I don't mind taking the time to read about them even if they're not informed by lots of experimental data. Specifically, having lots of general, useful ideas that are also correct (e.g. knowing when and how to add numbers) makes you more intelligent according to my definition above. And I consider myself intelligent enough to be able to tell apart the true general, useful ideas from the false ones through my own reasoning and experience at least somewhat reliably.

Broadly speaking, I think Less Wrong is counterproductive if specific general, useful ideas it promotes are false. (It’s hard to imagine how it could be counterproductive if they were true.) And at that point we’re talking about whether specific posts are true or false. Lukeprog has this list of points of agreement between the sequences and mainstream academia, which causes me to update in the direction of those points of agreement being true.

I think you're being overly hard on the AI box experiment. It's obviously testing something. It’d be great if we could fork the universe, import a superintelligence, set up a bunch of realistic safeguards, and empirically determine how things played out. But that’s not practical. We did manage to find an experiment that might shed some light on the scenario, but the experiment uses a smart human instead of a superintelligence and a single gatekeeper instead of a more elaborate set of controls. It seems to me that you aren't following your own standard: you preach the value of empiricism and then throw out some of the only data points available, for theoretical reasons, without producing any better data. I agree that it’s some pretty weak data, but it seems better to think about it than throw it out and just believe whatever we like, and I think weak data is about as well as you’re going to do in this domain.

Comment author: [deleted] 23 May 2015 08:01:51PM *  4 points [-]

You cite a thought experiment of Einstein's as being useful and correct.

I cite a thought experiment of Einstein's as being useful but insufficient. It was not correct until observation matched anticipation. I called out Einstein's thought experiment as being a useful pedagogical technique, but not an example of how to arrive at truth. Do you see the difference?

I think you're being overly hard on the AI box experiment. It's obviously testing something.

No, this is not obvious to me. Other than the ability of two humans to outwit each other within the confines of strict enforcement of arbitrarily selected rules, what is it testing, exactly? And what does that thing being tested have to do with realistic AIs and boxes anyway?

Comment author: RobbBB 23 May 2015 11:51:31PM *  8 points [-]

I called out Einstein's thought experiment as being a useful pedagogical technique, but not an example of how to arrive at truth.

What's your model of how Einstein in fact arrived at truth, if not via a method that is "an example of how to arrive at truth"? It's obvious the method has to work to some extent, because Einstein couldn't have arrived at a correct view by chance. Is your view that Einstein should have updated less from whatever reasoning process he used to pick out that hypothesis from the space of hypotheses, than from the earliest empirical tests of that hypothesis, contra Einstein's Arrogance?

Or is your view that, while Einstein may technically have gone through a process like that, no one should assume they are in fact Einstein -- i.e., Einstein's capabilities are so rare, or his methods are so unreliable (not literally at the level of chance, but, say, at the level of 1000-to-1 odds of working), that by default you should harshly discount any felt sense that your untested hypothesis is already extremely well-supported?

Or perhaps you should harshly discount it until you have meta-evidence, in the form of a track record of successfully predicting which untested hypotheses will turn out to be correct.

Other than the ability of two humans to outwit each other within the confines of strict enforcement of arbitrarily selected rules, what is it testing, exactly? And what does that thing being tested have to do with realistic AIs and boxes anyway?

The AI box experiment is a response to the claim 'superintelligences are easy to box, because no level of competence at social engineering would suffice for letting an agent talk its way out of a box'. It functions as an existence proof; if a human level of social competence is already sufficient to talk one's way out of a box with nonzero frequency, then we can't dismiss risk from superhuman levels of social competence.

If you think the claim Eliezer was responding to is silly on priors, or just not relevant (because it would be easy to assess an AI's social competence and/or prevent it from gaining such competence), then you won't be interested in that part of the conversation.

Comment author: Jiro 24 May 2015 03:27:29PM 1 point [-]

The AI box experiment only serves even as that if you assume that the AI box experiment sufficiently replicates the conditions that would actually be faced by someone with an AI in a box. Also, it only serves as such if it is otherwise a good experiment, but since we are not permitted to see the session transcripts for ourselves, we can't tell if it is a good experiment.

Comment author: RobbBB 24 May 2015 06:07:07PM 2 points [-]

Again, the AI box experiment is a response to the claim "superintelligences are easy to box, because no level of competence at social engineering would suffice for letting an agent talk its way out of a box". If you have some other reason to think that superintelligences are hard to box -- one that depends on a relevant difference between the experiment and a realistic AI scenario -- then feel free to bring that idea up. But this constitutes a change of topic, not an objection to the experiment.

since we are not permitted to see the session transcripts for ourselves, we can't tell if it is a good experiment.

I mean, the experiment's been replicated multiple times. And you already know the reasons the transcripts were left private. I understand assigning a bit less weight to the evidence because you can't examine it in detail, but the hypothesis that there's a conspiracy to fake all of these experiments isn't likely.

Comment author: Jiro 24 May 2015 11:18:27PM *  2 points [-]

If you have some other reason to think that superintelligences are hard to box -- one that depends on a relevant difference between the experiment and a realistic AI scenario -- then feel free to bring that idea up.

Not all relevant differences between an experiment and an actual AI scenario can be accurately characterized as "reason to think that superintelligences are hard to box". For instance, imagine an experiment with no gatekeeper or AI party at all, where the result of the experiment depends on flipping a coin to decide whether the AI gets out. That experiment is very different from a realistic AI scenario, but one need not have a reason to believe that intelligences are hard to box--or even hold any opinion at all on whether intelligences are hard to box--to object to the experimental design.

For the AI box experiment as stated, one of the biggest flaws is that the gatekeeper is required to stay engaged with the AI and can't ignore it. This allows the AI to win by either verbally abusing the gatekeeper to the extent that he doesn't want to stay around any more, or by overwhelming the gatekeeper with lengthy arguments that take time or outside assistance to analyze. These situations would not be a win for an actual AI in a box.

I mean, the experiment's been replicated multiple times. And you already know the reasons the transcripts were left private. I understand assigning a bit less weight to the evidence because you can't examine it in detail, but the hypothesis that there's a conspiracy to fake all of these experiments isn't likely.

Refusing to release the transcripts causes other problems than just hiding fakery. If the experiment is flawed in some way, for instance, it could hide that--and it would be foolish to demand that everyone name possible flaws one by one and ask you "does this have flaw A?", "does this have flaw B?", etc. in order to determine whether the experiment has any flaws. There are also cases where whether something is a flaw is an opinion that can be argued, and it might be that someone else would consider a flaw something that the experimenter doesn't.

Besides, in a real boxed AI situation, it's likely that gatekeepers will be tested on AI-box experiments and will be given transcripts of experiment sessions to better prepare them for the real AI. An experiment that simulates an AI boxing should likewise have participants be able to read other sessions.

Comment author: John_Maxwell_IV 27 May 2015 12:23:56PM 3 points [-]

BTW, I realized there's something else I agree with you on that's probably worth mentioning:

Eliezer in particular, I think, is indeed overconfident in his ability to reason things out from first principles. For example, I think he was overconfident in AI foom (see especially the bit at the end of that essay). And even if he's calibrated his ability correctly, it's totally possible that others who don't have the intelligence/rationality he does could pick up the "confident reasoning from first principles" meme and it would be detrimental to them.

That said, he's definitely a smart guy and I'd want to do more thinking and research before making a confident judgement. What I said is just my current estimate.

Insofar as I object to your post, I'm objecting to the idea that empiricism is the be-all and end-all of rationality tools. I'm inclined to think that philosophy (as described in Paul Graham's essay) is useful and worth learning about and developing.

Comment author: negamuhia 14 August 2015 11:55:54AM *  0 points [-]

See my above point about how reasoning by analogy at its best is reasoning from a weak reference class. (Do people think this is worth a toplevel post?)

Yes, I do. Intuitively, this seems correct. But I'd still like to see you expound on the idea.

Comment author: John_Maxwell_IV 26 June 2015 01:26:42AM *  0 points [-]

BTW, this discussion has some interesting parallels to mine & Mark's.

Comment author: V_V 21 May 2015 11:26:59PM *  12 points [-]

I generally agree with your position on the Sequences, but it seems to me that it is possible to hang around this website and have meaningful discussions without worshiping the Sequences or Eliezer Yudkowsky. At least it works for me.
As for being a highly involved/high status member of the community, especially the offline one, I don't know.

Anyway, regarding the point about super-intelligence that you raised, I charitably interpret the position of the AI-risk advocates not as the claim that super-intelligence would be in principle outside the scope of human scientific inquiry, but as the claim that a super-intelligent agent would be more efficient at understanding humans that humans would be at understanding it, giving the super-intelligent agent and edge over humans.

I think that the AI-risk advocates tend to exaggerate various elements of their analysis: they probably underestimate time to human-level AI and time to super-human AI, they may overestimate the speed and upper bounds to recursive self-improvement (their core arguments based on exponential growth seem, at best, unsupported).

Moreover, it seems that they tend to conflate super-intelligence with a sort of near-omniscience:
They seem to assume that a super-intelligent agent will be a near-optimal Bayesian reasoner with an extremely strong prior that will allow it to gain a very accurate model of the world, including all the nuances of human psychology, from a very small amount of observational evidence and little or no interventional experiments. Recent discussion here.
Maybe this is the community bias that you were talking about, the over-reliance on abstract thought rather than evidence, projected on an hypothetical future AI.
It seems dubious to me that this kind of extreme inference is even physically possible, and if it is, we are certainly not anywhere close to implementing it. All the recent advances in machine learning, for instance, rely on processing very large datasets.

Anyway, as much as they exaggerate the magnitude and urgency of the issue, I think that the AI-risk advocates have a point when they claim that keeping a system much intelligent than ourselves under control would be a non-trivial problem.

Comment author: [deleted] 22 May 2015 12:20:29AM *  9 points [-]

Maybe this is the community bias that you were talking to, the over-reliance on abstract thought rather than evidence, projected on an hypothetical future AI.

You nailed it. (Your other points too.)

The claim [is] that a super-intelligent agent would be more efficient at understanding humans that humans would be at understanding it, giving the super-intelligent agent['s] edge over humans.

The problem here is that intelligence is not some linear scale, even general intelligence. We human beings are insanely optimized for social intelligence in a way that is not easy for a machine to learn to replicate, especially without detection. It is possible for a general AI to be powerful enough to provide meaningful acceleration of molecular nanotechnology and medical science research whilst being utterly befuddled by social conventions and generally how humans think, simply because it was not programmed for social intelligence.

Anyway, as much as they exaggerate the magnitude and urgency of the issue, I think that the AI-risk advocates have a point when they claim that keeping a system much intelligent than ourselves under control would be a non-trivial problem.

There is however a substantial difference between a non-trivial problem and an impossible problem. Non-trivial we can work with. I solve non-trivial problems for a living. You solve a non-trivial problem by hacking at it repeatedly until it breaks into components that are themselves well understood enough to be trivial problems. It takes a lot of work, and the solution is to simply to do a lot of work.

But in my experience the AI-risk advocates claim that safe / controlled UFAI is an impossibility. You can't solve an impossibility! What's more, in that frame of mind any work done towards making AGI is risk-increasing. Thus people are actively persuaded to NOT work on artificial intelligence, and instead work of fields of basic mathematics which is at this time too basic or speculative to say for certain whether it would have a part in making a safe or controllable AGI.

So smart people who could be contributing to an AGI project, are now off fiddling with basic mathematics research on chalkboards instead. That is, in the view of someone who believes safe / controllable UFAI is non-trivial but possible mechanism to accelerate the arrival of life-saving anti-aging technologies, a humanitarian disaster.

Comment author: V_V 22 May 2015 12:13:46PM 5 points [-]

The problem here is that intelligence is not some linear scale, even general intelligence. We human beings are insanely optimized for social intelligence in a way that is not easy for a machine to learn to replicate, especially without detection. It is possible for a general AI to be powerful enough to provide meaningful acceleration of molecular nanotechnology and medical science research whilst being utterly befuddled by social conventions and generally how humans think, simply because it was not programmed for social intelligence.

Agree.

I think that since many AI risk advocates have little or no experience in computer science and specifically AI research, they tend to anthropomorphize AI to some extent. They get that an AI could have goals different than human goals but they seem to think that it's intelligence would be more or less like human intelligence, only faster and with more memory. In particular they assume that an AI will easily develop a theory of mind and social intelligence from little human interaction.

But in my experience the AI-risk advocates claim that safe / controlled UFAI is an impossibility.

I think they used to claim that safe AGI was pretty much an impossibility unless they were the ones who built it, so gib monies plox!
Anyway, it seems that in recent times they have taken a somewhat less heavy handed approach.

Comment author: RobbBB 22 May 2015 01:00:01AM *  12 points [-]

I think that the AI-risk advocates tend to exaggerate various elements of their analysis: they probably underestimate time to human-level AI and time to super-human AI

It's worth keeping in mind that AI-risk advocates tend to be less confident that AGI is nigh than the top-cited scientists within AI are. People I know at MIRI and FHI are worried about AGI because it looks like a technology that's many decades away, but one where associated safety technologies are even more decades away.

That's consistent with the possibility that your criticism could turn out to be right. It could be that we're less wrong than others on this metric and yet still very badly wrong in absolute terms. To make a strong prediction in this area is to claim to already have a pretty good computational understanding of how general intelligence works.

Moreover, it seems that they tend to conflate super-intelligence with a sort of near-omniscience: They seem to assume that a super-intelligent agent will be a near-optimal Bayesian reasoner

Can you give an example of a statement by a MIRI researcher that is better predicted by 'X is speaking of the AI as a near-optimal Bayesian' than by 'X is speaking of the AI as an agent that's as much smarter than humans as humans are smarter than chimpanzees, but is still nowhere near optimal'? (Or 'an agent that's as much smarter than humans as humans are smarter than dogs'...) I'm not seeing why saying 'Bob the AI could be 100x more powerful than a human', for example, commits one to a view about how close Bob is to optimal.

Comment author: V_V 22 May 2015 12:33:24PM *  2 points [-]

It's worth keeping in mind that AI-risk advocates tend to be less confident that AGI is nigh than the top-cited scientists within AI are.

Cite? I think I remember Eliezer Yudkowsky and Luke Muehlhauser going for the usual "20 years from now" (in 2009) time to AGI prediction.
By contrast Andrew Ng says "Maybe hundreds of years from now, maybe thousands of years from now".

Can you give an example of a statement by a MIRI research that is better predicted by 'X is speaking of the AI as a near-optimal Bayesian' than by 'X is speaking of the AI as an agent that's as much smarter than humans as humans are smarter than chimpanzees, but is still nowhere near optimal'?

Maybe they are not explicitly saying "near-optimal", but it seems to me that they are using models like Solomonoff Induction and AIXI as intuition pumps, and they are getting these beliefs of extreme intelligence from there.
Anyway, do you disagree that MIRI in general expects the kind of low-data, low-experimentation, prior-driven learning that I talked about to be practically possible?

Comment author: RobbBB 22 May 2015 07:39:19PM *  8 points [-]

Cite?

Müller and Bostrom's 2014 'Future progress in artificial intelligence: A survey of expert opinion' surveyed the 100 top-cited living authors in Microsoft Academic Search's "artificial intelligence" category, asking the question:

Define a "high-level machine intelligence" (HLMI) as one that can carry out most human professions at least as well as a typical human. [...] For the purposes of this question, assume that human scientific activity continues without major negative disruption. By what year would you see a (10% / 50% / 90%) probability for such HLMI to exist?

29 of the authors responded. Their median answer was a 10% probability of HLMI by 2024, a 50% probability of HLMI by 2050, and a 90% probability by 2070.

(This excludes how many said "never"; I can't find info on whether any of the authors gave that answer, but in pooled results that also include 141 people from surveys of a "Philosophy and Theory of AI" conference, an "Artificial General Intelligence" conference, an "Impacts and Risks of Artificial General Intelligence" conference, and members of the Greek Association for Artificial Intelligence, 1.2% of the people in the overall pool (2 / 170) said we'd "never" have a 10% chance of HLMI, 4.1% (7 / 170) said "never" for 50% probability, and 16.5% (28 / 170) said "never" for 90%.)

In Bostrom's Superintelligence (pp. 19-20), he cites the pooled results:

The combined sample gave the following (median) estimate: 10% probability of HLMI by 2022, 50% probability by 2040, and 90% probability by 2075. [...]

These numbers should be taken with some grains of salt: sample sizes are quite small and not necessarily representative of the general expert population. They are, however, in concordance with results from other surveys.

The survey results are also in line with some recently published interviews with about two-dozen researchers in AI-related fields. For example, Nils Nilsson has spent a long and productive career working on problems in search, planning, knowledge representation, and robotics, he has authored textbooks in artificial intelligence; and he has recently completed the most comprehensive history of the field written to date. When asked about arrival dates for [AI able to perform around 80% of jobs as well or better than humans perform], he offered the following opinion: 10% chance: 2030[;] 50% chance: 2050[;] 90% chance: 2100[.]

Judging from the published interview transcripts, Professor Nilsson's probability distribution appears to be quite representative of many experts in the area--though again it must be emphasized that there is a wide spread of opinion: there are practitioners who are substantially more boosterish, confidently expecting HLMI in the 2020-40 range, and others who are confident either that it will never happen or that it is indefinitely far off. In addition, some interviewees feel that the notion of a "human level" of artificial intelligence is ill-defined or misleading, or are for other reasons reluctant to go on record with a quantitative prediction.

My own view is that the median numbers reported in the expert survey do not have enough probability mass on later arrival dates. A 10% probability of HLMI not having been developed by 2075 or even 2100 (after conditionalizing on "human scientific activity continuing without major negative disruption") seems too low.

Luke has pretty much the same view as Bostrom. I don't know as much about Eliezer's views, but the last time I talked with him about this (in 2014), he didn't expect AGI to be here in 20 years. I think a pretty widely accepted view at MIRI and FHI is Luke's: "We can’t be confident AI will come in the next 30 years, and we can’t be confident it’ll take more than 100 years, and anyone who is confident of either claim is pretending to know too much."

Comment author: V_V 22 May 2015 07:59:25PM 5 points [-]

Thanks!

Comment author: jacob_cannell 15 June 2015 07:31:05AM 1 point [-]

Of course, there is a huge problem with expert surveys - at the meta-level they have a very poor predictive track record. There is the famous example that Stuart Russell likes to cite, where rutherford said "anyone who looked for a source of power in the transformation of the atoms was talking moonshine" - a day before leo szilard created a successful fission chain reaction. There is also the similar example of the Wright Brothers - some unknown guys without credentials claim to have cracked aviation when recognized experts like Langley have just failed in a major way and respected scientists such as Lord Kelvin claim the whole thing is impossible. The wright brothers then report their first successful manned flight and no newspaper will even publish it.

Comment author: RobbBB 22 May 2015 08:31:03PM *  8 points [-]

Maybe they are not explicitly saying "near-optimal", but it seems to me that they are using models like Solomonoff Induction and AIXI as intuition pumps, and they are getting these beliefs of extreme intelligence from there.

I don't think anyone at MIRI arrived at worries like 'AI might be able to deceive their programmers' or 'AI might be able to design powerful pathogens' by staring at the equation for AIXI or AIXItl. AIXI is a useful idea because it's well-specified enough to let us have conversations that are more than just 'here are my vague intuitions vs. your vague-intuitions'; it's math that isn't quite the right math to directly answer our questions, but at least gets us outside of our own heads, in much the same way that an empirical study can be useful even if it can't directly answer our questions.

Investigating mathematical and scientific problems that are near to the philosophical problems we care about is a good idea, when we still don't understand the philosophical problem well enough to directly formalize or test it, because it serves as a point of contact with a domain that isn't just 'more vague human intuitions'. Historically this has often been a good way to make intellectual progress, though it's important to keep in mind just how limited our results are.

AIXI is also useful because the problems we couldn't solve even if we (impossibly) had recourse to AIXI often overlap with the problems where our theoretical understanding of intelligence is especially lacking, and where we may therefore want to concentrate our early research efforts.

The idea that AI will have various 'superpowers' comes more from:

  • (a) the thought that humans often vary a lot in how much they exhibit the power (without appearing to vary all that much in hardware);

  • (b) the thought that human brains have known hardware limitations, where existing machines (and a fortiori machines 50 or 100 years down the line) can surpass humans by many orders of magnitude; and

  • (c) the thought that humans have many unnecessary software limitations, including cases where machines currently outperform humans. There's also no special reason to expect evolution's first stab at technology-capable intelligence to have stumbled on all the best possible software ideas.

A more common intuition pump is to simply note that limitations in human brains suggest speed superintelligence is possible, and it's relatively easy to imagine speed superintelligence allowing one to perform extraordinary feats without imagining other, less well-understood forms of cognitive achievement. Rates of cultural and technological progress in human societies are a better (though still very imperfect) source of data than AIXI about how much improvement intelligence makes possible.

Anyway, do you disagree that MIRI in general expects the kind of low-data, low-experimentation, prior-driven learning that I talked about to be practically possible?

This should be possible to some extent, especially when it comes to progress in mathematics. We should also distinguish software experiments from physical experiments, since it's a lot harder to keep an AI from performing the former, and the former are much easier to speed up in proportion to speed-ups in the experimenter's ability to analyze results.

I don't think there's any specific consensus view about how much progress requires waiting for results from slow experiments. I frequently hear Luke raise the possibility that slow natural processes could limit rates of self-improvement in AI, but I don't know whether he considers that a major consideration or a minor one.

Comment author: V_V 24 May 2015 09:09:59PM *  3 points [-]

I don't think anyone at MIRI arrived at worries like 'AI might be able to deceive their programmers' or 'AI might be able to design powerful pathogens' by staring at the equation for AIXI or AIXItl.

In his quantum physics sequence, where he constantly talks (rants, actually) about Solomonoff Induction, Yudkowsky writes:

A Bayesian superintelligence, hooked up to a webcam, would invent General Relativity as a hypothesis—perhaps not the dominant hypothesis, compared to Newtonian mechanics, but still a hypothesis under direct consideration—by the time it had seen the third frame of a falling apple. It might guess it from the first frame, if it saw the statics of a bent blade of grass.

Anna Salamon also mentions AIXI when discussing the feasibility of super-intelligence.

Mind you, I'm not saying that AIXI is not an interesting and possibly theoretically useful model, my objection is that MIRI people seem to have used it to set a reference class for their intuitions about super-intelligence.

Rates of cultural and technological progress in human societies are a better (though still very imperfect) source of data than AIXI about how much improvement intelligence makes possible.

Extrapolation is always an epistemically questionable endeavor.
Intelligence is intrinsically limited by how predictable the world is. Efficiency (time complexity/space complexity/energy complexity/etc.) of algorithms for any computational task is bounded. Hardware resources also have physical limits.

This doesn't mean that given our current understanding we can claim that human-level intelligence is an upper bound. That would be most likely false. But there is no particular reason to assume that the physically attainable bound will be enormously higher than human-level. The more extreme the scenario, the less probability we should assign to it, reasonably according to a light-tailed distribution.

This should be possible to some extent, especially when it comes to progress in mathematics.

Ok, but my point is that it has not been established that progress in mathematics will automatically grant an AI "superpowers" in the physical world.
And I'd even say that even superpowers by raw cognitive power alone are questionable. Theorem proving can be sped up, but there is more to math than theorem proving.

Comment author: RobbBB 24 May 2015 10:13:36PM *  2 points [-]

I think Eliezer mostly just used "Bayesian superintelligence" as a synonym for "superintelligence." The "Bayesian" is there to emphasize the fact that he has Bayes-optimality as a background idea in his model of what-makes-cognition-work and what-makes-some-cognition-work-better-than-other-kinds, but Eliezer thought AI could take over the world long before he knew about AIXI or thought Bayesian models of cognition were important.

I don't know as much about Anna's views. Maybe she does assign more weight to AIXI as a source of data; the example you cited supports that. Though since she immediately follows up her AIXI example with "AIXI is a theoretical toy. How plausible are smarter systems in the real world?" and proceeds to cite some of the examples I mentioned above, I'm going to guess she isn't getting most of her intuitions about superintelligence from AIXI either.

there is no particular reason to assume that the physically attainable bound will be enormously higher than human-level. The more extreme the scenario, the less probability we should assign to it, reasonably according to a light-tailed distribution.

I think our disagreement is about what counts as "extreme" or "extraordinary", in the "extraordinary claims require extraordinary evidence" sense.

If I'm understanding your perspective, you think we should assume at the outset that humans are about halfway between 'minimal intelligence' and 'maximal intelligence' -- a very uninformed prior -- and we should then update only very weakly in the direction of 'humans are closer to the minimum than to the maximum'. Claiming that there's plenty of room above us is an 'extreme' claim relative to that uninformed prior, so the epistemically modest thing to do is to stick pretty close the assumption that humans are 'average', that the range of intelligence exhibited in humans with different cognitive abilities and disabilities represents a non-tiny portion of the range of physically possible intelligence.

My view is that we should already have updated strongly away from the 'humans are average' prior as soon as we acquired information about how humans arose -- through evolution, a process that has computational resources and perseverance but no engineering ingenuity, constructing our current advantages out of chimpanzees' over the course of just 250,000 generations. At that point we are no longer a randomly selected mind; our prior is swamped by all the rich information we have about ourselves as an evolved species, enhanced by culture but not by any deliberate genetic engineering or fine-grained neurosurgery. We have no more reason in the year 2015 to think that we represent 1/10 (or for that matter 1/10,000) of what is cognitively possible, than a person in the year 1850 would have reason to think that the fastest possible flying machine was only 10x as fast as the fastest bird, or that the most powerful possible bomb was only 10x as strong as the most powerful existing bomb.

Our knowledge of physics and of present technological capabilities, though still extremely flawed as data to generalize from, is rich enough to strongly shift a flat prior (informed by nothing more than 'this bird / bomb / brain exists and the rest of the universe is unknown to me'). So although we should not be confident of any specific prediction, 'humans are near the practical intelligence maximum' is the extreme view that requires a lot of evidence to move toward. Superintelligence is unusual relative to what we can directly observe, not 'extreme' in an evidential sense.

Comment author: jacob_cannell 15 June 2015 07:19:17AM 0 points [-]

We should also distinguish software experiments from physical experiments, since it's a lot harder to keep an AI from performing the former, and the former are much easier to speed up in proportion to speed-ups in the experimenter's ability to analyze results.

This is actually completely untrue and is an example of a typical misconception about programming - which is far closer to engineering than math. Every time you compile a program, you are physically testing a theory exactly equivalent to building and testing a physical machine. Every single time you compile and run a program.

If you speedup an AI - by speeding up its mental algorithms or giving it more hardware, you actually slow down the subjective speed of the world and all other software systems in exact proportion. This has enormous consequences - some of which I explored here and here. Human brains operate at 1000hz or or less, which suggests that a near optimal (in terms of raw speed) human-level AGI could run at 1 million X time dilation. However that would effectively mean that the AGI's computers it had access to would be subjectively slower by 1 million times - so if it's compiling code for 10 GHZ CPUs, those subjectively run at 10 kilohertz.

Comment author: calef 21 May 2015 08:51:29PM *  9 points [-]

This argument is, however, nonsense. The human capacity for abstract reasoning over mathematical models is in principle a fully general intelligent behaviour, as the scientific revolution has shown: there is no aspect of the natural world which has remained beyond the reach of human understanding, once a sufficient amount of evidence is available. The wave-particle duality of quantum physics, or the 11-dimensional space of string theory may defy human intuition, i.e. our built-in intelligence. But we have proven ourselves perfectly capable of understanding the logical implications of models which employ them. We may not be able to build intuition for how a super-intelligence thinks. Maybe—that's not proven either. But even if that is so, we will be able to reason about its intelligent behaviour in advance, just like string theorists are able to reason about 11-dimensional space-time without using their evolutionarily derived intuitions at all.

This may be retreating to the motte's bailey, so to speak, but I don't think anyone seriously thinks that a superintelligence would be literally impossible to understand. The worry is that there will be such a huge gulf between how superintelligences reason versus how we reason that it would take prohibitively long to understand them.

I think a laptop is a good example. There probably isn't any single human on earth that knows how to build a modern laptop from scratch. There's are computer scientists that know how the operating system is put together--how the operating system is programmed, how memory is written and retrieved from the various buses; there are other computer scientists and electrical engineers who designed the chips themselves, who arrayed circuits efficiently to dissipate heat and optimize signal latency. Even further, there are material scientists and physicists who designed the transistors and chip fabrication processes, and so on.

So, as an individual human, I don't know what it's like to know everything about a laptop all at once in my head, at a glance. I can zoom in on an individual piece and learn about it, but I don't know all the nuances for each piece--just a sort of executive summary. The fundamental objects with which I can reason have a sort of characteristic size in mindspace--I can imagine 5, maybe 6 balls moving around with distinct trajectories (even then, I tend to group them into smaller subgroups). But I can't individually imagine a hundred (I could sit down and trace out the paths of a hundred balls individually, of course, but not all at once).

This is the sense in which a superintelligence could be "dangerously" unpredictable. If the fundamental structures it uses for reasoning greatly exceed a human's characteristic size of mindspace, it would be difficult to tease out its chain of logic. And this only gets worse the more intelligent it gets.

Now, I'll grant you that the lesswrong community likes to sweep under the rug the great competition of timescales and "size"scales that are going on here. It might be prohibitively difficult, for fundamental reasons, to move from working-mind-RAM of size 5 to size 10. It may be that artificial intelligence research progresses so slowly that we never even see an intelligence explosion--just a gently sloped intelligence rise over the next few millennia. But I do think it's a maybe not a mistake but certainly naiive to just proclaim, "Of course we'll be able to understand them, we are generalized reasoners!".

Edit: I should add that this is already a problem for, ironically, computer-assisted theorem proving. If a computer produces a 10,000,000 page "proof" of a mathematical theorem (i.e., something far longer than any human could check by hand), you're putting a huge amount of trust in the correctness of the theorem-proving-software itself.

Comment author: V_V 21 May 2015 10:11:12PM 7 points [-]

Edit: I should add that this is already a problem for, ironically, computer-assisted theorem proving. If a computer produces a 10,000,000 page "proof" of a mathematical theorem (i.e., something far longer than any human could check by hand), you're putting a huge amount of trust in the correctness of the theorem-proving-software itself.

No, you just need to trust a proof-checking program, which can be quite small and simple, in contrast with the theorem proving program, which can be arbitrarily complex and obscure.

Comment author: Sable 22 May 2015 01:46:12AM 2 points [-]

Isn't using a laptop as a metaphor exactly an example of

Most often reasoning by analogy?

I think one of the points trying to be made was that because we have this uncertainty about how a superintelligence would work, we can't accurately predict anything without more data.

So maybe the next step in AI should be to create an "Aquarium," a self-contained network with no actuators and no way to access the internet, but enough processing power to support a superintelligence. We then observe what that superintelligence does in the aquarium before deciding how to resolve further uncertainties.

Comment author: roystgnr 22 May 2015 10:59:44PM 4 points [-]

Isn't using a laptop as a metaphor exactly an example

The sentence could have stopped there. If someone makes a claim like "∀ x, p(x)", it is entirely valid to disprove it via "~p(y)", and it is not valid to complain that the first proposition is general but the second is specific.

Moving from the general to the specific myself, that laptop example is perfect. It is utterly baffling to me that people can insist we will be able to safely reason about the safety of AGI when we have yet to do so much as produce a consumer operating system that is safe from remote exploits or crashes. Are Microsoft employees uniquely incapable of "fully general intelligent behavior"? Are the OpenSSL developers especially imperfectly "capable of understanding the logical implications of models"?

If you argue that it is "nonsense" to believe that humans won't naturally understand the complex things they devise, then that argument fails to predict the present, much less the future. If you argue that it is "nonsense" to believe that humans can't eventually understand the complex things they devise after sufficient time and effort, then that's more defensible, but that argument is pro-FAI-research, not anti-.

Comment author: Manfred 22 May 2015 05:06:38AM 5 points [-]

There is a difference between argument by analogy and using an example. The relevant difference here is that examples illustrate arguments that are made separately, like how calef spent paragraphs 4 and 5 restating the arguments sans laptop.

If anything, the argument from analogy here is in the comparison between human working memory and computer RAM and a nebulous "size in mindspace," because it is used as an important part of the argument but is not supported separately. But don't fall for the fallacy fallacy - just because something isn't modus ponens doesn't mean it can't be Bayesian evidence.

Comment author: [deleted] 22 May 2015 07:05:09AM 2 points [-]

Isn't using a laptop as a metaphor exactly an example of "Most often reasoning by analogy"?

Precisely correct, thank you for catching that.

I think one of the points trying to be made was that because we have this uncertainty about how a superintelligence would work, we can't accurately predict anything without more data.

Also correct reading of my intent. The "aquarium" ides is basically what I have and would continue to advocate for: continue developing AGI technology within the confines of a safe experimental setup. By learning more about the types of programs which can perform limited general intelligence tasks in sandbox environments, we learn more about their various strengths and limitations in context, and from that experiance we can construct suitable safeguards for larger deployments.

Comment author: Sable 22 May 2015 01:55:58AM 6 points [-]

I'm relatively new here, so I have trouble seeing the same kinds of problems you do.

However, I can say that LessWrong does help me remember to apply the principles of rationality I've been trying to learn.

I'd also like to add that - much like writing a novel - the first draft rarely addresses all of the possible faults. LessWrong is one of (if not the first) community blogs devoted to "refining the art of human rationality." Of course we're going to get some things wrong.

What I really admire about this site, though, is that contrarian viewpoints end up being some of the most highly upvoted - people admire and discourse with dissenters here. So if you truly believe that LessWrong isn't the best use of your time, then I wish you the best with whatever efforts you pursue. But I think if you wrote a bit more on this subject and found a way to add it to the sequences, everyone would only thank you.

Comment author: [deleted] 23 May 2015 02:18:06PM *  3 points [-]

There's need for a rationality community. It just seems to me and with regard to certain issues that this one is a stick in the mud :\

Maybe a wiki would be a better structure than a communal blog.

Comment author: [deleted] 21 May 2015 08:46:19PM *  6 points [-]

We get instead definitive conclusions drawn from thought experiments only.

As a relatively new user here at LessWrong (and new to rationality) it is also curious to me that many here point me to articles written by Eliezer Yudkowsky to support their arguments. I have the feeling there is a general admiration for him and that some could be biased by that rather than approaching the different topics objectively.

Also, when I read the article about dissolving problems and how algorithms feel I didn't find any evidence that it is known exactly how neuron networks work to create these feelings.

That article was a good way of explaining how we might "feel" the existence of things and how to demystify them (like free will, time, ghosts, god, etc.) but I am not sure if the "extra dangling unit in the center" is something that we know exists or if it is another construct that was built to refute things by thought experiment rather than by empiric evidence.

Comment author: Dustin 21 May 2015 10:38:23PM 15 points [-]

it is also curious to me that many here point me to articles written by Eliezer Yudkowsky to support their arguments

It's been my experience that this is usually done to point to a longer and better-argued version of what the person wants to say rather than to say "here is proof of what I want to say".

I mean, if I agree with the argument made by EY about some subject, and EY has done a lot of work in making the argument, then I'm not going to just reword the argument, I'm just going to post a link.

The appropriate response is to engage the argument made in the EY argument as if it is the argument the person is making themselves.

Comment author: TheAncientGeek 22 May 2015 10:00:08AM 5 points [-]

It's been my experience that this is usually done to point to a longer and better-argued version of what the person wants to say rather than to say "here is proof of what I want to say".

Kind of. But its still done when there are unanswered criticisms in the comments section.

Comment author: Vaniver 21 May 2015 08:56:43PM 16 points [-]

On the other hand, de Grey and others who are primarily working on the scientific and/or engineering challenges of singularity and transhumanist technologies were far less likely to subject themselves to epistematic mistakes of significant consequences.

This part isn't clear to me. The researcher who goes into generic anti-cancer work, instead of SENS-style anti-aging work, probably has made an epistemic mistake with moderate consequences, because of basic replaceability arguments.

But to say that MIRI's approach to AGI safety is due to a philosophical mistake, and one with significant consequences, seems like it requires much stronger knowledge. Shooting very high instead of high is riskier, but not necessarily wronger.

Thankfully there is an institution that is doing that kind of work: the Future of Life institute (not MIRI).

I think you underestimate how much MIRI agrees with FLI.

Why they do not get more play in the effective altruism community is beyond me.

SENS is the second largest part of my charity budget, and I recommend it to my friends every year (on the obvious day to do so). My speculations on why EAs don't favor them more highly mostly have to do with the difficulty of measuring progress in medical research vs. fighting illnesses, and possibly also the specter of selfishness.

Comment author: Kaj_Sotala 22 May 2015 11:57:10AM *  15 points [-]

I think you underestimate how much MIRI agrees with FLI.

Agreed - or, at least, he underestimates how much FLI agrees with MIRI. This is pretty obvious e.g. in the references section of the technical agenda that was attached to FLI's open letter. Out of a total of 95 references:

  • Six are MIRI's technical reports that've only been published on their website: Vingean Reflection, Realistic World-Models, Value Learning, Aligning Superintelligence, Reasoning Under Logical Uncertainty, Toward Idealized Decision Theory
  • Five are written by MIRI's staff or Research Associates: Avoiding Unintended AI Behaviors, Ethical Artificial Intelligence, Self-Modeling Agents and Reward Generator Corruption, Problem Equilibirum in the Prisoner's Dilemma, Corrigibility,
  • Eight are ones that tend to agree with MIRI's stances and which have been cited in MIRI's work: Superintelligence, Superintelligent Will, Singularity A Philosophical Analysis, Speculations concerning the first ultraintelligent machine, The nature of self-improving AI, Space-Time Embedded Intelligence, FAI: the Physics Challenge, The Coming Technological Singularity

That's 19/95 (20%) references produced either directly by MIRI or people closely associated with them, or that have MIRI-compatible premises.

Comment author: [deleted] 23 May 2015 12:51:23PM *  4 points [-]

I think you and Vaniver both misunderstood my endorsement of FLI. I endorse them not because of their views on AI risk, which are in line MIRI's and entirely misguided in my opinion. But the important question is not what you believe, but what you do about it. Despite those views they are still willing to fund practical, evidence-based research into artificial intelligence, engaging with the existing community rather than needlessly trying to reinvent the field.

Comment author: Mirzhan_Irkegulov 21 May 2015 11:01:28PM *  6 points [-]

Yudkowsky obviously supports immortality. Quote from his letter on his brother's death:

If you object to the Machine Intelligence Research Institute then consider Dr. Aubrey de Grey's Methuselah Foundation, which hopes to defeat aging through biomedical engineering.

If SENS is not sufficiently promoted as a target for charity, I have no idea why is that, and I dispute that it's because of LW community's philosophical objections, unless somebody can convince me otherwise. BTW, EA community != LW community, so maybe lot's of Effective Altruists just don't consider immortality the same way the do malaria (cached thoughts etc).

Comment author: [deleted] 21 May 2015 11:31:40PM *  2 points [-]

If SENS is not sufficiently promoted as a target for charity, I have no idea why is that, and I dispute that it's because of LW community's philosophical objections, unless somebody can convince me otherwise.

To be clear this is not an intended implication. I'm aware that Yudkowsky supports SENS, and indeed my memory is fuzzy but it might have been though exactly the letter you quote that I first heard about SENS.

Comment author: [deleted] 21 May 2015 11:25:58PM 2 points [-]

I recommend it to my friends every year (on the obvious day to do so)

Just out of curiosity, what day is that? Both Christmas and April 15th came to mind.

Comment author: Vaniver 22 May 2015 02:12:38AM 14 points [-]

My birthday. It is both when one is supposed to be celebrating aging / one's continued survival, and when one receives extra attention from others.

Comment author: [deleted] 22 May 2015 02:24:49AM 8 points [-]

Oh that's a great idea. I'm going to start suggesting people who ask to donate to one of my favorite charities on my birthday. It beats saying I don't need anything which is what I currently do.

Comment author: Kaj_Sotala 22 May 2015 07:22:05AM 10 points [-]

Consider also doing an explicit birthday fundraiser. I did one on my most recent birthday and raised $500 for charitable causes.

Comment author: Mirzhan_Irkegulov 21 May 2015 09:11:27PM 27 points [-]

I understand “politics is the mind-killer” enough to not consider LW community as a tribe that I have to belong to, and I could easily turn away from LW and say “the Sequences and FAI is nonsense”, just like I turned away from various gurus and ideologies before. But I disagree with what you saying, not the Sequences or MIRI criticism, but with your evaluation of LW community and your unwillingness to engage anymore. Honestly, I'm upset that you suddenly stopped the reading group.

Despite Yudkowsky's obvious leanings, the Sequences are not about FAI, nor they are about Many-Worlds, Tegmark Mathematical Universe, Roko's Basilisk or whatever. They are first and foremost about how to not end up an idiot. They are about how to not become immune to criticism, they are about Human's Guide to Words, they are about System 1 and System 2.

I don't care about Many Worlds, FAI, Fun theory and Jeffreyssai stuff, but LW was the thing that stopped me from being a complete and utter idiot. Now I see that people I care about, due to not internalizing LW's simple truths, are being complete and utter idiots, with their death spirals, and tribal affiliations, and meaningless usage of words, and theories that don't predict shit, and it breaks my heart.

If you want to criticize LW for lack of actual instrumental rationality, you're not the first, Yvain did that in 2009, and he was right in understanding the problem, though he couldn't provide a solution either. I personally believe that combating akrasia is the most important task in the world, not FAI, because if a cure for akrasia could be found, we could train armies of superhuman-scientists, who then would solve cancer, nanotechnology and AI-risk. That's why reading modern cognitive sciences and CBT and neuroscience is probably more important than everything, at least that's what I think.

And here I am, somebody who wishes to be part of LW community, but also disagreeing, either conceptually or politically, with much of the LW memes. Yet you don't want to engage with me anymore. LW is not a monolith, where everybody follows Yudkowsky, it's the most contrarian (and thus mentally healthy) place I've ever seen on the Web.

LW is not end of it all, but the Sequences are the bare minimum that people require to be sane. Hey, some people through sheer study of maths and physics can develop correct epistemology, so they don't need the Sequences, but I wasn't, and many people aren't.

It's not about tribal things. If you had your own forum with lots of people, who share similar criticism of LW, hey, I'd go there and leave LW. But you don't have such forum, so by leaving LW you just leave people like me alone. What's the point of that? Do you really believe leaving LW like that is more utility, than trying to create an island within it?

Honestly, I even started thinking the only reason you wrote this post because you realized you're too lazy to continue the reading group, so you needed a good excuse. But that's ridiculous, and I assign very low probability to that.

The sole point of my comment is this. I'm not upset because of your fundamental disagreement with Yudkowsky and LW's ideology and memes. I'm upset because you stop the reading group, which is important, because, like I said, the Sequences are about basic rational thinking, not deep philosophy, in which Yudkowsky indeed might be completely wrong. I'm upset, because your departure would mean that you think that LW is completely lost, and there is not at least a sizable minority, who'd say “you know what, you're right, let's do something about it”. That's sad.

(I'll update this post with more thoughts)

Comment author: Error 22 May 2015 02:56:13AM *  8 points [-]

Despite Yudkowsky's obvious leanings, the Sequences are not about FAI, nor [etc]...they are first and foremost about how to not end up an idiot. They are about how to not become immune to criticism, they are about Human's Guide to Words, they are about System 1 and System 2.

I've always had the impression that Eliezer intended them to lead a person from zero to FAI. So I'm not sure you're correct here.

...but that being said, the big Less Wrong takeaways for me were all from Politics is the Mind-Killer and the Human's Guide to Words -- in that those are the ones that have actually changed my behavior and thought processes in everyday life. They've changed the way I think to such an extent that I actually find it difficult to have substantive discussions with people who don't (for example) distinguish between truth and tribal identifiers, distinguish between politics and policy, avoid arguments over definitions, and invoke ADBOC when necessary. Being able to have discussions without running over such roadblocks is a large part of why I'm still here, even though my favorite posters all seem to have moved on. Threads like this one basically don't happen anywhere else that I'm aware of.

Someone recently had a blog post summarizing the most useful bits of LW's lore, but I can't for the life of me find the link right now.

Comment author: jam_brand 22 May 2015 03:45:39AM 4 points [-]

I'm not sure if this is what you were thinking of (seeing as how it's about a year old now), but "blog post summarizing the most useful bits of LW's lore" makes me think of Yvain's Five Years and One Week of Less Wrong.

Comment author: [deleted] 22 May 2015 06:30:07AM *  6 points [-]

I've always had the impression that Eliezer intended them to lead a person from zero to FAI. So I'm not sure you're correct here.

Eliezer states this explicitly on numerous occasions, that his reason for writing the blog posts was to motivate people to work with him on FAI. I'm having trouble coming up with exact citations however, since it's not very google-able.

My prior perception of the sequences was that EY started from a firm base of generlaly good advice about thinking. Sequences like Human's guide to words and How to actually change your mind stand on their own. He then however went off the deep end trying to extend and apply these concepts to questions in the philosophy of the mind, ethics, and decision theory in order to motivate an interest in friendly AI theory.

I thought that perhaps the mistakes made in those sequences where correctable one-off errors. Now I am of the opinion that the way in which that philosophical inquiry was carried out doomed the project to failure from the start, even if the details of the failure is subject to Yudkowsky's own biases. Reasoning by thought experiment only over questions that are not subject to experimental validation basically does nothing more than expose one's priors. And either you agree with the priors, or you don't. For example, does quantum physics support the assertion that identity is the instance of computation or the information being computed? Neither. But you could construct a thought experiment which validates either view based on the priors you bring to the discussion, and I have wasted much time countering his thought experiments with those of my own creation before I understood the Sisyphean task I was undertaking :\

Comment author: Jiro 23 May 2015 02:51:44AM *  4 points [-]

As another person who thinks that the Sequences and FAI are nonsense (more accurately, the novel elements in the Sequences are nonsense; most of them are not novel), I have my own theory: LW is working by accidentally being counterproductive. You have people with questionable beliefs, who think that any rational person would just have to believe them. So they try to get everyone to become rational, thinking it would increase belief in those things. Unfortunately for them, when they try this, they succeed too well--people listen to them and actually become more rational, and actually becoming rational doesn't lead to belief in those things at all. Sometimes it even provides more reasons to oppose those things--I hadn't heard of Pascal's Mugging before I came here, and it certainly wasn't intended to be used as an argument against cryonics or AI risk, but it's pretty useful for that purpose anyway.

Comment author: [deleted] 23 May 2015 01:56:48PM 2 points [-]

How is Pascal's Mugging an argument against cryonics?

Comment author: Jiro 23 May 2015 02:41:37PM 1 point [-]

It's an argument against "even if you think the chance of cryonics working is low, you should do it because if it works, it's a very big benefit".

Comment author: [deleted] 23 May 2015 02:53:57PM *  1 point [-]

Ok, it's an argument against a specific argument for cryonics. I'm ok with that (it was a bad argument for cryonics to start with). Cryonics does have a lot of problems, not least of which is cost. The money spent annually on life insurance premiums for cryopreservation of a ridiculously tiny segment of the population is comparable to the research budget for SENS which would benefit everybody. What is up with that.

That said, I'm still signing up for Alcor. But I'm aware of the issues :\

Comment author: Error 23 May 2015 01:42:57PM 2 points [-]

As another person who thinks that the Sequences and FAI are nonsense

Clarification: I don't think they're nonsense, even though I don't agree with all of them. Most of them just haven't had the impact of PMK and HGW.

Comment author: [deleted] 21 May 2015 11:10:03PM *  14 points [-]

Despite Yudkowsky's obvious leanings, the Sequences are ... first and foremost about how to not end up an idiot

My basic thesis is that even if that was not the intent, the result has been the production of idiots. Specifically, a type of idiotic madness that causes otherwise good people, self-proclaimed humanitarians to disparage the only sort of progress which has the potential to alleviate all human suffering, forever, on accelerated timescales. And they do so for reasons that are not grounded in empirical evidence, because they were taught though demonstration modes of non-empirical thinking from the sequences, and conditioned to think this was okay through social engagement on LW.

When you find yourself digging a hole, the sensible and correct thing to do is stop digging. I think we can do better, but I'm burned out on trying to reform from the inside. Or perhaps I'm no longer convinced that reform can work given the nature of the medium (social pressures of blog posts and forums work counter to the type of rationality that should be advocated for).

I don't care about Many Worlds, FAI, Fun theory and Jeffreyssai stuff, but LW was the thing that stopped me from being a complete and utter idiot.

I don't want to take that away. But for me LW was not just a baptismal fount for discovering rationality, it was also an effort to get people to work on humanitarian relief and existential risk reduction. I hope you don't think me crazy for saying that LW has had a subject matter bias in these directions. But on at least some of these accounts the effect had by LW and/or MIRI and/or Yudkowsky's specific focus on these issues may be not just suboptimal, but actually negative. To be precise: it may actually be causing more suffering than would otherwise exist.

We are finally coming out of a prolonged AI winter. And although funding is finally available to move the state of the art in automation forward, to accelerate progress in life sciences and molecular manufacturing that will bring great humanitarian change, we have created a band of Luddites that fear the solution more than the problem. And in a strange twist of double-think, consider themselves humanitarians for fighting progress.

If you had your own forum with lots of people, who share similar criticism of LW, hey, I'd go there and leave LW. But you don't have such forum, so by leaving LW you just leave people like me alone. What's the point of that? Do you really believe leaving LW like that is more utility, than trying to create an island within it?

I am myself working on various projects in my life which I expect to have positive effects on the world. Outside of work, LW has at times occupied a significant fraction of my leisure time. This must be seen as an activity of higher utility than working more hours on my startup, making progress on my molecular nanotech and AI side projects, or enriching myself personally in other ways (family time, reading, etc.). I saw the Rationality reading group as a chance to do something which would conceivably grow that community by a measurable amount, thereby justifying a time expenditure. However if all I am doing is bringing more people into a community that is actively working against developments in artificial intelligence that have a chance of relieving human suffering within a single generation… the Hippocratic corpus comes to mind: “first, do no harm.”

I am not sure yet what I will fill the time with. Maybe I'll get off my butt and start making more concrete progress on some of the nanotech and AI stuff that I have been letting slide in recent years.

I recognize also that I am making broad generalizations which do not always apply to everyone. You seem to be an exception, and I wish I had engaged with you more. I also will miss TheAncientGeek's contrarian posts, as well as many others who deserve credit for not following a herd mentality.

Comment author: Mirzhan_Irkegulov 21 May 2015 11:32:55PM 13 points [-]

Thank you for your response, that's really important for me.

I've never seen disparaging of actually helping people on LW. Can you point to examples? Can you argue that it is a tendency? You say that there is lots of outright hostility to anything against x-risks and human misery, except if it's MIRI. I wouldn't even imagine anyone would say that of LW, but maybe I'm blind, so I'll be grateful if you prove me wrong. Yudkowsky is definitely pro-immortality and supported donating to SENS.

I don't even think MIRI and MIRI-leaning LWers are against ongoing AI research. I've never heard anything like “please stop doing any AI until we figure out friendliness”, only “hey, can you please put more effort into friendliness too, it's very important?” And even if you think that MIRI's focus on friendliness is order of magnitude misplaced, it's just a mistake of prioritizing, not a fundamental philosophical blunder. Again, if you can expand on this topic, I would only say thank you.

Maybe “reform” isn't the right word. The Sequences aren't going anywhere, so of course LW will be FAI-centric for a long time, but within LW there is already a substantial amount of people (that's my impression, I never actually counted) who are not simply contrarian, but actually assign different priorities on what should be done about the world. More inline with your thoughts, than Yudkowsky's. Maybe you can still stay and steer this substantial minority in the right direction, instead of useless splitting.

I bet most people on LW are not even high-karma prolific writers, they are less knowledge, less confident, but also more open to contrary views, such as yours. Just by writing one big article about how you think LW's focus is misplaced can be of extreme help for such people. Which, BTW, includes me, because I never posted anything.

I'd actually would love to see you writing articles on all your theses here, on LW. LW-critical articles were already promoted a few times, including Yvain's article, so it's not like LW is criticism-intolerant.

If you actually do that, and provide lots of examples and evidence, it would be a breathe of fresh air for all those people, who will continue to be attracted to LW. You don't have to put titanic effort into “reform”, just erect a pole.

Comment author: [deleted] 23 May 2015 02:08:06PM 2 points [-]

You say that there is lots of outright hostility to anything against x-risks and human misery, except if it's MIRI.

I was actually making a specific allusion to the hostility towards practical, near-term artificial general intelligence work. I have at times in the past advocated for working on AGI technology now, not later, and been given robotic responses that I'm offering reckless and dangerous proposals, and helpfully directed to go read the sequences. I once joined #lesswrong on IRC and introduced myself as someone interested in making progress in AGI in the near-term, and received two separate death threats (no joke). Maybe that's just IRC—but I left and haven't gone back.

Comment author: TheAncientGeek 22 May 2015 10:50:39AM 2 points [-]

actually would love to see you writing articles on all your theses here, on LW. LW-critical articles were already promoted a few times, including Yvain's article, so it's not like LW is criticism-intolerant.

Things have changed, believe me.

Comment author: Mirzhan_Irkegulov 22 May 2015 01:10:35PM 4 points [-]

Can you point to some examples? Yvain's article was recently on the Main page under Featured articles, for example.

Comment author: Nornagest 22 May 2015 06:19:33PM 1 point [-]

I don't know exactly what process generates the featured articles, but I don't think it has much to do with the community's current preoccupations.

Comment author: Mirzhan_Irkegulov 22 May 2015 06:30:24PM *  4 points [-]

I don't know exact process either, but I always thought somebody deliberately chooses them each week, because often they are around the same topic. So somebody thought it's a good idea to encourage everybody to read an LW-critical article.

My point is, I don't believe LW community suddenly became intolerant to criticism. Or incapable of dialog on whether FAI is a good thing. Or fanatically believing in FAI and Yudkowsky's ideas. Oh, and I'm happy to be proven otherwise!

Seriously, look at top 30 day contributors:

  • Lumifer (629)
  • JonahSinick (626)
  • Vaniver (355)
  • ChristianKl (329)
  • DeVliegendeHollander (251)
  • Richard_Loosemore (242)
  • NancyLebovitz (232)
  • Viliam (209)
  • gjm (184)
  • So8res (180)
  • VoiceOfRa (178)
  • IlyaShpitser (166)
  • Error (162)
  • Mark_Friedenbach (146)
  • JohnMaxwellIV (135)

Only So8res is associated with MIRI, AFAIK. My impression from comments of the people above is that they are pretty much capable of dialog and are not fanatical about FAI at all.

Meaning that in Mark's map LW community is something different than in territory. He think he leaves a crazy cult producing a memetic hazard. I think he leaves a community of pretty much independent-thinking people, who could easily counter MIRI's memes.

That is, even if Mark is completely correct about MIRI, his leaving is irrelevant, it's not a net improvement, but some strange unrelated act with negative utility.

Comment author: TheAncientGeek 23 May 2015 11:34:19AM 1 point [-]

My point is, I don't believe LW community suddenly became intolerant to criticism.

My point was that it has become a lot more tolerant.

Comment author: [deleted] 23 May 2015 02:10:43PM 1 point [-]

Maybe, but the core beliefs and cultural biases haven't changed, in the years that I've been here.

Comment author: TheAncientGeek 23 May 2015 02:35:08PM 4 points [-]

But you didn't get karmassinated or called an idiot.

Comment author: TheAncientGeek 26 May 2015 07:22:29PM 0 points [-]
Comment author: Sarunas 22 May 2015 04:58:11PM *  3 points [-]

But on at least some of these accounts the effect had by LW and/or MIRI and/or Yudkowsky's specific focus on these issues may be not just suboptimal, but actually negative. To be precise: it may actually be causing more suffering than would otherwise exist.

If I understand correctly, you think that LW, MIRI and other closely related people might have a net negative impact, because they distract some people from contributing to the more productive subareas/approaches of AI research and existential risk prevention, directing them to subareas which you estimate to be much less productive. For the sake of argument, let's assume that is correct and if all people who follow MIRI's approach to AGI turned to those subareas of AI that are more productive, it would be a net benefit to the world. But you should consider the other side of the medallion, that is, doesn't blogs like LessWrong or books like that of N.Bostrom's actually attract some students to consider working on AI, including the areas you consider beneficial, who would otherwise be working in areas that are unrelated to AI? Wouldn't the number of people who have even heard about the concept of existential risks be smaller without people like Yudkowsky and Bostrom? I don't have numbers, but since you are concerned about brain drain in other subareas of AGI and existential risk research, do you think it is unlikely that popularization work done by these people would attract enough young people to AGI in general and existential risks in general that would compensate for the loss of a few individuals, even in subareas of these fields that are unrelated to FAI?

We are finally coming out of a prolonged AI winter. And although funding is finally available to move the state of the art in automation forward, to accelerate progress in life sciences and molecular manufacturing that will bring great humanitarian change, we have created a band of Luddites that fear the solution more than the problem. And in a strange twist of double-think, consider themselves humanitarians for fighting progress.

But do people here actually fight progress? Has anyone actually retired from (or was dissuaded from pursuing) AI research after reading Bostrom or Yudkowsky?

If I understand you correctly, you fear that concerns about AI safety, being a thing that might invoke various emotions in a listener's mind, is a thing that is sooner or later bound to be picked up by some populist politicians and activists who would sow and exploit these fears in the minds of general population in order to win elections/popularity/prestige among their peers/etc., thus leading to various regulations and restrictions on funding, because that is what these activists (who got popular and influential by catering to the fears of the masses) would demand?

Comment author: [deleted] 23 May 2015 02:15:52PM 2 points [-]

I'm not sure how someone standing on a soapbox and yelling "AI is going to kill us al!" (Bostrom, admittedly not a quote) can be interpreted as actually helping get more people into practical AI research and development.

You seem to be presenting a false choice: is there more awareness of AI in a world with Bostrom et al, or the same world without? But it doesn't have to be that way. Ray Kurzweil has done quite a bit to keep interest in AI alive without fear mongering. Maybe we need more Kurzweils and fewer Bostroms.

Comment author: Kaj_Sotala 02 June 2015 12:58:22PM 1 point [-]

Data point: a feeling that I ought to do something about AI risk is the only reason why I submitted an FLI grant proposal that involves some practical AI work, rather than just figuring that the field isn't for me and doing something completely different.

Comment author: Sarunas 31 May 2015 08:50:21AM 0 points [-]

I'm not sure how someone standing on a soapbox and yelling "AI is going to kill us al!" (Bostrom, admittedly not a quote) can be interpreted as actually helping get more people into practical AI research and development.

I don't know how many copies of Bostrom's book were sold, but it was on New York Times list. Some of those books were read by high school students. Since very few people leave practical AI research for FAI research, even if only a tiny fraction of these young readers read the book and think "This AI thing is really exciting and interesting. Instead of majoring in X (which is unrelated to AI), I should major in computer science and focus on AI", it would probably result in net gain for practical AI research.

You seem to be presenting a false choice: is there more awareness of AI in a world with Bostrom et al, or the same world without? But it doesn't have to be that way. Ray Kurzweil has done quite a bit to keep interest in AI alive without fear mongering. Maybe we need more Kurzweils and fewer Bostroms.

I argued against this statement:

specific focus on these issues may be not just suboptimal, but actually negative. To be precise: it may actually be causing more suffering than would otherwise exist.

When people say that an action leads to a negative outcome, they usually mean that taking that action is worse than not taking it, i.e. they compare the result to zero. If you add another option, then the word "suboptimal" should be used instead. Since I argued against "negativity", and not "suboptimality", I dont' think that the existence of other options is relevant here.

Comment author: CellBioGuy 22 May 2015 12:37:20AM *  1 point [-]

I also will miss TheAncientGeek's contrarian posts, as well as many others who deserve credit for not following a herd mentality.

Interesting, I seem to buck the herd in nearly exactly the opposite manner as you.

Comment author: [deleted] 22 May 2015 12:49:15AM 2 points [-]

Meaning?

Comment author: CellBioGuy 22 May 2015 01:06:27AM 6 points [-]

You buck the herd by saying their obsession with AI safety is preventing them from participating in the complete transformation of civilization.

I buck the herd by saying that the whole singulatarian complex is a chimera that has almost nothing to do with how reality will actually play out and its existence as a memeplex is explained primarily by sociological factors rather than having much to do with actual science and technology and history.

Comment author: [deleted] 22 May 2015 01:13:45AM 3 points [-]

Oh, well I mostly agree with you there. Really ending aging will have a transformative effect on society, but the invention of AI is not going to radically alter power structures in the way that singulatarians imagine.

Comment author: CellBioGuy 22 May 2015 12:45:22PM 6 points [-]

See, I include the whole 'immanent radical life extension' and 'Drexlerian molecular manufacturing' idea sets in the singulatarian complex...

Comment author: [deleted] 23 May 2015 02:16:54PM 2 points [-]

The craziest person in the world can still believe the sky is blue.

Comment author: CellBioGuy 23 May 2015 02:57:25PM 2 points [-]

Ah, but in this case as near as i can tell it is actually orange.

Comment author: drethelin 22 May 2015 06:45:52PM 1 point [-]

This just seems stupid to me. Ending aging is fundamentally SLOW change. In 100 or 200 or 300 years from now, as more and more people gain access to anti-aging (since it will start off very expensive), we can worry about that. But conscious AI will be a force in the world in under 50 years. And it doesn't even have to be SUPER intelligent to cause insane amounts of social upheaval. Duplicability means that even 1 human level AI can be world-wide or mass produced in a very short time!

Comment author: Lumifer 22 May 2015 07:33:56PM 2 points [-]

But conscious AI will be a force in the world in under 50 years.

"Will"? You guarantee that?

Comment author: [deleted] 22 May 2015 08:25:46AM 1 point [-]

Really ending aging will have a transformative effect on society

"The medical revolution that began with the beginning of the twentieth century had warped all human society for five hundred years. America had adjusted to Eli Whitney's cotton gin in less than half that time. As with the gin, the effects would never quite die out. But already society was swinging back to what had once been normal. Slowly; but there was motion. In Brazil a small but growing, alliance agitated for the removal of the death penalty for habitual traffic offenders. They would be opposed, but they would win."

Larry Niven: The Gift From Earth

Comment author: [deleted] 22 May 2015 08:44:46AM 7 points [-]

Well there are some serious ramifications that are without historical precedent. For example, without menopause it may perhaps become the norm for women to wait until retirement to have kids. It may in fact be the case that couples will work for 40 years, have a 25-30 year retirement where they raise a cohort of children, and then re-enter the work force for a new career. Certainly families are going to start representing smaller and smaller percentages of the population as birth rates decline while people get older and older without dying. The social ramifications alone will be huge, which was more along the lines of what I was talking about.

Comment author: [deleted] 22 May 2015 08:24:09AM *  1 point [-]

Can you link to a longer analysis of yours regarding this?

I simply feel overwhelmed when people discuss AI. To me intelligence is a deeply anthropomorphic category, includes subcategories like having a good sense of humor. Reducing it to optimization, without even sentience or conversational ability with self-consciousness... my brain throws out the stop sign already at this point and it is not even AI, it is the pre-studies of human intelligence that already dehumanize, deanthromorphize the idea of intelligence and make it sound more like a simple and brute-force algorithm. Like Solomonoff Induction, another thing that my brain completely freezes over: how can you have truth and clever solutions without even really thinking, just throwing a huge number of random ideas in and seeing what survives testing? Would it all be so quantitative? Can you reduce the wonderful qualities of the human mind to quantities?

Comment author: FeepingCreature 22 May 2015 11:06:03AM 9 points [-]

Intelligence to what purpose?

Nobody's saying AI will be human without humor, joy, etc. The point is AI will be dangerous, because it'll have those aspects of intelligence that make us powerful, without those that make us nice. Like, that's basically the point of worrying about UFAI.

Comment author: [deleted] 22 May 2015 11:34:08AM 1 point [-]

But is it possible to have power without all the rest?

Comment author: estimator 22 May 2015 03:13:07PM *  3 points [-]

Certainly. Why not?

Computers already can outperform you in a wide variety of tasks. Moreover, today, with the rise of machine learning, we can train computers to do pretty high-level things, like object recognition or senitment analysis (and sometimes outperform humans in these tasks). Isn't it power?

As for Solomonoff induction... What do you think your brain is doing when you are thinking? Some kind of optimized search in hypotheses space, so you consider only a very very small set of hypotheses (compared to the entire space), hopefully good enough ones. While Solomonoff induction checks all of them, every single hypothesis, and finds the best.

Solomonoff induction is so much thinking that it is incomputable.

Since we don't have that much raw computing power (and never will have), the hypotheses search must be heavily optimized. Throwing off unpromising directions of search. Searching in regions with high probability of success. Using prior knowledge to narrow search. That's what your brain is doing, and that's what machines will do. That's not like "simple and brute-force", because simple and brute-force algorithms are either impractically slow, or incomputable at all.

Comment author: [deleted] 22 May 2015 03:28:21PM 1 point [-]

Computers already can outperform you in a wide variety of tasks.

Eagles, too: they can fly and I not. The question is whether the currently foreseeable computerizable tasks are closer to flying or to intelligence. Which in turn depends on how high and how "magic" we see intelligence.

As for Solomonoff induction... What do you think your brain is doing when you are thinking?

Ugh, using Aristotelean logic? So it is not random hypotheses but causality and logic based.

Solomonoff induction is so much thinking that it is incomputable.

I think using your terminology, thinking is not the searching, it is the findinging logical relationships so not a lot of space must be searched.

That's not like "simple and brute-force", because simple and brute-force algorithms are either impractically slow, or incomputable at all.

OK, that makes sense. Perhaps we can agree that logic and casuality and actual reasoning is all about narrowing the hypothesis space to search. This is intelligence, not the search.

Comment author: Dentin 22 May 2015 02:59:33PM 1 point [-]

Yes. Absolutely. When that happens inside a human being's head, we generally call them 'mass murderers'. Even I only cooperate with society because there is a net long term gain in doing so; if that were no longer the case, I honestly don't know what I would do. Awesome, that's something new to think about. Thanks.

Comment author: estimator 22 May 2015 03:19:43PM 0 points [-]

That's probably irrelevant, because mass murderers don't have power without all the rest. They are likely to have sentience and conversational ability with self-consciousness, at least.

Comment author: FeepingCreature 22 May 2015 02:49:03PM *  0 points [-]

Not sure. Suspect nobody knows, but seems possible?

I think the most instructive post on this is actually Three Worlds Collide, for making a strong case for the arbitrary nature of our own "universal" values.

Comment author: [deleted] 22 May 2015 03:41:08AM 5 points [-]

:( Bye, thanks for your reading group. I really appreciated it and your summaries.

What were some of the errors you found in the sequences? Was it mostly AI stuff?

Even though I'm two-thirds through the sequences and have been around the site for two months now, I still don't really understand AI/haven't been convinced to donate to something other than givewell's charities. I feel like when I finally understand it, I probably will switch over to existential risk reduction charities though, so thanks for your thoughts there. I might have figured it was safe to assume MIRI was the best without really thinking for myself, just because I generally feel like the people here are smart and put a lot of thought into things.

Comment author: [deleted] 22 May 2015 08:18:33AM 13 points [-]

On a personal level, I am no longer sure engagement with such a community is a net benefit.

I am simply treating LW as a 120+ IQ version of Reddit. Just generic discussion with mainly bright folks. The point is, I don't know any other. I used to know Digg and MetaFilter and they are not much better either. If we could make a list of cerebral discussion boards, forums, and suchlike, that would be a good idea I guess. Where do you expect to hang out in the future?

Comment author: [deleted] 22 May 2015 08:39:03AM 9 points [-]

I'll probably spend less time hanging out in online communities, honestly.

Comment author: Viliam 22 May 2015 02:13:30PM 4 points [-]

I am simply treating LW as a 120+ IQ version of Reddit. Just generic discussion with mainly bright folks. The point is, I don't know any other.

When you mention IQ, Mensa seems like an obvious answer. They are already filtering people for IQ, so it could be worth:

a) finding out whether they have an online forum for members, and if not, then

b) creating such forum.

Obviously, the important thing is not merely filtering of the users, but also moderation mechanisms, otherwise you can get a few "high IQ idiots" spam the whole system with some conspiracy theories or flame wars.

Unfortunately, you would probably realize that IQ is not the most important thing.

Comment author: [deleted] 22 May 2015 03:07:23PM 6 points [-]

Unfortunately, you would probably realize that IQ is not the most important thing.

Precisely. The Mensa members I know are too interested in rather pointless brain-teasers.

Comment author: Benito 22 May 2015 06:16:21PM 4 points [-]

I'd like to offer some responses.

[one argument of people who think that the superintelligence alginment problem is incredibly important] is that just as a chimpanzee would be unable to predict what a human intelligence would do or how we would make decisions (aside: how would we know? Were any chimps consulted?), we would be equally inept in the face of a super-intelligence... The human capacity for abstract reasoning over mathematical models is in principle a fully general intelligent behaviour, as the scientific revolution has shown: there is no aspect of the natural world which has remained beyond the reach of human understanding, once a sufficient amount of evidence is available.

Yes, when you wrote this, I had a funny feeling that this argument was muhc weaker than I had previously thought. I think the relevant points are as follows: The people interested in Superintelligence Alignment tend to use this as an example of agents, with a given optimization power, being outperformed by agents with greater levels of optimization power, of which I think the example is apt. Perhaps you would like to argue that optimization power is not so coherent a concept, as humans' mathematical modelling and scientific reasoning is qualitatively different to anything that a chimpanzee can do, rather than just a proportional increase in this notion of 'optimization power'. I think that science is a powerful insight, and perhaps there is not another insight of equivalent power, but I don't think this invalidates the concept of optimization power. You can still see the semblence of a scale on which you put mice, humans, and Superintelligent AI, to do with how well the agent can effect its goals in the world. Humans being able to reason about AI does not invalidate this.

Then you go on to say that this is a specific example of a general problem. As to the general problem of reasoning by analogy rather than forming predictive models... I assign a high probability to the statement that most Superintelligence Alignment folk use analogies to communicwte intuition to the public, as opposed to those being their main reasons for holding their beliefs.

There's lip service done to empiricism throughout, but in all the “applied” sequences relating to quantum physics and artificial intelligence it appears to be forgotten. We get instead definitive conclusions drawn from thought experiments only. It is perhaps not surprising that these sequences seem the most controversial.

I think that, between nuanced abstract reasoning and using clear empirical evidence to resolve disputes, humanity is better at the second. I agree that many people on this planet need to learn to do it much better, but there are other places to learn to do that. Eliezer is here trying to write about the more complex problem of using your reasoning to 'not shoot yourslf in the foot', repeatedly. However, to say he does little more than 'pay lip service' to empiricism seems to miss all of his posts that just explain a particular psychological study.

There is work that could be done now to alleviate both of these issues. But within the LessWrong community there is actually outright hostility to work that has a reasonable chance of alleviating suffering (e.g. artificial general intelligence applied to molecular manufacturing and life-science research) due to concerns arrived at by flawed reasoning.

I think the first point here, is that most people in this world are probably hostile to important work due to their flawed epistemologies. Or at least, they have a very flawed set of priorities. Arbitrary political issues dominate discussion. I think that if you were to offer a post with considerable substance, focusing on just this claim, or any of your above claims, most good rationalists would change their minds ('good rationalists' here defined as people who have gone out of their way in their life to do EA work, like been to meetings or done research). You'd get a better chance of changing people's minds by poeting here than on Reddit, I bet.

...much of our uncertainty about the actions of an unfriendly AI could be resolved if we were to know more about how such agents construct their thought models, and relatedly what language were used to construct their goal systems. We could also stand to benefit from knowing more practical information (experimental data) about in what ways AI boxing works and in what ways it does not, and how much that is dependent on the structure of the AI itself. Thankfully there is an institution that is doing that kind of work: the Future of Life institute (not MIRI).

If you are not familiar with MIRI's current technical agenda, then you may wish to retract this claim, specifically regarding the point about how agents construct their thought models: this is currently their entire research project, loosely speaking. Figuring out how agents model logical uncertainty in the environment, and model the environment with itself as a subset, are two of it's main focus areas. If you are familiar with their research, you may wish to flesh out that argument a little more. I personally didn't think that FLI did any research atm, just worked on making the cause better-appreciated in the public eye, so I'll go read up on that at some point.

I will no longer be running the Rationality: From AI to Zombies reading group

Alas. I was enjoying the series.

I hope this comment is taken in good faith, your post suggested you had good reasons behind your claims, and I'd be interested in hearing them.

Comment author: [deleted] 23 May 2015 02:23:54PM *  3 points [-]

If you are not familiar with MIRI's current technical agenda, then you may wish to retract this claim.

I am familiar with MIRI's technical agenda and I stand by my words. The work MIRI is choosing for itself is self-isolating and not relevant to the problems at hand in practical AGI work.

Comment author: Kaj_Sotala 23 May 2015 09:11:36PM 2 points [-]

The work MIRI is choosing for itself is self-isolating

AFAIK, part of why the technical agenda contains the questions it does is that they're problems that are of interest to people to mathematicians and logicians even if those people aren't interested in AI risk. (Though of course, that doesn't mean that AI researchers would be interested in that work, but it's at least still more connecting with the academic community than "self-isolating" would imply.)

Comment author: jacob_cannell 24 May 2015 06:26:54AM 1 point [-]

AFAIK, part of why the technical agenda contains the questions it does is that they're problems that are of interest to people to mathematicians and logicians even if those people aren't interested in AI risk.

This is concerning if true - the goal of the technical agenda should be to solve AI risk, not appeal to mathematicians and logicians (by say making them feel important).

Comment author: Kaj_Sotala 24 May 2015 10:03:15AM 3 points [-]

That sounds like an odd position to me. IMO, getting as many academics from other fields as possible working on the problems is essential if one wants to make maximal progress on them.

Comment author: [deleted] 24 May 2015 05:42:28PM *  1 point [-]

The academic field which is most conspicuously missing is artificial intelligence. I agree with Jacob that it is and should be concerning that the machine intelligence research institute has adopted a technical agenda which is non-inclusive of machine intelligence researchers.

Comment author: Kaj_Sotala 24 May 2015 11:18:20PM *  2 points [-]

I agree with Jacob that it is and should be concerning

That depends on whether you believe that machine intelligence researchers are the people who are currently the most likely to produce valuable progress on the relevant research questions.

One can reasonably disagree on MIRI's current choices about their research program, but I certainly don't think that their choices are concerning in the sense of suggesting irrationality on their part. (Rather the choices only suggest differing empirical beliefs which are arguable, but still well within the range of non-insane beliefs.)

Comment author: [deleted] 26 May 2015 06:23:12PM 3 points [-]

On the contrary, my core thesis is that AI risk advocates are being irrational. It's implied in the title of the post ;)

Specifically I think they are arriving at their beliefs via philosophical arguments about the nature of intelligence which are severely lacking in empirical data, and then further shooting themselves in the foot by rationalizing reasons to not pursue empirical tests. Taking a belief without evidence, and then refusing to test that belief empirically--I'm willing to call a spade a spade: that is most certainly irrational.

Comment author: jacob_cannell 27 May 2015 03:38:07PM 0 points [-]

That's a good summary of your post.

I largely agree, but to be fair we should consider that MIRI started working on AI safety theory long before the technology required for practical experimentation with human-level AGI - to do that you need to be close to AGI in the first place.

Now that we are getting closer, the argument for prioritizing experiments over theory becomes stronger.

Comment author: jacob_cannell 27 May 2015 03:48:03PM 0 points [-]

There are many types of academics - does your argument extend to french literature experts?

Clearly, if there is a goal behind the technical agenda, changing the technical agenda to appeal to certain groups detracts from that goal. You could argue that enlisting the help of mathematicians and logicians is so important it justifies changing the agenda ... but I doubt there is much historical support for such a strategy.

I suspect part of the problem is that the types of researchers/academics which could most help (machine learning, statistics, comp sci types) are far too valuable to industry and thus are too expensive for non-profits such as MIRI.

Comment author: Kaj_Sotala 02 June 2015 01:03:38PM 0 points [-]

There are many types of academics - does your argument extend to french literature experts?

Well, if MIRI happened to know of technical problems they thought were relevant for AI safety and which they thought French literature experts could usefully contribute to, sure.

I'm not suggesting that they would have taken otherwise uninteresting problems and written those up simply because they might be of interest to mathematicians. Rather my understanding is that they had a set of problems that seemed about equally important, and then from that set, used "which ones could we best recruit outsiders to help with" as an additional criteria. (Though I wasn't there, so anything I say about this is at best a combination of hearsay and informed speculation.)

Comment author: Dentin 22 May 2015 02:50:26PM 4 points [-]

Thank you for the shout out to SENS. This is where I send my charitable donations as well, and it's good to see someone else who thinks they're as effective as I do.

Comment author: Bound_up 21 May 2015 10:51:16PM *  4 points [-]

Am I safe if I just maintain a level of skepticism in the presence of thought-experimental "evidence" for a conclusion?

You've mentioned the role of the Sequences in reference to teaching specific conclusions about things like AI, altruism, cryonics, I presume; that's a minority of the whole (in my reading, so far). Would you dispute them for their use in identifying reasoning flaws?

EDIT: I do appreciate your going further in your criticism of the LW mainstream than most (though I've appreciated that of others also). I take it as an invitation to greater care and skepticism.

Of course, you're familiar with "Evaporative Cooling of Group Beliefs," right?

Comment author: Nornagest 22 May 2015 06:08:28PM *  1 point [-]

Evaporative cooling is a concern, but it seems to me that it's a more directly relevant concern for moderators or other types of organizers than for individuals. For it to be driving a decision to leave or not to leave, you effectively have to value the group's well-being over your own, which is unlikely for a group you're considering leaving in the first place.

Comment author: Valentine 23 May 2015 04:48:45PM *  8 points [-]

Thank you for this.

I see you as highlighting a virtue that the current Art gestures toward but doesn't yet embody. And I agree with you, a mature version of the Art definitely would.

In his Lectures on Physics, Feynman provides a clever argument to show that when the only energy being considered in a system is gravitational potential energy, then the energy is conserved. At the end of that, he adds the following:

It is a very beautiful line of reasoning. The only problem is that perhaps it is not true. (After all, nature does not have to go along with our reasoning.) For example, perhaps perpetual motion is, in fact, possible. Some of the assumptions may be wrong, or we may have made a mistake in reasoning, so it is always necessary to check. It turns out experimentally, in fact, to be true.

This is such a lovely mental movement. Feynman deeply cared about knowing how the world really actually works, and it looks like this led him to a mental reflex where even in cases of enormous cultural confidence he still responds to clever arguments by asking "What does nature have to say?"

In my opinion, people in this community update too much on clever arguments. I include myself in that. I disagree with your claim that people shouldn't update at all on clever arguments, but I very much agree that there would be much more strength in the Art if it were to emphasize an active hunger for asking nature its opinion.

I think there's a flavor of mistake that comes from overemphasizing the direction I see you pointing at the expense of other virtues. I've known quite a number of scientists who think the way I see you suggesting who feel like they can't have any opinions or thoughts about things they haven't seem empirical tests of. I think they're in part trying to protect themselves against what Eliezer calls "privileging the hypothesis", but they also make themselves unnecessarily stupid in some ways. The most common and blatant I recall is their getting routinely blindsided by predictable social expectations and drama.

But I think Feynman gets it right.

And I think we ought to, too.

So again, thank you for bringing this up. It clarified something that had been nagging me, and now I think I see how to fix it.

Comment author: RobbBB 22 May 2015 09:49:24PM *  11 points [-]

Thanks for taking the time to explain your reasoning, Mark. I'm sorry to hear you won't be continuing the discussion group! Is anyone else here interested in leading that project, out of curiosity? I was getting a lot out of seeing people's reactions.

I think John Maxwell's response to your core argument is a good one. Since we're talking about the Sequences, I'll note that this dilemma is the topic of the Science and Rationality sequence:

In any case, right now you've got people dismissing cryonics out of hand as "not scientific", like it was some kind of pharmaceutical you could easily administer to 1000 patients and see what happened. "Call me when cryonicists actually revive someone," they say; which, as Mike Li observes, is like saying "I refuse to get into this ambulance; call me when it's actually at the hospital". Maybe Martin Gardner warned them against believing in strange things without experimental evidence. So they wait for the definite unmistakable verdict of Science, while their family and friends and 150,000 people per day are dying right now, and might or might not be savable—

—a calculated bet you could only make rationally [i.e., using your own inference skills, without just echoing data from an experimental study, and without just echoing established, expert-verified scientific conclusions].

The drive of Science is to obtain a mountain of evidence so huge that not even fallible human scientists can misread it. But even that sometimes goes wrong, when people become confused about which theory predicts what, or bake extremely-hard-to-test components into an early version of their theory. And sometimes you just can't get clear experimental evidence at all.

Either way, you have to try to do the thing that Science doesn't trust anyone to do—think rationally, and figure out the answer before you get clubbed over the head with it.

(Oh, and sometimes a disconfirming experimental result looks like: "Your entire species has just been wiped out! You are now scientifically required to relinquish your theory. If you publicly recant, good for you! Remember, it takes a strong mind to give up strongly held beliefs. Feel free to try another hypothesis next time!")

This is why there's a lot of emphasis on hard-to-test ("philosophical") questions in the Sequences, even though people are notorious for getting those wrong more often than scientific questions -- because sometimes (e.g., in the case of cryonics and existential risk) the answer matters a lot for our decision-making, long before we have a definitive scientific answer. That doesn't mean we should despair of empirically investigating these questions, but it does mean that our decision-making needs to be high-quality even during periods where we're still in a state of high uncertainty.

The Sequences talk about the Many Worlds Interpretation precisely because it's an unusually-difficult-to-test topic. The idea isn't that this is a completely typical example, or that it's a good idea to disregard evidence when it is available; the idea, rather, is that we sometimes do need to predicate our decisions on our best guess in the absence of perfect tests.

Its placement in Rationality: From AI to Zombies immediately after the 'zombies' sequence (which, incidentally, is an example of how and why we should reject philosophical thought experiments, no matter how intuitively compelling they are, when they don't accord with established scientific theories and data) is deliberate. Rather than reading either sequence as an attempt to defend a specific fleshed-out theory of consciousness or of physical law, they should primarily be read as attempts to show that extreme uncertainty about a domain doesn't always bleed over into 'we don't know anything about this topic' or 'we can't rule out any of the candidate solutions'.

We can effectively rule out epiphenomenalism as a candidate solution to the hard problem of consciousness even if we don't know the answer to the hard problem (which we don't), and we can effectively rule out 'consciousness causes collapse' and 'there is no objective reality' as candidate solutions to the measurement problem in QM even if we don't know the answer to the measurement problem (which, again, we don't). Just advocating 'physicalism' or 'many worlds' is a promissory note, not a solution.

In discussions of EA and x-risk, we likewise need to be able to prioritize more promising hypotheses over less promising ones long before we've answered all the questions we'd like answered. Even deciding what studies to fund presupposes that we've 'philosophized', in the sense of mentally aggregating, heuristically analyzing, and drawing tentative conclusions from giant complicated accumulated-over-a-lifetime data sets.

You wrote:

The human capacity for abstract reasoning over mathematical models is in principle a fully general intelligent behaviour, as the scientific revolution has shown: there is no aspect of the natural world which has remained beyond the reach of human understanding, once a sufficient amount of evidence is available.

That's true, and it's one of the basic assumptions behind MIRI research: that understanding agents smarter than us isn't obviously hopeless, because our human capacity for abstract reasoning makes it possible for us to model systems even when they're extremely complex and dynamic. MIRI's research is intended to make this likelier to happen.

It's not the default that we're always able to predict what our inventions will do before we run them to see what happens; and there are some basic limits on our ability to do so when the system we're predicting is smarter than the predictor. But with enough intellectual progress we may become able to model abstract safety-relevant features of AGI behavior, even though we can't predict in detail the exact decisions the AGI will make. (If we could predict the exact decisions of the AGI, we'd have to be at least as smart as the AGI.)

If it isn't possible to learn a variety of generalizations about smarter autonomous systems, then, interestingly, that also undermines the case for intelligence explosion. Both 'humans trying to make superintelligent AI safe' and 'AI undergoing a series of recursive self-improvements' are cases where less intelligent agents are trying to reliably generate agents that meet various abstract criteria (including superior intelligence). The orthogonality thesis, likewise, simultaneously supports the claim 'many possible AI systems won't have humane goals' and 'it is possible for an AI system to have human goals'. This is why Bostrom/Yudkowsky-type arguments don't uniformly inspire pessimism.

Are you familiar with MIRI's technical agenda? You may also want to check out the AI Impacts project, if you think we should be prioritizing forecasting work at this point rather than object-level mathematical research.

Comment author: [deleted] 23 May 2015 02:35:25PM 2 points [-]

Are you familiar with MIRI's technical agenda? You may also want to check out the AI Impacts project, if you think we should be prioritizing forecasting work at this point rather than object-level mathematical research.

Yes I'm familiar with the technical agenda. What do you mean by "forecasting work"--AI impacts? That seems to be of near-zero utility to me.

What MIRI should be doing, what I've advocated MIRI to do from the start, and which I can't get a straight answer on why they are not doing that does not in some way terminate in referencing the more speculative sections of the sequences I take issue with, is this: build artificial general intelligence and study it. Not a provably-safe-from-first-principles-before-we-touch-a-single-line-of-code AGI. Just a regular, run of the mill AGI using any one of the architectures presently being researched in the artificial intelligence community. Build it and study it.

Comment author: RobbBB 23 May 2015 08:10:19PM *  5 points [-]

A few quick concerns:

  • The closer we get to AGI, the more profitable further improvements in AI capabilities become. This means that the more we move the clock toward AGI, the more likely we are to engender an AI arms race between different nations or institutions, and the more (apparent) incentives there are to cut corners on safety and security. At the same time, AGI is an unusual technology in that it can potentially be used to autonomously improve on our AI designs -- so that the more advanced and autonomous AI becomes, the likelier it is to undergo a speed-up in rates of improvement (and the likelier these improvements are to be opaque to human inspection). Both of these facts could make it difficult to put the brakes on AI progress.

  • Both of these facts also make it difficult to safely 'box' an AI. First, different groups in an arms race may simply refuse to stop reaping the economic or military/strategic benefits of employing their best AI systems. If there are many different projects that are near or at AGI-level when your own team suddenly stops deploying your AI algorithms and boxes them, it's not clear there is any force on earth that can compel all other projects to freeze their work too, and to observe proper safety protocols. We are terrible at stopping the flow of information, and we have no effective mechanisms in place to internationally halt technological progress on a certain front. It's possible we could get better at this over time, but the sooner we get AGI, the less intervening time we'll have to reform our institutions and scientific protocols.

  • A second reason speed-ups make it difficult to safely box an AGI is that we may not arrest its self-improvement in the (narrow?) window between 'too dumb to radically improve on our understanding of AGI' and 'too smart to keep in a box'. We can try to measure capability levels, but only using imperfect proxies; there is no actual way to test how hard it would be for an AGI to escape a box beyond 'put the AGI in the box and see what happens'. Which means we can't get much of a safety assurance until after we've done the research you're talking about us doing on the boxed AI. If you aren't clear on exactly how capable the AI is, or how well measures of its apparent capabilities in other domains transfer to measures of its capability at escaping boxes, there are limits to how confident you can be that the AI is incapable of finding clever methods to bridge air gaps, or simply adjusting its software in such a way the methods we're using to inspect and analyze the AI compromise the box.

  • 'AGI' is not actually a natural kind. It's just an umbrella term for 'any mind we could build that's at least as powerful as a human'. Safe, highly reliable AI in particular is likely to be an extremely special and unusual subcategory. Studying a completely arbitrary AGI may tell as about as much about how to build a safe AGI as studying nautilus ecology would tell us about how to safely keep bees and farm their honey. Yes, they're both 'animals', and we probably could learn a lot, but not as much as if we studied something a bit more bee-like. But in this case that presupposes that we understand AI safety well enough to build an AGI that we expect to look at least a little like our target safe AI. And our understanding just isn't there yet.

We already have seven billion general intelligences we can study in the field, if we so please; it's not obvious that a rushed-to-completion AGI would resemble a highly reliable safe AGI in all that much more detail than humans resemble either of those two hypothetical AGIs.

(Of course, our knowledge would obviously improve! Knowing about a nautilus and a squirrel really does tell us a lot more about beekeeping than either of those species would on its own, assuming we don't have prior experience with any other animals. But if the nautilus is a potential global catastrophic risk, we need to weigh those gains against the risk and promise of alternative avenues of research.)

Was any of that unclear?

Comment author: IlyaShpitser 23 May 2015 12:50:08PM 1 point [-]

unusually-difficult-to-test topic

No, MWI is not unusually difficult to test. It is untestable.

Comment author: RobbBB 24 May 2015 01:03:20AM *  2 points [-]

That's not true. (Or, at best, it's misleading for present purposes.)

First, it's important to keep in mind that if MWI is "untestable" relative to non-MWI, then non-MWI is also "untestable" relative to MWI. To use this as an argument against MWI, you'd need to talk specifically about which hypothesis MWI is untestable relative to; and you would then need to cite some other reason to reject MWI (e.g., its complexity relative to the other hypothesis, or its failures relative to some third hypothesis that it is testable relative to).

With that in mind:

  • 1 - MWI is testable insofar as QM itself is testable. We normally ignore this fact because we're presupposing QM, but it's important to keep in mind if we're trying to make a general claim like 'MWI is unscientific because it's untestable and lacks evidential support'. MWI is at least as testable as QM, and has at least as much supporting evidence.

  • 2 - What I think people really mean to say (or what a steel-manned version of them would say) is that multiverse-style interpretations of QM are untestable relative to each other. This looks likely to be true, for practical purposes, when we're comparing non-collapse interpretations: Bohmian Mechanics doesn't look testable relative to Many Threads, for example. (And therefore Many Threads isn't testable relative to Bohmian Mechanics, either.)

(Of course, many of the things we call "Many Worlds" are not fully fleshed out interpretations, so it's a bit risky to make a strong statement right now about what will turn out to be testable in the real world. But this is at least a commonly accepted bit of guesswork on the part of theoretical physicists and philosophers of physics.)

  • 3 - But, importantly, collapse interpretations generally are empirically distinguishable from non-collapse interpretations. So even though non-collapse interpretations are generally thought to be 'untestable' relative to each other, they are testable relative to collapse interpretations. (And collapse interpretations as a rule are falsifiable relative to each other.)

To date, attempts to test collapse interpretations have falsified the relevant interpretations. It is not technologically possible yet to test the most popular present-day ones, but it is possible for collapse theorists to argue 'our views should get more attention because they're easier to empirically distinguish', and it's also possible for anti-collapse theorists to try to make inductive arguments from past failures to the likelihood of future failures, with varying amounts of success.

Comment author: IlyaShpitser 24 May 2015 10:06:56AM *  4 points [-]

To use this as an argument against MWI

I don't have an argument against MWI specifically, no.

and you would then need to cite some other reason to reject MWI

No, that is not how it works: I don't need to either accept or reject MWI. I can also treat it as a causal story lacking empirical content. Nothing wrong with such stories, they are quite helpful for understanding systems. But not a part of science.

MWI is testable insofar as QM itself is testable.

By that logic, if I invent any crazy hypothesis in addition to an empirically testable theory, then it inherits testability just on those grounds. You can do that with the word "testabiity" if you want, but that seems to be not how people use words.

If some smart catholic says that evolution is how God unfolds creation when it comes to living systems, then any specific claims we can empirically check pertaining to evolution (including those that did not pan out, and required repairs of evolutionary theory) also somehow are relevant to the catholic's larger hypothesis? I suppose that is literally true, but silly. There is no empirical content to what this hypothetical catholic is saying, over and above the actual empirical stuff he is latching his baggage onto. I am not super interested in having catholic theologians read about minimum descriptive complexity, and then weaving a yarn about their favorite hypotheses based on that.

so it's a bit risky to make a strong statement right now about what will turn out to be testable in the real world

I like money! I am happy to discuss bet terms on this.

collapse interpretations generally are empirically distinguishable from non-collapse interpretations

Yes if you have an interpretation that gives different predictions than QM, then yes that will render that interpretation falsifiable of course (and indeed some were). That is super boring, though, and not what this argument is about. But also, I don't see what falsifiability of X has to do with falsifiabiliity of Y, if X and Y are different. Newtonian mechanics is both falsifiable and falsified, but that has little to do with falsifiability of any story fully consistent with QM predictions.


My personal take on MWI is I want to waste as little energy as possible on it and arguments about it, and actually go read Feynman instead. (This is not a dig at you, I am just explaining where I am coming from when it comes to physics).

Comment author: RobbBB 24 May 2015 07:02:50PM *  5 points [-]

No, that is not how it works: I don't need to either accept or reject MWI. I can also treat it as a causal story lacking empirical content.

To say that MWI lacks empirical content is also to say that the negation of MWI lacks empirical content. So this doesn't tell us, for example, whether to assign higher probability to MWI or to the disjunction of all non-MWI interpretations.

Suppose your ancestors sent out a spaceship eons ago, and by your calculations it recently traveled so far away that no physical process could ever cause you and the spaceship to interact again. If you then want to say that 'the claim the spaceship still exists lacks empirical content,' then OK. But you will also have to say 'the claim the spaceship blipped out of existence when it traveled far enough away lacks empirical content'.

And there will still be some probability, given the evidence, that the spaceship did vs. didn't blip out of existence; and just saying 'it lacks empirical content!' will not tell you whether to design future spaceships so that their life support systems keep operating past the point of no return.

By that logic, if I invent any crazy hypothesis in addition to an empirically testable theory, then it inherits testability just on those grounds. You can do that with the word "testabiity" if you want, but that seems to be not how people use words.

There's no ambiguity if you clarify whether you're talking about the additional crazy hypothesis, vs. talking about the conjunction 'additional crazy hypothesis + empirically testable theory'. Presumably you're imagining a scenario where the conjunction taken as a whole is testable, though one of the conjuncts is not. So just say that.

Sean Carroll summarizes collapse-flavored QM as the conjunction of these five claims:

  1. Quantum states are represented by wave functions, which are vectors in a mathematical space called Hilbert space.

  2. Wave functions evolve in time according to the Schrödinger equation.

  3. The act of measuring a quantum system returns a number, known as the eigenvalue of the quantity being measured.

  4. The probability of getting any particular eigenvalue is equal to the square of the amplitude for that eigenvalue.

  5. After the measurement is performed, the wave function “collapses” to a new state in which the wave function is localized precisely on the observed eigenvalue (as opposed to being in a superposition of many different possibilities).

Many-worlds-flavored QM, on the other hand, is the conjunction of 1 and 2, plus the negation of 5 -- i.e., it's an affirmation of wave functions and their dynamics (which effectively all physicists agree about), plus a rejection of the 'collapses' some theorists add to keep the world small and probabilistic. (If you'd like, you could supplement 'not 5' with 'not Bohmian mechanics'; but for present purposes we can mostly lump Bohm in with multiverse interpretations, because Eliezer's blog series is mostly about rejecting collapse rather than about affirming a particular non-collapse view.)

If we want 'QM' to be the neutral content shared by all these interpretations, then we can say that QM is simply the conjunction of 1 and 2. You are then free to say that we should assign 50% probability to claim 5, and maintain agnosticism between collapse and non-collapse views. But realize that, logically, either collapse or its negation does have to be true. You can frame denying collapse as 'positing invisible extra worlds', but you can equally frame denying collapse as 'skepticism about positing invisible extra causal laws'.

Since every possible way the universe could be adds something 'extra' on top of what we observe -- either an extra law (e.g., collapse) or extra ontology (because there are no collapses occurring to periodically annihilate the ontology entailed by the Schrodinger equation) -- it's somewhat missing the point to attack any given interpretation for the crime of positing something extra. The more relevant question is just whether simplicity considerations or indirect evidence helps us decide which 'something extra' (a physical law, or more 'stuff', or both) is the right one. If not, then we stick with a relatively flat prior.

Claims 1 and 2 are testable, which is why we were able to acquire evidence for QM in the first place. Claim 5 is testable for pretty much any particular 'collapse' interpretation you have in mind; which means the negation of claim 5 is also testable. So all parts of bare-bones MWI are testable (though it may be impractical to run many of the tests), as long as we're comparing MWI to collapse and not to Bohmian Mechanics.

(You can, of course, object that affirming 3-5 as fundamental laws has the advantage of getting us empirical adequacy. But 'MWI (and therefore also 'bare' QM) isn't empirically adequate' is a completely different objection from 'MWI asserts too many unobserved things', and in fact the two arguments are in tension: it's precisely because Eliezer isn't willing to commit himself to a mechanism for the Born probabilities in the absence of definitive evidence that he's sticking to 'bare' MWI and leaving almost entirely open how these relate to the Born rule. In the one case you'd be criticizing MWI theorists for refusing to stick their neck out and make some guesses about which untested physical laws and ontologies are the real ones; in the other case you'd be criticizing MWI theorists for making guesses about which untested physical laws and ontologies are the real ones.)

I am not super interested in having catholic theologians read about minimum descriptive complexity, and then weaving a yarn about their favorite hypotheses based on that.

Are you kidding? I would love it if theologians stopped hand-waving about how their God is 'ineffably simple no really we promise' and started trying to construct arguments that God (and, more importantly, the package deal 'God + universe') is information-theoretically simple, e.g., by trying to write a simple program that outputs Biblical morality plus the laws of physics. At best, that sort of precision would make it much clearer where the reasoning errors are; at worst, it would be entertainingly novel.

Comment author: IlyaShpitser 29 May 2015 01:16:16PM *  2 points [-]

To say that MWI lacks empirical content is also to say that the negation of MWI lacks empirical content.

Yes.

So this doesn't tell us, for example, whether to assign higher probability to MWI or to the disjunction of all non- MWI interpretations.

Right. But I think it's a waste of energy to assign probabilities to assertions lacking empirical content, because you will not be updating anyways, and a prior without possibility of data is just a slightly mathier way to formulate "your taste." I don't argue about taste.

[spaceship example]

One can assume a reasonable model here (e.g. leave a copy of the spaceship in earth orbit, or have it travel in a circle in the solar system, and assume similar degradation of modules). Yes, you will have indirect evidence only applicable due to your model. But I think the model here would have teeth.

But realize that, logically, either collapse or its negation does have to be true.

Or we are thinking about the problem incorrectly and those are not exhaustive/mutually exclusive. Compare: "logically the electron must either be a particle or must not be a particle."

it's somewhat missing the point to attack any given interpretation for the crime of positing something extra.

I am not explicitly attacking MWI, as I think I said multiple times. I am not even attacking having interpretations or preferring one over another for reasons such as "taste" or "having an easier time thinking about QM." I am attacking the notion that there is anything more to the preference for MWI than this.


To summarize my view: "testability" is about "empirical claims," not about "narratives." MWI is, by its very nature, a narrative about empirical claims. The list of empirical claims it is a narrative about can certainly differ from another list of empirical claims with another narrative. For example, we can imagine some sort of "billiard ball Universe" narrative around Newtonian physics.

But I would not say "MWI is testable relative to the Newtonian narrative", I would say "the list of empirical claims 'QM' is testable relative to the list of empirical claims 'Newtonian physics."

The problem with the former statement is first it is a "type error," and second, there are infinitely many narratives around any list of empirical claims. You may prefer MWI for [reasons] over [infinitely long list of other narratives], but it seems like "argument about taste." What's even the point of the argument?

Let's say I prefer some other interpretation of 'QM' than MWI. What does that say about me? Does it say anything bad? Am I a 'bad rationalist?' Do I have 'bad taste?' Does it matter what my favorite interpretation is? I think this lesswrongian MWI thing is belief-as-attire.


Full disclaimer: I am not a professional philosopher, and do not think about testability for a living. I reviewed a paper about testability once.

Comment author: RobbBB 29 May 2015 11:54:24PM *  0 points [-]

Could you restate your response to the spaceship example? This seems to me to be an entirely adequate response to

it's a waste of energy to assign probabilities to assertions lacking empirical content, because you will not be updating anyways, and a prior without possibility of data is just a slightly mathier way to formulate "your taste." I don't argue about taste.

Favoring simpler hypotheses matters, because if you're indifferent to added complexity when it makes no difference to your observations (e.g., 'nothing outside the observable universe exists') you may make bad decisions that impact agents that you could never observe, but that might still live better or worse lives based on what you do.

This matters when you're making predictions about agents far from you in space and/or time. MWI is a special case of the same general principle, so it's a useful illustrative example even if it isn't as important as those other belief-in-the-implied-invisible scenarios.

I am not even attacking having interpretations or preferring one over another for reasons such as "taste" or "having an easier time thinking about QM." I am attacking the notion that there is anything more to the preference for MWI than this.

Collapse and non-collapse interpretations are empirically distinguishable from each other. I've been defining 'QM' in a way that leaves it indifferent between collapse and non-collapse -- in which case you can't say that the distinction between bare QM and MWI is just a matter of taste, because MWI adds the testable claim that collapse doesn't occur. If you prefer to define 'QM' so that it explicitly rejects collapse, then yes, MWI (or some versions of MWI) is just a particular way of talking about QM, not a distinct theory. But in that case collapse interpretations of QM are incompatible with QM itself, which seems like a less fair-minded way of framing a foundations-of-physics discussion.

Comment author: IlyaShpitser 30 May 2015 05:26:50PM *  0 points [-]

Collapse and non-collapse interpretations are empirically distinguishable from each other.

You are not engaging with my claim that testability is a property of empirical claims, not narratives. Not sure there is a point to continue until we resolve the disagreement about the possible category error here.


There is another weird thing where you think we test claims against other claims, but actually we test against Nature. If Nature says your claim is wrong, it's falsified. If there is a possibility of Nature saying that, it's falsifiable. You don't need a pair of claims here. Testability is not a binary relation between claims. But that's not central to the disagreement.

Comment author: RobbBB 31 May 2015 08:37:27PM *  0 points [-]

Why do you think collapse interpretations are 'narratives', and why do you think they aren't empirical claims?

Regarding testability: if you treat testability as an intrinsic feature of hypotheses, you risk making the mistake of thinking that if there is no test that would distinguish hypothesis A from hypothesis B, then there must be no test that could distinguish hypothesis A from hypothesis C. It's true that you can just speak of a test that's better predicted by hypothesis 'not-A' than by hypothesis A, but the general lesson that testability can vary based on which possibilities you're comparing is an important one, and directly relevant to the case we're considering.

Comment author: IlyaShpitser 02 June 2015 11:41:19AM *  1 point [-]

There are two issues, what I view as non-standard language use, and what I view as a category error.

You can use the word 'testability' to signify a binary relation, but that's not what people typically mean when they use that word. They typically mean "possibility Nature can tell you that you are wrong."

So when you responded many posts back with a claim "MWI is hard to test" you are using the word "test" in a way probably no one else in the thread is using. You are not wrong, but you will probably miscommunicate.


An empirical claim has this form: "if we do experiment A, we will get result B." Nature will sometimes agree, and sometimes not, and give you result C instead. If you have a list of such claims, you can construct a "story" about them, like MWI, or something else. But adding the "story" is an extra step, and what Nature is responding to is not the story but the experiment.

The mapping from stories to lists of claims is always always always many to one. If you have [story1] about [list1] and [story2] about [list2], and Nature agrees with [list1], and disagrees with [list2], then you will say:

"story1 was falsified, story2 was falsifiable but not falsified."

I will say:

"list1 was falsified, list2 was falsifiable but not falsified."

What's relevant here isn't the details of story1 or story2, but what's in the lists.

When I say "MWI is untestable" what I mean is:

"There is a list of empirical claims called 'quantum mechanics.' There is a set of stories about this list, one of which is MWI. There is no way to tell these stories apart empirically, so you pick the one you like best for non-empirical reasons."

When you say "MWI is testable" what I think you mean is:

"There are two lists of empirical claims, called 'quantum mechanics' and 'quantum mechanics prime,' a story 'story 1' about the former, and a story 'story 2' about the latter. Nature will agree with the list 'quantum mechanics' and disagree with the list 'quantum mechanics prime.' Therefore, 'story 1' is testable relative to 'story 2.'"

That's fine, I understand what you mean, and I think you are right, up to the last sentence. But I think the last sentence is a category error.

Because you are equating lists of claims with stories, you are carrying over the testability property of the list 'quantum mechanics' to your favorite story about this list, 'MWI.' But there is an infinite list of stories consistent with 'quantum mechanics'. I can replace 'MWI' in your argument with any other consistent story, including those involving the flying spaghetti monster, etc.

Then you get unintuitive statements like 'the flying spaghetti interpretation of quantum mechanics is testable relative to X.' This is a sufficiently weird use of the word "testable" that I think we should not use the word "testable" in this way. And indeed, I believe the standard usage of the word "testable" is not this.

Comment author: Kaj_Sotala 24 May 2015 11:04:55PM 1 point [-]

At one point I started developing a religious RPG character who applied theoretical computer science to his faith.

I forget details, but among other details he believed that although the Bible prescribed the best way to live, the world is far too complex for any finite set of written rules to cover every situation. The same limitation applies to human reason: cognitive science and computational complexity theory have shown all the ways in which we are bounded reasoners, and can only ever hope to comprehend a small part of the whole world. Reason works best when it can be applied to constrained problems where clear objective answer can be found, but it easily fails once the number of variables grows.

Thus, because science has shown that both the written word of the Bible and human reason are fallible and easily lead us astray (though the word of the Bible is less likely to do so), the rational course of action for one who believes in science is to pray to God for guidance and trust the Holy Spirit to lead us to the right choices.

Comment author: TheAncientGeek 15 September 2017 02:11:02PM 0 points [-]

Many-worlds-flavored QM, on the other hand, is the conjunction of 1 and 2, plus the negation of 5

Plus 6: There is a preferred basis.

Comment author: gjm 15 September 2017 03:03:29PM 0 points [-]

In so far as I understand what the "preferred basis problem" is actually supposed to be, the existence of a preferred basis seems to me to be not an assumption necessary for Everettian QM to work but an empirical fact about the world; if it were false then the world would not, as it does, appear broadly classical when one doesn't look too closely. Without a preferred basis, you could still say "the wavefunction just evolves smoothly and there is no collapse"; it would no longer be a useful approximation to describe what happens in terms of "worlds", but for the same reason you could not e.g. adopt a "collapse" interpretation in which everything looks kinda-classical on a human scale apart from random jumps when "observations" or "measurements" happen. The world would look different in the absence of a preferred basis.

But I am not very expert on this stuff. Do you think the above is wrong, and if so how?

Comment author: TheAncientGeek 15 September 2017 02:04:14PM 0 points [-]

First, it's important to keep in mind that if MWI is "untestable" relative to non-MWI, then non-MWI is also "untestable" relative to MWI. To use this as an argument against MWI,

I think it's being used as an argument against beliefs paying rent.

MWI is testable insofar as QM itself is testable.

Since there is more than one interpretation of QM, empirically testing QM does not prove any one interpretation over the others. Whatever extra arguments are used to support a particular interpretation over the others are not going to be, and have not been, empirical.

But, importantly, collapse interpretations generally are empirically distinguishable from non-collapse interpretations.

No they are not, because of the meaning of the word "interpretation" but collapse theories, such as GRW, might be.

Comment author: TheAncientGeek 15 September 2017 01:48:34PM 0 points [-]

This is why there's a lot of emphasis on hard-to-test ("philosophical") questions in the Sequences, even though people are notorious for getting those wrong more often than scientific questions -- because sometimes [..] the answer matters a lot for our decision-making,

Which is one of the ways in which beliefs that don't pay rent do pay rent.

Comment author: knb 24 May 2015 12:18:12AM 3 points [-]

So if I understand correctly, you're leaving LW because you think LW is too hostile to AGI research and nanotechnology? I don't mind your decision to leave, but I'm not sure why you think this. My own impression is that a lot of us either feel unqualified to have an opinion or don't think AGI is likely any time soon.

I think you're way off if you believe that MIRI or LW are slowing down AI progress. I don't think MIRI/LW have that much reach, and AGI is likely decades away in any case. In fact, I don't even know of any work that MIRI has even argued should be stopped, let alone work they successfully stopped.

Comment author: JenniferRM 22 May 2015 07:31:40AM 6 points [-]

If you'll permit a restatement... it sounds like you surveyed the verbal output of the big names in the transhumanist/singularity space and classified them in terms of seeming basically "correct" or "mistaken".

Two distinguishing features seemed to you to be associated with being mistaken: (1) a reliance on philosophy-like thought experiments rather than empiricism and (2) relatedness to the LW/MIRI cultural subspace.

Then you inferred the existence of an essential tendency to "thought-experiments over empiricism" as a difficult to change hidden variable which accounted for many intellectual surface traits.

Then you inferred that this essence was (1) culturally transmissible, (2) sourced in the texts of LW's founding (which you have recently been reading very attentively), and (3) an active cause of ongoing mistakenness.

Based on this, you decided to avoid the continued influence of this hypothetical pernicious cultural transmission and therefore you're going to start avoiding LW and stop reading the founding texts.

Also, if the causal model here is accurate... you presumably consider it a public service to point out what is going on and help others avoid the same pernicious influence.

My first question: Am I summarizing accurately?

My second question assumed a yes and seeks information relevant to repair: Can you spell out the mechanisms by which you think mistake-causing reliance on thought experiment is promoted and/or transmitted? Is it an explicit doctrine? Is it via social copying of examples? Is it something else?

Comment author: [deleted] 22 May 2015 08:36:26AM *  3 points [-]

If you'll permit a restatement... it sounds like you surveyed the verbal output of the big names in the transhumanist/singularity space and classified them in terms of seeming basically "correct" or "mistaken".

This is certainly not correct. It was more like "providing evidence-based justifications for their beliefs, or hand-waving arguments." I'm not commenting on the truth of their claims, just the reported evidence supporting them.

(2) relatedness to the LW/MIRI cultural subspace.

I don't association with LW/MIRI cultural subspace is a bad thing. I'm a cryonicist and that is definitely also very much in line with LW/MIRI cultural norms. There are just particular positions espoused by MIRI and commonly held in this community which I believe to be both incorrect and harmful.

My first question: Am I summarizing accurately?

Other than the above, yes, I believe you have summarized correctly.

Can you spell out the mechanisms by which you think mistake-causing reliance on thought experiment is promoted and/or transmitted? Is it an explicit doctrine? Is it via social copying of examples? Is it something else?

It isn't LW-specific. The problem lies actually with non-analytic, or at the very least casual philosophy, which unfortunately has become the norm for debate here, thought it is not at all confined to LW. Reasoning by means of loose analogies and thought experiments is dangerous, in no small part because it is designed to trigger heuristics of proof-by-comparison which is in fact an insufficient condition to change one's mind. Finding a thought experiment or analogous situation can give credibility to a theory. In the best case it takes a theory from possible to plausible. However there is a gulf from plausible to probable and/or correct which we must be careful not to cross without proper evidence.

The solution to this, in the sciences, is rigorous peer review. I am honestly not sure how that transfers to the forum / communal blog format.

Comment author: CellBioGuy 24 May 2015 07:39:28PM *  2 points [-]

100% of my charitable donations are going to SENS. Why they do not get more play in the effective altruism community is beyond me.

Probably because they're unlikely to lead to anything special over and above general biology research.

Comment author: Kaj_Sotala 24 May 2015 11:21:50PM 0 points [-]

Funding for SENS might fund research that could be considered too speculative for more conventional bio funders, though.

Comment author: [deleted] 24 May 2015 10:44:57PM 0 points [-]

That's very cynical. What makes you say that?

Comment author: the-citizen 24 May 2015 07:20:45AM *  2 points [-]

mistakes that are too often made by those with a philosophical background rather than the empirical sciences: the reasoning by analogy instead of the building and analyzing of predictive models

While there are quite a few exceptions, most actual philosophy is not done through metaphors and analogies. Some people may attempt to explain philosophy that way, while others with a casual interest in philosophy might not known the difference, but few actual philosophers I've met are silly enough not to know an analogy is an analogy. Philosophy and empirical science aren't conflicting approaches or competing priorities. They interact and refine eachother in useful ways. For example philosophy may help improve reasoning where we have only limited evidence, or it may help us understand the appropriate way for evidence to be used, classified or interpreted. It's only problematic when its used for social purposes or motivated reasoning rather than challenging our own assumptions.

I think there are certain specific instances where LW's dominant philosophical interpretations are debatable, and I'd like to hear more of what your objections of those kind are.

I now regard the sequences as a memetic hazard, one which may at the end of the day be doing more harm than good

I think just being wrong or misleading (assuming you think the main thrust of the sequences is problematic), isn't enough to be a memetic hazard. Otherwise we'd be banning all sorts of stuff floating around in books and the internet. I suggest memetic hazard ought to be things that are uniquely dangerous in leading to immediate harm to mental health (suicide, extreme depression or extreme aggression).

Comment author: [deleted] 22 May 2015 11:03:01AM 2 points [-]

I think it is possible to use LW for generating testable hypotheses, though sadly testing would require lots of resources, but then it is usually so anyway. For example, I tried to see how LWers would estimate probabilities of statements for botanical questions, and there was even one volunteer. Well. Perhaps it would be more in-group to ask for probabilities for technical stuff - not AI or math, rather something broadly engineering that would still allow people to generate more than one alternative - and watch how they connect the dots and make assumptions explicit? People seem to find some questions easier than others, regardless of how right their answers. It would be relevant to teaching rationality to know how people decide upon this.

(Of course, I am not a specialist and all this might be solved already.)

Comment author: MrMind 22 May 2015 07:51:20AM *  5 points [-]

This argument is, however, nonsense. The human capacity for abstract reasoning over mathematical models is in principle a fully general intelligent behaviour

While it's true that humans are Turing complete, there does not exist only computability as a barrier to understanding.
Brains are, compared to some computers, quite slow and imperfect at storage. Let's say that the output of a super-intelligence would require to be understood, in human terms, the effort of a thousand-years-long computation written with the aid of a billion sheets of paper. While it would not be, in principle, unintelligible, it doesn't matter because nobody will ever understand it.
You can combine, if you want a more principled approach, Chaitin's theorem and Blum's speed-up theorem to show that, whatever complexity is intelligible for a human being, there's always a better machine whose output is, for that human, totally random.

My suspect is that you are making the same mistake you are accusing LWers: by reasoning by analogy, you see errors in what really are missing steps in your understanding.
What an irony.

Comment author: Gunnar_Zarncke 24 May 2015 07:52:46AM 4 points [-]

I recommend this for Main.

Comment author: jbay 24 May 2015 06:02:02AM *  2 points [-]

Hi Mark,

Thanks for your well-considered post. Your departure will be a loss for the community, and sorry to see you go.

I also feel that some of the criticism you're posting here might be due to a misunderstanding, mainly regarding the validity of thought experiments, and of reasoning by analogy. I think both of these have a valid place in rational thought, and have generally been used appropriately in the material you're referring to. I'll make an attempt below to elaborate.

Reasoning by analogy, or, the outside view

What you call "reasoning by analogy" is well described in the sequence on the outside view. However, as you say,

The fundamental mistake here is that reasoning by analogy is not in itself a sufficient explanation for a natural phenomenon, because it says nothing about the context sensitivity or insensitivity of the original example and under what conditions it may or may not hold true in a different situation.

This is exactly the same criticism that Eliezer has of outside-view thinking, detailed in the sequences!

In outside view as a conversation halter:

Of course Robin Hanson has a different idea of what constitutes the reference class and so makes a rather different prediction - a problem I refer to as "reference class tennis"[...] But mostly I would simply decline to reason by analogy, preferring to drop back into causal reasoning in order to make weak, vague predictions.

You're very right that the uncertainty in the AI field is very high. I hope that work is being done to get a few data points and narrow down the uncertainty, but don't think that you're the first to object to an over-reliance on "reasoning by analogy". It's just that when faced with a new problem with no clear reference class, it's very hard to use the outside vew, but unfortunately also hard to trust predictions from a model which has sensitive parameters with high uncertainties.

Thought experiments are a tool of deduction, not evidence

We get instead definitive conclusions drawn from thought experiments only.

This is similar to complaining about people arriving at definitive conclusions drawn from mathematical derivation only.

I want to stress that this is not a problem in most cases, especially not in physics. Physics is a field in which models are very general and held with high confidence, but often hard to use to handle complicated cases. We have a number of "laws" in physics that we have fairly high certainty of; nonetheless, the implications of these laws are not clear, and even if we believe them we may be unsure of whether certain phenomena are permitted by these laws or not. Of course we also do have to test our basic laws, which is why we have CERN and such, especially because we suspect they are incomplete (thanks in part to thought experiments!).

A thought experiment is not data, and you do not use conclusions from thought experiments to update your beliefs as though the thought experiment were producing data. Instead, you use thought experiments to update your knowledge of the predictions of the beliefs you already have. You can't just give an ordinary human the laws of physics written down on a piece of paper and expect them to immediately and fully understand the implications of the truth of those laws, or even to verify that the laws are not contradictory.

Thus, Einstein was able to use thought experiments very profitably to identify that the laws of classical mechanics (as formulated at the time) led to a contradiction with the laws of electrodynamics. No experimental evidence was needed; the thought experiment is a logical inference procedure which identifies one consequence of Maxwell's equations being that light travels at speed 'c' in all reference frames, and shows that to be incompatible with Galilean relativity. A thought experiment, just like a mathematical proof-by-contradiction, can be used to show that certain beliefs are mutually inconsistent and one must be changed or discarded.

Thus, I take issue with this statement:

(thought experiments favored over real world experiments)

Thought experiments are not experiments at all, and cannot even be compared to experiments. They are a powerful tool for exploring theory, and should be compared to other tools of theory such as mathematics. Experiments are a powerful tool for checking your theory, but experiments alone are just data; they won't tell you what your theory predicted, or whether your theory is supported or refuted by the data. Theory is a powerful tool for exploring the spaces of mutually compatible beliefs, but without data you cannot tell whether a theory has relevance to reality or not.

It would make sense to protest that thought experiments are being used instead of math, which some think is a more powerful tool for logical inference. On the other hand, math fails at being accessible to a wide audience, while thought experiments are. But the important thing is that thought experiments are similar to math in their purpose. They are not at all like experiments; don't get their purposes confused!

Within Less Wrong, I have only ever seen thought experiments used for illustrating the consequences of beliefs, not for being taken as evidence. For example, the belief that "humans have self-sabotaging cognitive flaws, and a wide variation of talents" and the belief that "humans are about as intelligent as intelligent things can get" would appear to be mutually incompatible, but it's not entirely obvious and a valid space to explore with thought experiments.

Comment author: Houshalter 22 May 2015 05:25:42PM 2 points [-]

Perhaps I'm wrong, but reading your comments, it seems like you mostly disagree with AI. You think that we should focus on developing AI as fast as possible and then boxing it and experimenting with it. Maybe you even go further and think all the work that MIRI is doing is a waste and irrelevant to AI.

If so I totally agree with you. You're not alone. I don't think you should leave the site over it though.

Comment author: TheAncientGeek 22 May 2015 09:14:23AM 2 points [-]

There's lip service done to empiricism throughout, but in all the “applied” sequences relating to quantum physics and artificial intelligence it appears to be forgotten.

It's kind of amazing that an organisation dedicated to empiricism didn't execute an initial phase of research to find out what AI researchers are actually doing. Why, it's almost as if it's intended to be a hobby horse.

"> I am no longer in good conscious

S/b conscience.

Comment author: ChristianKl 21 May 2015 08:17:44PM 2 points [-]

I wish I could recommend a skepticism, empiricism, and rationality promoting institute. Unfortunately I am not aware of an organization which does not suffer from the flaws I identified above.

It seems to me that CFAR engages into empiricism. They are trying to teach various different ways to make people more rational and they are willing to listen to the results and change teaching content and methods.

Is your main objection against them that till now they haven't published any papers?

Comment author: V_V 21 May 2015 10:24:21PM 4 points [-]

It seems to me that CFAR engages into empiricism. They are trying to teach various different ways to make people more rational and they are willing to listen to the results and change teaching content and methods.

How do they measure whether they are actually making people more rational?

There are hundreds, maybe thousands self-help/personal development groups in the world. From the secular ones (e.g. Landmark, which is in some ways the spiritual ancestor of CFAR), to traditional religions, to mumbo jumbo new age stuff. From the "outside view", how can I distinguish CFAR from these ones?

Comment author: ChristianKl 22 May 2015 12:13:22AM 1 point [-]

How do they measure whether they are actually making people more rational?

I think they did a bit of polling but I'm no good person to speak about the details. I haven't attended one of their courses yet.

From the "outside view", how can I distinguish CFAR from these ones?

A core difference is that CFAR intends to publish papers in the future that show effectiveness of techniques. The others that you listed don't. I also understand that doing work to getting techniques into a form were you can test them well in a study takes time.

In a lot of New Age frameworks there the belief that everything happens as it's supposed to be. If a person get's ill the day after a workshop, it's because they are processing negative emotions or karma. You can't do any science when you assume that any possible outcome of an experiment is by definition a good outcome and the challenge is about trusting that it's a good outcome.

The importance of trusting the process is also a core feature of traditional religion. If your prayer doesn't seem to be working, it's just because you don't understand how god moves in mysterious ways. Trust isn't inherently bad but it prevents scientific learning.

I don't know Landmarks position on trust and skepticism.

Landmark does practices like creating an expectation that participants invite guests to the Evening Session that I'm uncomfortable with. The might be effective recruiting tools but they feel culty.

Comment author: [deleted] 21 May 2015 10:05:12PM *  0 points [-]

So I know less about CFAR than I do the other two sponsors of the site. There is in my mind some unfortunate guilt by association given partially shared leadership & advisor structure, but I would not want to unfairly prejudice an organization for that reason alone.

However there are some worrying signs which is why I feel justified in saying something at least in a comment, with the hope that someone might prove me wrong. CFAR used donated funds to pay for Yudkowsky's time in writing HPMoR. It is an enjoyable piece of fiction, and I do not object to the reasoning they gave for funding his writing. But it is a piece of fiction whose main illustrative character suffers from exactly the flaw that I talked about above, in spades. It is my understanding also that Yudkowsky is working on a rationality textbook for release by CFAR (not the sequences which was released by MIRI). I have not seen any draft of this work, but Yudkowsky is currently 0 for 2 on this issue, so I'm not holding my breath. And given that further donations to CFAR are likely to pay for the completion of this work which has a cringe-inducing Bayesian prior, I would be hesitant to endorse them. That and, as you said, publications have been sparse or non-existent.

But I know very little other than that about CFAR and I remain open to having my mind changed.

Comment author: Valentine 22 May 2015 05:48:23PM *  17 points [-]

As Chief Financial Officer for CFAR, I can say all the following with some authority:

CFAR used donated funds to pay for Yudkowsky's time in writing HPMoR.

Absolutely false. To my knowledge we have never paid Eliezer anything. Our records indicate that he has never been an employee or contractor for us, and that matches my memory. I don't know for sure how he earned a living while writing HPMOR, but at a guess it was as an employed researcher for MIRI.

It is my understanding also that Yudkowsky is working on a rationality textbook for release by CFAR (not the sequences which was released by MIRI).

I'm not aware of whether Eliezer is writing a rationality textbook. If he is, it's definitely not with any agreement on CFAR's part to release it, and we're definitely not paying him right now whether he's working on a textbook or not.

And given that further donations to CFAR are likely to pay for the completion of this work…

Not a single penny of CFAR donations go into paying Eliezer.

I cannot with authority promise that will never happen. I want to be clear that I'm making no such promise on CFAR's behalf.

But we have no plans to pay him for anything to the best of my knowledge as the person in charge of CFAR's books and financial matters.

Comment author: [deleted] 22 May 2015 06:58:10PM *  3 points [-]

Thank you for correcting me on this.

So the source of the confusion is the Author's notes to HPMoR. Eliezer promotes both CFAR and MIRI workshops and donation drives, and is ambiguous about his full employment status--it's clear that he's a researcher at MIRI, but if was ever explicitly mentioned who was paying for his rationality work, I missed it. Googling "CFAR site:hpmor.com" does show that on http://hpmor.com/applied-rationality/, a page I never read he discloses not having a financial relationship with CFAR. But he notes many times elsewhere that "his employer" has been paying for him to write a rationality textbook, and at times given him paid sabbaticals to finish writing HPMOR because he was able to convince his employer that it was in their interest to fund his fiction writing.

As I said I can understand the argument that it would be beneficial to an organization like CFAR to have as fun and interesting an introduction to rationality as HPMOR is, ignoring for a moment the flaws in this particular work I pointed out elsewhere. It makes very little sense for MIRI to do so--I would frankly be concerned about them losing their non-profit status as a result, as writing rationality textbooks let alone harry potter fanfics is so, so far outside of MIRI's mission.

But anyway, it appears that I assumed it was CFAR employing him, not MIRI. I wonder if I was alone in this assumption.

EDIT: To be clear, MIRI and CFAR have shared history--CFAR is an offshoot of MIRI, and both organizations have shared offices and staff in the past. You staff page lists Eliezer Yudkowsky as a "Curriculum Consultant" and specifically mentions his work on HPMOR. I'll take your word that none of it was done with CFAR funding, but that's not the expectation a reasonable person might have from your very own website. If you want to distance yourself from HPMOR you might want to correct that.

Comment author: Valentine 24 May 2015 05:52:18PM 3 points [-]

To be clear, I can understand where your impression came from. I don't blame you. I spoke up purely to crush a rumor and clarify the situation.

I'll take your word that none of it was done with CFAR funding, but that's not the expectation a reasonable person might have from your very own website. If you want to distance yourself from HPMOR you might want to correct that.

That's a good point. I'll definitely consider it.

We're not trying to distance ourselves from HPMOR, by the way. We think it's useful, and it does cause a lot of people to show interest in CFAR.

But I agree, as a nonprofit it might be a good idea for us to be clearer about whom we are and are not paying. I'll definitely think about how to approach that.

Comment author: Mirzhan_Irkegulov 22 May 2015 05:57:48PM 1 point [-]

Crap, I had to notice that I am confused once I've read about CFAR paying to Eliezer. In the back of my head I thought “that's too much!”, but I shrugged it off and believed.

Thank you for pointing this out. I think Mark should edit his comment to make clear he erred.

Comment author: ChristianKl 21 May 2015 11:02:13PM 1 point [-]

But it is a piece of fiction whose main illustrative character suffers from exactly the flaw that I talked about above, in spades.

I was pleasantly surprised by empiricism in HPMOR.

It starts out with Harry's father believing that there's no way magic can exist, his mother believing it does and then Harry advocating using the empiric method to find out. Harry runs experiments to find out about the inheritance of magic. He runs experiments where he varies various factors to find out when a spell works with Hermione.

What's wrong with that kind of empiricism?

Comment author: [deleted] 21 May 2015 11:20:41PM *  12 points [-]

You have exhausted all of the examples that I can recall from the entire series. That's what's wrong.

The rest of the time Harry thinks up a clever explanation, and once the explanation is clever enough to solve all the odd constraints placed on it, (1) he stops looking for other explanations, and (2) he doesn't check to see if he is actually right.

Nominally, Harry is supposed to have learned his lesson in his first failed experimentation in magic with Hermoine. But in reality and in relation to the overarching plot, there was very little experimentation and much more "that's so clever it must be true!" type thinking.

"That's so clever it must be true!" basically sums up the sequence's justification for many-worlds, to tie us back to the original complaint in the OP.

Comment author: Kaj_Sotala 22 May 2015 12:49:50PM 18 points [-]

The rest of the time Harry thinks up a clever explanation, and once the explanation is clever enough to solve all the odd constraints placed on it, (1) he stops looking for other explanations, and (2) he doesn't check to see if he is actually right.

Examples:

Comed-tea in ch. 14

Hariezer decides in this chapter that comed-tea MUST work by causing you to drink it right before something spit-take worthy happens. The tea predicts the humor, and then magics you into drinking it. Of course, he does no experiments to test this hypothesis at all (ironic that just a few chapters ago he lecture Hermione about only doing 1 experiment to test her idea).

Wizards losing their power in chap. 22

Here is the thing about science, step 0 needs to be make sure you’re trying to explain a real phenomena. Hariezer knows this, he tells the story of N-rays earlier in the chapter, but completely fails to understand the point.

Hariezer and Draco have decided, based on one anecdote (the founders of Hogwarts were the best wizards ever, supposedly) that wizards are weaker today than in the past. The first thing they should do is find out if wizards are actually getting weaker. After all, the two most dangerous dark wizards ever were both recent, Grindelwald and Voldemort. Dumbledore is no slouch. Even four students were able to make the marauders map just one generation before Harry. (Incidentally, this is exactly where neoreactionaries often go wrong- they assume things are getting worse without actually checking, and then create elaborate explanations for non-existent facts).

Anyway, for the purposes of the story, I’m sure it’ll turn out that wizards are getting weaker, because Yudkoswky wrote it. But this would have been a great chance to teach an actually useful lesson, and it would make the N-ray story told earlier a useful example, and not a random factoid.

Atlantis in chap. 24

Using literally the exact same logic that Intelligent Design proponents use (and doing exactly 0 experiments), Hariezer decides while thinking over breakfast:

Some intelligent engineer, then, had created the Source of Magic, and told it to pay attention to a particular DNA marker.

The obvious next thought was that this had something to do with “Atlantis”.

Gateway to the after life in chap. 39:

Here is Hariezer’s response to the gateway to the afterlife:

That doesn’t even sound like an interesting fraud,“ Harry said, his voice calmer now that there was nothing there to make him hope, or make him angry for having hopes dashed. "Someone built a stone archway, made a little black rippling surface between it that Vanished anything it touched, and enchanted it to whisper to people and hypnotize them.”

Do you see how incurious Hariezer is? If someone told me there was a LITERAL GATEWAY TO THE AFTERLIFE I’d want to see it. I’d want to test it, see it. Can we try to record and amplify the whispers? Are things being said?

Laws of magic in chap. 85:

No surprise, then, that the wizarding world lived in a conceptual universe bounded - not by fundamental laws of magic that nobody even knew - but just by the surface rules of known Charms and enchantments…Even if Harry’s first guess had been mistaken, one way or another it was still inconceivable that the fundamental laws of the universe contained a special case for human lips shaping the phrase ‘Wingardium Leviosa’. …What were theultimate possibilities of invention, if the underlying laws of the universe permitted an eleven-year-old with a stick to violate almost every constraint in the Muggle version of physics?

You know what would be awesome? IF YOU GOT AROUND TO DOING SOME EXPERIMENTS AND EXPLORING THIS IDEA. The absolute essence of science is NOT asking these questions, it’s deciding to try to find out the fucking answers! You can’t be content to just wonder about things, you have to put the work in! Hariezer’s wonderment never gets past the stoned-college-kid wondering aloud and into ACTUAL exploration, and its getting really frustrating.

Vision and the invisibility cloak in chap. 95:

arry had set the alarm upon his mechanical watch to tell him when it was lunchtime, since he couldn’t actually look at his wrist, being invisible and all that. It raised the question of how his eyeglasses worked while he was wearing the Cloak. For that matter the Law of the Excluded Middle seemed to imply that either the rhodopsin complexes in his retina were absorbing photons and transducing them to neural spikes, or alternatively, those photons were going straight through his body and out the other side, but not both. It really did seem increasingly likely that invisibility cloaks let you see outward while being invisible yourself because, on some fundamental level, that was how the caster had - not wanted - but implicitly believed - that invisibility should work.

This would be an excellent fucking question to explore, maybe via some experiments. But no. I’ve totally given up on this story exploring the magic world in any detail at all. Anyway, Hariezer skips straight from “I wonder how this works” to “it must work this way, how could we exploit it"

Centaurs and astrology in chap. 101

Still in the woods, Hariezer encounters a centaur who tries to kill him, because he divines that Hariezer is going to make all the stars die.

There are some standard anti-astrology arguments, which again seems to be fighting the actual situation because the centaurs successfully use astrology to divine things.

We get this:

“Cometary orbits are also set thousands of years in advance so they shouldn’t correlate much to current events. And the light of the stars takes years to travel from the stars to Earth, and the stars don’t move much at all, not visibly. So the obvious hypothesis is that centaurs have a native magical talent for Divination which you just, well, project onto the night sky.”

There are so, so many other hypothesis Hariezer. Maybe starlight has a magical component that waxes and wanes as stars align into different magical symbols or some such. The HPMOR scientific method:

observation -> generate 1 hypothesis -> assume you are right -> it turns out that you are right.

Comment author: Eitan_Zohar 23 May 2015 09:34:19PM *  4 points [-]

Using literally the exact same logic that Intelligent Design proponents use (and doing exactly 0 experiments), Hariezer decides while thinking over breakfast:

Some intelligent engineer, then, had created the Source of Magic, and told it to pay attention to a particular DNA marker.

The obvious next thought was that this had something to do with “Atlantis”.

How is this 'literally the exact same logic that ID proponents use?' Creationists fallacize away the concept of natural selection, but I don't see how Harry is being unreasonable, given what he knows about the universe.

Comment author: [deleted] 24 May 2015 05:53:15PM *  4 points [-]

He's saying "I don't understand how magic could have come into being, it must have been invented by somebody." When in fact there could be dozens of other alternative theories.

I'll give you one that took me only three seconds to think up: the method for using magic isn't a delusion of the caster as Harry thought, but a mass delusion of all wizards everywhere. E.g. confounding every wizard in existence, or at least some threshold to think that Fixus Everthingus was a real spell would make it work. Maybe all it would have take to get his experiments with Hermoine to work is to confound himself as well, making it a double-blind experiment as it really should have been.

His argument here really is exactly the same as an intelligent designer: "magic is too complicated and arbitrary to be the result of some physical process."

Comment author: dxu 28 May 2015 01:27:22AM *  2 points [-]

His argument here really is exactly the same as an intelligent designer: "magic is too complicated and arbitrary to be the result of some physical process."

He actually does kind of address that, by pointing out that there are only two known processes that produce purposeful effects:

There were only two known causes of purposeful complexity. Natural selection, which produced things like butterflies. And intelligent engineering, which produced things like cars.

Magic didn't seem like something that had self-replicated into existence. Spells were purposefully complicated, but not, like a butterfly, complicated for the purpose of making copies of themselves. Spells were complicated for the purpose of serving their user, like a car.

Some intelligent engineer, then, had created the Source of Magic, and told it to pay attention to a particular DNA marker.

So, yeah, I disagree strongly that the two arguments are "exactly the same". That's the sort of thing you say more for emphasis than for its being true.

Comment author: [deleted] 28 May 2015 05:29:32AM *  -1 points [-]

I stand by my claim that they are the same.

An intelligent designer says "I have exhausted every possible hypothesis, there must be a god creator behind it all" when in fact there was at least one perfectly plausible hypothosis (natural selection) which he failed to thoroughly consider.

Harry says essentially "I have exhausted every possible hypothesis--natural selection and intelligent design--and there must be an Atlantean engineer behind it all" when in fact there were other perfectly plausible arguments such as the coordinated belief of a quorum of wizardkind explanation that I gave.

Comment author: dxu 28 May 2015 03:16:40PM *  2 points [-]

That doesn't address the question of why magic exists (not to mention it falls afoul of Occam's Razor). You seem to be answering a completely different question.

Comment author: Eitan_Zohar 24 May 2015 06:09:50PM *  1 point [-]

You may be right, but it is still more parsimonious than your idea (which requires some genuinely bizarre mechanism, far more than it being a self-delusion).

Comment author: [deleted] 24 May 2015 10:21:49PM *  0 points [-]

Not really. You've seen the movie Sphere, or read the book? Magic could be similar: the source of magic is a wish-granting device that makes whatever someone with wizard gene think of, actually happen. Of course this is incredibly dangerous--all I have to do is shout "don't think of the Apocalypse!" in a room of wizards and watch the world end. So early wizards like Merlin interdicted by using their magic to implant false memories into the entire wizarding population to provide a sort of basic set of safety rules -- magic requires wands, enchantments have to be said correctly with the right hand motion, creating new spells requires herculean effort, etc. None of that would be true, but the presence of other wizards in the world thinking it were true would be enough to make the wish-granting device enforce the rules anyway.

Comment author: RobbBB 22 May 2015 11:18:11PM *  3 points [-]

I think you're missing the point of the Many Worlds posts in the Sequences. I'll link to my response here.

Regarding HPMoR, Eliezer would agree that Harry's success rate is absurdly unrealistic (even for a story about witchcraft and wizardry). He wrote about this point in the essay "Level 2 Intelligent Characters":

In the real world, everything is harder than it is for characters in stories; clever insights are less likely to be true and clever strategems are overwhelmingly less likely to work. In real life, I have to try literally ten good ideas before one of them works at all, often putting in years of effort before giving up or succeeding. Yes, I’ve been known to pull off implausible tricks like “Write a Harry Potter fanfiction good enough to recruit International Mathematical Olympiad gold medalists” but that’s not the only implausible-sounding thing I’ve ever tried to do. You just don’t hear as much about the clever ideas that didn’t work, over the many years I’ve been trying weird and nonweird ways to get my task done.

In fiction you as the author can decide that the bright ideas do work, being careful to accompany this by an appropriate amount of sweat and pain and unintended consequence so that the reader feels the character has earned it. You cannot evade the curse of building your story out of clever ideas that would be far less likely to work in real life, not just because you have no way to test the ideas to find the ones that actually work, but because in real life we’re talking about a 10:1 ratio of failures to successes. We get to see Harry fail once in Ch. 22, because I felt like I had to make the point about clever ideas not always working. A more realistic story with eight more failed ideas passing before Harry’s first original discovery in Ch. 28 would not have been fun to read, or write.

I would agree with you, however, that HPMoR lets Harry intuit the right answer on the first guess too much. I would much prefer that the book prioritize pedagogy over literary directness, and in any case I have a taste for stories that meander and hit a lot of dead ends. (Though I'll grant that this is an idiosyncratic taste on my part.)

As a last resort, I think HPMoR could just have told us, in narration, about a bunch of times Harry failed, before describing in more detail the time he succeeded. A few sentences like this scattered throughout the story could at least reduce the message to system 2 that rationalist plans should consistently succeed, even if the different amounts of vividness mean that it still won't get through to system 1. But this is a band-aid; the deeper solution is just to find lots of interesting lessons and new developments you can tell about while Harry fails in various ways, so you aren't just reciting a litany of undifferentiated failures.

Comment author: Jiro 23 May 2015 10:22:49AM 3 points [-]

There's a difference between succeeding too often and succeeding despite not testing his ideas. The problem isn't having too many failed ideas, the problem is that testing is how one rules out a failed idea, so he seems unreasonably lucky in the sense that his refusal to test has unreasonably few consequences.

Comment author: drethelin 21 May 2015 08:47:36PM 1 point [-]

SENS is fundamentally in competition with dozens or hundreds of profit seeking organizations in the world. Donations to SENS are like donations to a charity researching better plastic surgery techniques. They will get invented no matter what, and the amount of money you can throw at it is trivial compared to the potential customer base of said techniques. If you cure aging, billionaires everywhere will fall over themselves to give you money.

Comment author: [deleted] 21 May 2015 11:48:25PM *  7 points [-]

The things that SENS is working on right now are not ready for investment. I'm going to call you out on this one: please name something SENS is or has researched which is or was the subject of private industry or taxpayer research at the time that SENS was working on it. I think you'll find that such examples, if they exist at all, are isolated. It is nevertheless the goal of SENS to create a vibrant rejuvenation industry with the private sector eventually taking the reins. But until then, there is a real need for a non-profit to fund research that is too speculative and/or too far from clinical trials to achieve return-on-investment on a typical funding horizon.

Comment author: Valentine 22 May 2015 06:03:07PM 3 points [-]

I generally quite agree with you here. I really enormously appreciate the effort SENS is putting into addressing this horror, and there does seem to be a hyperbolic discounting style problem with most of the serious anti-aging tech that SENS is trying to address.

But I think you might be stating your case too strongly:

please name something SENS is or has researched which is or was the subject of private industry or taxpayer research at the time that SENS was working on it. I think you'll find that such examples, if they exist at all, are isolated.

If I recall correctly, one of Aubrey's Seven Deadly Things is cancer, and correspondingly one of the seven main branches of SENS is an effort to eliminate cancer via an idea Aubrey came up with via inspiration. (I honestly don't remember the strategy anymore. It has been about six years since I've read Ending Aging.)

If you want to claim that no one else was working on Aubrey's approach to ending all cancers or that anyone else doing it was isolated, I think that's fair, but kind of silly. And obviously there's a ton of money going into cancer research in general, albeit I wouldn't be surprised if most of it was dedicated to solving specific cancers rather than all cancer at once.

But I want to emphasize that this is more of a nitpick on the strength of your claim. I agree with the spirit of it.

Comment author: [deleted] 22 May 2015 07:21:07PM *  2 points [-]

What I'm saying is the actual research project being funded by SENS are those which are not being adequately funded elsewhere. For example, stem cell therapy is one of the seven pillars of the SENS research agenda, but SENS does almost no work on this whatsoever because it is being adequately funded elsewhere. Likewise, cancer forms another pillar of SENS research, but to my knowledge SENS has only worked on avenues of early-stage research that is not being pursued elsewhere, like the case you mentioned.

I interpreted drethelin's comment as saying that donating to SENS was a waste of money since it's a drop in the bucket compared to for-profit and government research programs. My counter-point is that for-profit and public programs are not pursuing the same research as SENS is doing.

Comment author: ChristianKl 22 May 2015 09:05:09PM 1 point [-]

And obviously there's a ton of money going into cancer research in general, albeit I wouldn't be surprised if most of it was dedicated to solving specific cancers rather than all cancer at once.

I think the consensus in the field is at the moment that cancer isn't a single thing. Therefore "solve all cancer at once" unfortunately doesn't make a good goal.

Comment author: Valentine 22 May 2015 10:17:42PM 2 points [-]

That's my vague impression too. But if I remember correctly, the original idea of OncoSENS (the part of SENS addressing cancer) was something that in theory would address all cancer regardless of type.

I also seem to recall that most experimental biologists thought that many of Aubrey's ideas about SENS, including OncoSENS, were impractical and that they betrayed a lack of familiarity with working in a lab. (Although I should note, I don't really know what they're talking about. I, too, lack familiarity with working in a lab!)

Comment author: ChristianKl 23 May 2015 01:19:45AM 2 points [-]

But if I remember correctly, the original idea of OncoSENS (the part of SENS addressing cancer) was something that in theory would address all cancer regardless of type.

I just reread the OncoSENS page.

The idea was to nuke the gene for telomerase from every cell in the body and also nuke a gene for alternative lengthening of telomeres (ALT).

Nuking out telomerase in every cell doesn't need further cancer research but gene therapy research. Albeit most gene therapy is about adding gene's instead of deleting them.

As far as research funding goes SENS seems to be currently funding research into ALT (http://www.sens.org/research/intramural/the-alt-mechanism). I think it's plausible that ALT research is otherwise underfunded but I don't know the details.

Comment author: Mirzhan_Irkegulov 21 May 2015 10:53:45PM 3 points [-]

But is that true that rich people all over the world actually steer the anti-aging research that much? Because they don't even throw money at cryonics, and I haven't heard that rich people massively started freezing themselves. Even if some third-world dictators say they want to be immortal, they don't go more than one small step far.

Comment author: Vaniver 21 May 2015 09:02:44PM 1 point [-]

SENS is fundamentally in competition with dozens or hundreds of profit seeking organizations in the world.

Sort of. It might be nice if, say, a cure for diabetes was owned by a non-profit that released it into the public domain rather than by a corporation that would charge for it. (Obviously forcing corporations to be non-profits has predictably terrible side effects, and so the only proper way to do this is by funding participants in the race yourself.) From the perspective of the for-profits, a single non-profit competitor only slightly adjusts their environment, and so may not significantly adjust their incentive structure.

They will get invented no matter what

It would seem that the sooner a treatment is developed / cure found, the less people will suffer or die from the disease. Moving that sooner seems like a valuable activity.

Comment author: Gleb_Tsipursky 24 May 2015 02:07:51AM 0 points [-]

Sorry to see you going, your contrarian voice will be missed.

Wanted to mention that Intentional Insights is a nonprofit specifically promoting rationality for the masses, including encouraging empirical and evidence-based approaches (I'm the President). So consider recommending us in the future, and get in touch with me at gleb@intentionalinsights.org if you want to discuss this.

Comment author: gjm 15 September 2017 04:17:26PM 0 points [-]

Wanted to mention [...] Intentional Insights

Of course you did.

Comment author: ChristianKl 24 May 2015 10:00:25PM 0 points [-]

Wanted to mention that Intentional Insights is a nonprofit specifically promoting rationality for the masses, including encouraging empirical and evidence-based approaches (I'm the President).

What did you do from which you learned empirically? Which assumptions that you had when you started, have you managed to falsify?

Comment author: Gleb_Tsipursky 25 May 2015 12:45:37AM 0 points [-]

Well, one thing we learned from empirically was that the most responsive audience to our content was in the secular community. We assumed that people into health and wellness would be more responsive, but our content is not sufficiently "woo" for most. So we decided to focus our early efforts on spreading rationality among the secular community, and then branch out later after we develop solid ties there.

Comment author: halcyon 27 May 2015 12:50:43PM 0 points [-]

The problem with your point regarding chimpanzees is that it is true only if the chimpanzee is unable to construct a provably friendly human. This is true in the case of chimpanzees because they are unable to construct humans period, friendly or unfriendly, but I don't think it has been established that present day humans are unable to construct a provably friendly superintelligence.

Comment author: [deleted] 27 May 2015 03:24:50PM 1 point [-]

That's wholly irrelevant. The important question is this: which can be constructed faster: a provably-safe-by-design friendly AGI, or a fail-safe not-proven-friendly tool AI? Lives hang in the balance: about 100,000 a day to be exact.

(There's an aside about whether an all-powerful "friendly" AI outcome is even desirable--I don't think it is. But that's a separate issue.)

Comment author: Vaniver 27 May 2015 04:34:10PM *  2 points [-]

which can be constructed faster: a provably-safe-by-design friendly AGI, or a fail-safe not-proven-friendly tool AI?

Mark, I get that it's terrible that people are dying. As pointed out in another thread, I support SENS. But there's a disaster response tool called "Don't just do something, stand there!", which argues that taking the time to make sure you do the right thing is worth it, especially in emergencies, when there is pressure to act too soon. Mistakes made because you were in a hurry aren't any less damaging because you were in a hurry for a good reason.

I don't think anyone expects that it's possible and desirable to slow down general tech development, and most tool AI is just software development. If I write software that helps engineers run their tools more effectively, or a colleague writes software that helps doctors target radiation at tumors more effectively, or another colleague writes software that helps planners decide which reservoirs to drain for electrical power, that doesn't make a huge change to the trajectory of the future; each is just a small improvement towards more embedded intelligence and richer, longer lives.

Comment author: halcyon 27 May 2015 03:36:57PM *  0 points [-]

So you don't think the invention of AI is inevitable? If it is, shouldn't we pool our resources to find a formula for friendliness before that happens?

How could you possibly prevent it from occurring? If you stop official AI research, that will just mean gangsters will find it first.

I mean, computing hardware isn't that expensive, and we're just talking about stumbling across patterns in logic here. (If an AI cannot be created that way, then we are safe regardless.) If you prevent AI research, maybe the formula won't be discovered in 50 years, but are you okay with an unfriendly AI within 500?

(In any case, I think your definition of friendliness is too narrow. For example, you may disagree with EY's definition of friendliness, but you have your own: Preventing people from creating an unfriendly AI will take nothing less than intervention from a friendly AI.)

(I should mention that unlike many LWers, I don't want you to feel at all pressured to help EY build his friendly AI. My disagreement with you is purely intellectual. For instance, if your friendliness values differ from his, shouldn't you set up your own rival organization in opposition to MIRI? Just saying.)

Comment author: Eitan_Zohar 23 May 2015 09:11:57PM *  0 points [-]

Philosophy is getting too much flak, I think. It didn't take me a lot of effort to realize that any correct belief we have puts us on some equal footing with the AI.

Comment author: [deleted] 24 May 2015 05:56:12PM 0 points [-]

You're anthropomorphizing far too much. It's possible for things which are easy for us to think to be very difficult for an AGI, if it is constructed in a different way.

Comment author: Eitan_Zohar 24 May 2015 06:04:40PM *  0 points [-]

Um... I don't follow this at all. Where do I anthropomorphize? I think that the concept of 'knowledge' would be pretty universal.

Comment author: [deleted] 24 May 2015 06:22:28PM 0 points [-]

Hrm. Maybe I'm reading you wrong? I thought you were making a commonly made argument that any belief we arrive at, the artificial intelligence would as well. And my response was that because the AI is running different mental machinery, it is entirely possible that there are beliefs we arrive at which the AI just doesn't consider and vice versa. Were you saying something different?

Comment author: Eitan_Zohar 24 May 2015 06:44:24PM *  0 points [-]

You, quite rightly, criticized the notion that an AI would have just as much of an epistemic advantage on humans that we would have on a cow. Any correct notions we have are just that- correct. It's not like there's more truth to them that a godlike intelligence could know.

Comment author: bortels 01 June 2015 05:51:07AM 0 points [-]

I now regard the sequences as a memetic hazard, one which may at the end of the day be doing more harm than good.

To your own cognition, or just to that of others?

I just got here. I have no experience with the issues you cite, but it strikes me that disengagement does not, in general, change society. If you think ideas, as presented, are wrong - show the evidence, debate, fight the good fight. This is probably one of the few places it might actually be acceptable - you can't lurk on religious boards and try to convince them of things, they mostly cannot or will not listen, but I suspect/hope most here do?

I actually agree, a lot of the philosophy tips over to woo woo and sophistry - but it is perhaps better to light a candle than curse the dark.

what is taught by the sequences is a form of flawed truth-seeking (thought experiments favored over real world experiments) which inevitably results in errors,

Well - let's fix it then! I tend to agree, I see rationalism as only one of many useful tools. I would add formal Logic, and Science (refinement via experiment - do those sequences actually suggest that experiment is unnecessary somehow? I'd love to see it, could use the laugh. ) and perhaps even foggy things like "experience" (I find I do not control, to a large extent, my own problem solving and thought processes nearly as well as I would imagine). The good carpenter has many tools, and uses the appropriate ones at the appropriate time.

Or is this one of those academic "we need to wait for the old guard to die off" things? If so, again, providing a counterpoint for those interested in truth as opposed to dogma seems like a fun thing to do. But I'm weird that way. (I strongly believe in the value of re-examination of what we hold true, for refinement or discarding if it no longer fits reality, as well as for personal growth - so the idea of sequences that people take as gospel of sorts is simply argument from authority to mock unless it stands up to critical analysis)

But within the LessWrong community there is actually outright hostility to...

Ghandi said "First they ignore you, then they laugh at you, then they fight you, then you win."

...but he was pretty pragmatic for a philosopher. If you get hostility to ideas, that means they're listening, which means you actually have some chance of causing reform, if that is a goal. If you are not willing to piss off a few people in the name of truth... well, I understand. We get tired, and human beings do not generally seek confrontation continually (or at least the ones who survive and reproduce do not). But if your concern is that they are hostile toward ideas that more effectively help humanity, disengagement isn't gonna change that, although it may help your own sanity.