Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: diegocaleiro 29 November 2015 11:04:21AM 4 points [-]

Yes I am.

Step 1: Learn Bayes

Step 2: Learn reference class

Step 3: Read 0 to 1

Step 4: Read The Cook and the Chef

Step 5: Reason why are the billionaires saying the people who do it wrong are basically reasoning probabilistically

Step 6: Find the connection between that and reasoning from first principles, or the gear hypothesis, or whichever other term you have for when you use the inside view, and actually think technically about a problem, from scratch, without looking at how anyone else did it.

Step 7: Talk to Michael Valentine about it, who has been reasoning about this recently and how to impart it at CFAR workshops.

Step 8: Find someone who can give you a recording of Geoff Anders' presentation at EAGlobal.

Step 9: Notice how all those steps above were connected, become a Chef, set out to save the world. Good luck!

Comment author: endoself 30 November 2015 07:06:53AM *  2 points [-]

I model probabilistic thinking as something you build on top of all this. First you learn to model the world at all (your steps 3-8), then you learn the mathematical description of part of what your brain is doing when it does all this. There are many aspects of normative cognition that Bayes doesn't have anything to say about, but there are also places where you come to understand what your thinking is aiming at. It's a gears model of cognition rather than the object-level phenomenon.

If you don't have gears models at all, then yes, it's just another way to spout nonsense. This isn't because it's useless, it's because people cargo-cult it. Why do people cargo-cult Bayesianism so much? It's not the only thing in the sequences. The first post, The Simple Truth, big parts of Mysterious Answers to Mysterious Questions, and basically all of Reductionism are about the gears-model skill. Even the name rationalism evokes Descartes and Leibniz, who were all about this skill. My own guess is that Eliezer argued more forcefully for Bayesianism than for gears models in the sequences because, of the two, it is the skill that came less naturally to him, and that stuck.

What would cargo-cult gears models look like? Presumably, scientism, physics envy, building big complicated models with no grounding in reality. This too is a failure mode visible in our community.

Comment author: Yaacov 26 July 2015 04:57:04AM *  13 points [-]

Hi LW! My name is Yaacov, I've been lurking here for maybe 6 months but I've only recently created an account. I'm interested in minimizing human existential risk, effective altruism, and rationalism. I'm just starting a computer science degree at UCLA, so I don't know much about the topic now but I'll learn more quickly.

Specific questions:

What can I do to reduce existential risk, especially that posed by AI? I don't have an income as of yet. What are the best investments I can make now in my future ability to reduce existential risk?

Comment author: endoself 27 July 2015 09:48:37PM 4 points [-]

Hi Yaacov!

The most active MIRIx group is at UCLA. Scott Garrabrant would be happy to talk to you if you are considering research aimed at reducing x-risk. Alternatively, some generic advice for improving your future abilities is to talk to interesting people, try to do hard things, and learn about things that people with similar goals do not know about.

Comment author: joaolkf 08 April 2015 05:48:50PM *  3 points [-]

Worth mentioning that some parts of Superintelligence are already a less contrarian version of many arguments made here in the past.

Also note that although some people do believe that FHI is some sense "contrarian", when you look at the actual hard data on this the fact is FHI has been able to publish in mainstream journals (within philosophy at least) and reach important mainstream researchers (within AI at least) at rates comparable, if not higher, to excellent "non-contrarian" institutes.

Comment author: endoself 09 April 2015 12:24:36AM *  2 points [-]

Yeah, I didn't mean to contradict any of this. I wonder how much a role previous arguments from MIRI and FHI played in changing the zeitgeist and contributing to the way Superintelligence was received. There was a slow increase in uninformed fear-of-AI sentiments over the preceding years, which may have put people in more of a position to consider the arguments in Superintelligence. I think that much of this ultimately traces back to MIRI and FHI; for example many anonymous internet commenters refer to them or use phrasing inspired by them, though many others don't. I'm more sceptical that this change in zeitgeist was helpful though.

Of course specific people who interacted with MIRI/FHI more strongly, such as Jaan Tallinn and Peter Thiel, were helpful in bring the discourse to where it is today.

Comment author: IlyaShpitser 08 April 2015 06:38:53AM 4 points [-]

At least Ng's career though can be credited to Hawkins.

'At least a part'? Also,


Comment author: endoself 08 April 2015 08:36:57AM *  1 point [-]

The quote from Ng is

The big AI dreams of making machines that could someday evolve to do intelligent things like humans could, I was turned off by that. I didn’t really think that was feasible, when I first joined Stanford. It was seeing the evidence that a lot of human intelligence might be due to one learning algorithm that I thought maybe we could mimic the human brain and build intelligence that’s a bit more like the human brain and make rapid progress. That particular set of ideas has been around for a long time, but [AI expert and Numenta cofounder] Jeff Hawkins helped popularize it.

I think it's pretty clear that he would have worked on different things if not for Hawkins. He's done a lot of work in robotics, for example, so he could have continued working on robotics if he didn't get interested in general AI. Maybe he would have moved into deep learning later in his career, as it started to show big results.

Even when contrarians win, they lose: Jeff Hawkins

13 endoself 08 April 2015 04:54AM

Related: Even When Contrarians Win, They Lose

I had long thought that Jeff Hawkins (and the Redwood Center, and Numentia) were pursuing an idea that didn't work, and were continuing to fail to give up for a prolonged period of time. I formed this belief because I had not heard of any impressive results or endorsements of their research. However, I recently read an interview with Andrew Ng, a leading machine learning researcher, in which he credits Jeff Hawkins with publicizing the "one learning algorithm" hypothesis - the idea that most of the cognitive work of the brain is done by one algorithm. Ng says that, as a young researcher, this pushed him into areas that could lead to general AI. He still believes that AGI is far though.

I found out about Hawkins' influence on Ng after reading an old SL4 post by Eliezer and looking for further information about Jeff Hawkins. It seems that the "one learning algorithm" hypothesis was widely known in neuroscience, but not within AI until Hawkins' work. Based on Eliezer's citation of Mountcastle and his known familiarity with cognitive science, it seems that he learned of this hypothesis independently of Hawkins. The "one learning algorithm" hypothesis is important in the context of intelligence explosion forecasting, since hard takeoff is vastly more likely if it is true. I have been told that further evidence for this hypothesis has been found recently, but I don't know the details.

This all fits well with Robin Hanson's model. Hawkins had good evidence that better machine learning should be possible, but the particular approaches that he took didn't perform as well as less biologically-inspired ones, so he's not really recognized today. Deep learning would definitely have happened without him; there were already many people working in the field, and they started to attract attention because of improved performance due to a few tricks and better hardware. At least Ng's career though can be credited to Hawkins.

I've been thinking about Robin's hypothesis a lot recently, since many researchers in AI are starting to think about the impacts of their work (most still only think about the near-term societal impacts rather than thinking about superintelligence though). They recognize that this shift towards thinking about societal impacts is recent, but they have no idea why it is occurring. They know that many people, such as Elon Musk, have been outspoken about AI safety in the media recently, but few have heard of Superintelligence, or attribute the recent change to FHI or MIRI.

Comment author: Sherincall 18 February 2015 08:51:28PM *  11 points [-]

Reddit is giving away 10% of their ad revenue to 10 charities that receive the most votes from the community. You can vote for as many charities as you want, with any account that has been created before 10AM PST today.

You can vote for your favorite charities here. I've had problems with the search by name, so if you don't find something, try searching by EIN instead.

Quick links: CFAR, MIRI

Comment author: endoself 18 February 2015 10:09:01PM *  9 points [-]

GiveWell, GiveDirectly, Evidence Action/Deworm the World. You can vote for multiple charities.

Comment author: endoself 24 October 2014 04:23:01AM 47 points [-]

I took the survey.

Comment author: Wei_Dai 22 July 2014 06:17:44AM 3 points [-]

This looks more like a problem with updating than with MMEU though. It seems possible to design a variant of UDT that uses MMEU, without it wanting to self-modify into something else (at least not for this reason).

Comment author: endoself 22 July 2014 06:53:51AM 3 points [-]

I can't see how this would work. Wouldn't the UDT-ish approach be to ask an MMEU agent to pick a strategy once, before making any updates? The MMEU agent would choose a strategy that makes it equivalent to a Bayesian agent, as I describe. The characteristic ambiguity-averse behaviour only appears if the agent is allowed to update.

Given a Cartesian boundary between agent and environment, you could make an agent that prefers to have its future actions be those that are prescribed by MMEU, and you'd then get MMEU-like behaviour persisting upon reflection, but I assume this isn't what you mean since it isn't UDT-ish at all.

Comment author: endoself 22 July 2014 05:01:59AM *  6 points [-]

MMEU isn't stable upon reflection. Suppose that in addition to the mysterious [0.4, 0.6] coin, you had a fair coin, and I tell you that all offer bet 1 ("pay 50¢ to be payed $1.10 if the coin came up heads") if the fair coin comes up heads and bet 2 if the fair coin comes up tails, but you have to choose whether to accept or reject before flipping the fair coin to decide which bet will be chosen. In this case, the Knighian uncertainty cancels out, and your expected winnings are +5¢ no matter which value is [0.4, 0.6] is taken to be the true probabilty of the mysterious coin, so you would take this bet on MMEU.

Upon seeing how the fair coin turns out, however, MMEU would tell you to reject whichever of bets 1 and 2 is offered. Thus, if I offer to let you see the result of the fair coin before deciding whether to accept the bet, you will actually prefer not to see the coin, for an expected outcome of +5¢, rather than see the coin, reject the bet, and win nothing with certainty. Alternatively, if given the chance, you would prefer to self-modify so as to not exhibit ambiguity aversion in this scenario.

In general, any agent using a decision rule that is not generalized Bayesian performs strictly worse than some generalized Bayes decision rule. Note, though, that this does not mean that such an agent is forced to accept at least one of bets 1 and 2, since rejecting whichever of them is offered is a Bayes rule; for example, a Bayesian agent who believes that the bookie knows something that they don't will behave in this way. It does mean, though, that there are many situations where MMEU cannot work, such as in my example above, since in such scenarios it is not equivalent to any Bayes rule.

Comment author: Manfred 21 July 2014 10:44:08PM 7 points [-]

Does anyone know how much people are typically willing to pay to switch options in the Ellsberg paradox? Among those that would pay to switch, my expectation is around 10%, not the ~50% predicted by max-min. This sort of mild ambiguity aversion is probably better captured by prospect theory.

Comment author: endoself 22 July 2014 04:57:24AM 6 points [-]

This is a very general point. Most of the uncertainty people face is of the sort that they would naively classify as Knighian, so if people actually behaved according to MMEU, then they would essentially be playing minimax against the world.

View more: Next