Thanks for writing this up!

As a participant, I think the claim that MSFP was a resounding success is a little strong. It's not at all clear to me that anyone gained new skills by attending (at least, I don't feel like I did), as distinct from learning about new ideas, using their existing skills, becoming convinced of various positions, and making social connections (which are more than enough to explain the new hires). To me it was an interesting experiment whose results I find hard to evaluate.

Thanks for the detailed update! Donated $1,500.

You're absolutely right. CfAR could get statistics by measuring quantifiable goals across its students: Grade point average, wealth, weight-loss; preferably with a control group. Until then, I'm just looking for any info I can get.

A question that has been asked before, and so may be stupid: What concrete examples are there of gains from CfAR training (or self-study based on LessWrong)? These would have to come in the form of very specific examples, preferably quantitative.

E.g. "I was $100,000 in debt and unemployed for 2 years, and now I have employment earning twice what I ever have before and am out of debt."

"I never had a relationship that lasted more than 2 months, but now am happily married."

"My grade point average went up from 2.2 to 3.8"

"After struggling to diet and exercise for years, I finally got on track and am now in the best shape of my life."

etc.

Sure. Say you have to make some decision now, and you will be asked to make a decision later about something else. Your decision later may depend on your decision now as well as part of the world that you don't control, and you may learn new information from the world in the meantime. Then the usual way of rolling all of that up into a single decision now is that you make your current decision as well as a decision about how you would act in the future for all possible changes in the world and possible information gained.

This is vaguely analogous to how you can curry a function of multiple arguments. Taking one argument X and returning (a function of one argument Y that returns Z) is equivalent to taking two arguments X and Y and returning X.

There's potentially a huge computational complexity blowup here, which is why I stressed mathematical equivalence in my posts.

What sort of things do you think experts (professors, generally) might value that a less-expert person like myself might be able to offer?

Lots of things, of course: adoration, bacon, sexual favors, etc. etc. :-D

In practice, I suspect that some attention, gratefulness, and a demonstration that you're not a clueless idiot with some agenda will go a long way towards making the expert willing to answer your questions. The last part is the problematic one in online communications -- by default you're just "another guy from the internet" and we all know what the average of that looks like.

However something in this vein seems like not a bad start to me: "Dear Professor X, I read your papers/books Y and Z and was amazed how you figured out A, B, and C. However I have a question about D because while E it seems to me that F." Demonstrate cluefulness and use flattery :-)

P.S. Another important issue is scope. Ask questions that can be concisely answered in a couple of paragraphs. Do not ask questions the answers to which are a graduate degree, a shelf of books, and a really tall stack of printed-out papers (e.g. "What should I eat for health and fitness?").

A third possibility is that AGI becomes the next big scare.

There's always a market for the next big scare, and a market for people who'll claim putting them in control will save us from the next big scare.

Having the evil machines take over has always been a scare. When AI gets more embodied, and start working together autonomously, people will be more likely to freak, IMO.

Getting beat on Jeopardy is one thing, watching a fleet of autonomous quad copters doing their thing is another. It made me a little nervous, and I'm quite pro AI. When people see machines that seem like they're alive, like they think, communicate among themselves, and cooperate in action, many will freak, and others will be there to channel and make use of that fear.

That's where I disagree with EY. He's right that a smarter talking box will likely just be seen as an nonthreatening curiosity. Watson 2.0, big deal. But embodied intelligent things that communicate and take concerted action will press our base primate "threatening tribe" buttons.

"Her" would have had a very different feel if all those AI operating systems had bodies, and got together in their own parallel and much more quickly advancing society. Kurzweil is right in pointing out that with such advanced AI, Samantha could certainly have a body. We'll be seeing embodied AI well before any human level of AI. That will be enough for a lot of people to get their freak out on.

Does MIRI have a prediction market on this stuff?

Let me rephrase: would you like to describe your arguments against utility functions in more detail?

For example, as I mentioned, there's an obvious mathematical equivalence between making a plan at the beginning of time and planning as you go, which is directly analogous to how one converts games from extensive-form to normal-form. As such, all aspects of acquiring information is handled just fine (from a mathematical standpoint) in the setup of vNM.

The standard response to the discussion of knowing probabilities exactly and to concerns about computational complexity (in essence) is that we may want to throw aside epistemic concerns and simply learn what we can from a theory that is not troubled by them (a la air resistance in physics..)? Is your objection essentially that those factors are more dominant in human morality than LW acknowledges? And if so, is the objection to the normal-form assumption essentially the same?

You are of course correct about the concrete scenario of being Dutch Booked in a hypothetical gamble (and I am not a gambler for reasons similar to this: we all know the house always wins!). However, if we're going to discard the Dutch Book criterion, then we need to replace it with some other desiderata for preventing self-contradictory preferences that cause no-win scenarios.

Even if your own mind comes preprogrammed with decision-making algorithms that can go into no-win scenarios under some conditions, you should recognize those as a conscious self-patching human being, and consciously employ other algorithms that won't hurt themselves.

I mean, let me put it this way, probabilities aside, if you make decisions that form a cyclic preference ordering rather than even forming a partial ordering, isn't there something rather severely bad about that?