it seems you've been unable to escape a certain amount of reasoning by analogy in your post. You state that experimental investigation of asteroid impacts was useful, so by analogy, experimental investigation of AI risks should be useful.

It seems I should have picked a different phrase to convey my intended target of ire. The problem isn't concept formation by means of comparing similar reference classes, but rather using thought experiments as evidence and updating on them.

To be sure, thought experiments are useful for noticing when you are confused. They can also be semi-dark art in providing intuition pumps. Einstein did well in introducing special relativity by means of a series of thought experiments, by getting the reader to notice their confusion over classical electromagnetism in moving reference frames, then providing an intuition pump for how his own relativity worked in contrast. It makes his paper one of the most beautiful works in all of physics. However it was the experimental evidence which proved Einstein right, not the gedankenexperimenten.

If a thought experiment shows something to not feel right, that should raise your uncertainty about whether your model of what is going on is correct or not (notice your confusion), to whit the correct response should be “how can I test my beliefs here?” Do NOT update on thought experiments, as thought experiments are not evidence. The thought experiment triggers an actual experiment—even if that experiment is simply looking up data that is already collected—and the actual experimental results is what updates beliefs.

My impression is that MIRI thinks most possible AGI architectures wouldn't meet its standards for safety.

MIRI has not to my knowledge released any review of existing AGI architectures. If that is their belief, the onus is on them to support it.

but note that Eliezer Yudkowsky from MIRI was the one who invented the AI box experiment

He invented the AI box game. If it's an experiment, I don't know what it is testing. It is a setup totatly divorced from any sane reality for how AGI might actually develop and what sort of controls might be in place, with built-in rules that favor the AI.

Yet nevertheless, time and time again people such as yourself point me to the AI box games as if it demonstrated anything of note, anything which should cause me to update my beliefs.

It is, I think, the examples of the sequences and the character of many of the philosophical discussions which happen here that drive people to feel justified in making such ungrounded inferences. And it is that tendency which possibly makes the sequences and/or less wrong a memetic hazard.

And obviously there's a ton of money going into cancer research in general, albeit I wouldn't be surprised if most of it was dedicated to solving specific cancers rather than all cancer at once.

I think the consensus in the field is at the moment that cancer isn't a single thing. Therefore "solve all cancer at once" unfortunately doesn't make a good goal.

The things that SENS is working on right now are not ready for investment. I'm going to call you out on this one: please name something SENS is or has researched which is or was the subject of private industry or taxpayer research at the time that SENS was working on it. I think you'll find that such examples, if they exist at all, are isolated. It is nevertheless the goal of SENS to create a vibrant rejuvenation industry with the private sector eventually taking the reins. But until then, there is a real need for a non-profit to fund research that is too speculative and/or too far from clinical trials to achieve return-on-investment on a typical funding horizon.

So I know less about CFAR than I do the other two sponsors of the site. There is in my mind some unfortunate guilt by association given partially shared leadership & advisor structure, but I would not want to unfairly prejudice an organization for that reason alone.

However there are some worrying signs which is why I feel justified in saying something at least in a comment, with the hope that someone might prove me wrong. CFAR used donated funds to pay for Yudkowsky's time in writing HPMoR. It is an enjoyable piece of fiction, and I do not object to the reasoning they gave for funding his writing. But it is a piece of fiction whose main illustrative character suffers from exactly the flaw that I talked about above, in spades. It is my understanding also that Yudkowsky is working on a rationality textbook for release by CFAR (not the sequences which was released by MIRI). I have not seen any draft of this work, but Yudkowsky is currently 0 for 2 on this issue, so I'm not holding my breath. And given that further donations to CFAR are likely to pay for the completion of this work which has a cringe-inducing Bayesian prior, I would be hesitant to endorse them. That and, as you said, publications have been sparse or non-existent.

But I know very little other than that about CFAR and I remain open to having my mind changed.

I wouldn't update a lot or revise too much based on this report. The simple fact is that there was so much packed into 4 days that there was just no way anyone could remember it all. I suspect different attendees understood, remembered, implemented, and forgot different subsets of the material.

I will note that it is extremely helpful to have the spiral bound notebook with detailed notes from the sessions. I've been skimming it every couple of weeks just to jog my memory, and give me ideas about what I should be working on. Usually I just toss these handouts after a conference or workshop, but this one's been really helpful.

Without making concrete plans, I'd like to offer some ideas off-line, PM me?

This is incredibly helpful. Thank you! This part stood out to me:

Similarly I completely did not understand the concepts of inside view vs. outside view at the workshop; and worse yet I don't think that I even realized that I didn't understand these. However now that I've read Thinking Fast and Slow, the lightbulb has gone on. Inside view is simply me deciding how likely I (or my team) is likely to accomplish something based on my judgement of the problem and our capabilities. Outside view is a statistical question about how people and teams like us have done when confronted with similar problems in the past. As long as there are similar teams and similar problems to compare with, the outside view is likely to be much more accurate.

This and the similar comment about having forgotten the Planning Kata makes for really useful feedback. Yours is not the only report we've recently collected like this, but I think this is the most detailed. We'll revise this and test some revisions. Thank you!

Not that I know of

Any advice on how to set one up? In particular how to add entries to it retrospectively - I was thinking about searching the comments database for things like "I intend to", "guard against", "publication bias" etc. and manually find the relevant ones. This is somewhat laborious, but the effect I want to avoid is "oh I've just finished my write-up (or am just about to), now I'll go and add the original comment to the anti-publication bias registry".

On the other hand it seems like anyone can safely add anyone else's comment to the registry as long as it's close enough in time to when the comment was written.

Any advice? (I figured if you're involved at CFAR you might know a bit about this stuff).

This is useful to me as I'll be attending the March workshop. If I successfully digest any of the insights presented here then I'll have a better platform to start from. (Two particular points are the stuff about the parasympathetic nervous system, which I'd basically never heard of before, and the connection between the concepts of "epistemic rationality" and "knowing about myself" which is more obvious-in-retrospect).

Thanks for the write-up!

And yes, I'll stick up at least a brief write-up of my own after I'm done. Does LW have an anti-publication-bias registry somewhere?

I was waiting for Val to answer this, but I'll give it a shot. The relevant CFAR unit is called "againstness." You can think of sympathetic dominance as related to (being?) a sensation of "againstness," e.g. when you get angry during a heated argument your feelings are directed against the person you're arguing with. Val gave us both mental and physical techniques for releasing againstness ("fighting againstness" is kind of againsty). The mental techniques (which I'm just going to quote verbatim from the worksheet; hopefully Val won't mind):

• Gratitude: appreciate the opportunity to practice releasing againstness in adverse situations. (Also, appreciate the pleasure of PNS activation to help stabilize it.)
• When againstness is related to others:
  • Empathy: model the other person in enough detail to appreciate likely reasons why what they're doing is, to them, the most sensible thing they could be doing in that moment.
  • Connection: intentionally increase your sense of tribal togetherness; see the other person as a fellow human being and wish to help them realize their full potential.

That might sound a little woo but the above is intended to be a description of specific mental algorithms that you can actually run. The physical techniques:

• Open your posture. Shoulders back, spine straight & upright, head balanced on spine, belly exposed.
• Breathe. Deeply, smoothly, gently, and low - without pushing it low or sucking air in.
• Relax. Especially the hands, arms, shoulders, and eyes. (Smiling sometimes helps!)

I've found gratitude together with the physical techniques to be reasonably effective and have used them several times since the workshop already. I have not extensively tried using empathy or connection.

As for tests, one of the reasons it was valuable to learn this material at the workshop is that Val is very good at spotting the physical indicators of sympathetic dominance. Accordingly, the againstness unit had a practical component where Val or one of the other instructors would stress out participants in various ways in order for them to practice using the techniques above, and Val would diagnose to what extent the techniques were working. So testing yourself doesn't sound easy to me. If you just want some tips for noticing when your SNS is dominant, try looking for the following:

• hunched shoulders
• rubbing the neck
• positioning arms to protect the belly
• tensed muscles

Unfortunately I don't think it's easy to notice that you're doing these things.

Each of the CFAR worksheets also included a list of further resources. For the againstness unit, the further resources were the Wikipedia articles and two papers:

Fredrickson, B.L., Mancuso, R.A., Branigan, C., & Tugade, M.M. (2000). The undoing effect of positive emotions. Motivation and Emotion. 24, 237-258. http://goo.gl/AP920

Heinrichs M., von Dawans B., and Domes G. (2009). Oxytocin, vasopressin, and human social behavior. Frontiers in Neuroendocrinology, 30, 548-557. http://goo.gl/kGaz6