I suppose this highlights risks from a general increase in credibility-giving by close temporal association with other new ideas you're giving credibility to?

Yes, this.

Usually this risk is low, but here it was actually quite high. This particular instance was an Ugly example, because the category - ideas with close temporal association - was false. But there were many scary examples based on good categories. The most outlandish was meditation. Remember that other people's brains are part of evidence, now witness quite a few people who have just spent the last few days on activities that convinced you they are pretty decent (compared to baseline, damn good) at doing their research, discarding bullshit, not strongly espousing ideas they don't strongly hold, examining the ideas they do hold, etc. etc... witness them say with a straight face that meditation, which you (I) assumed was a crock of mystic religion that just took a different turn than the Western religions you're familiar with... witness them say that meditation is super-useful. Then watch your brain say "Bull! Wait, they're good at things. Maybe not bull? Hey, argument from authority, bull after all! Wait, argument from authority is evidence... :S I... have to take this seriously..."

IFS, NVC, nootropics? Guess I have to take them seriously too.

(I exaggerate slightly, but my feelings were stronger than I think they should have been, so that story is in line with how I felt, if not precisely what my beliefs were)

I suppose this highlights risks from a general increase in credibility-giving by close temporal association with other new ideas you're giving credibility to?

Yes, this.

Usually this risk is low, but here it was actually quite high. This particular instance was an Ugly example, because the category - ideas with close temporal association - was false. But there were many scary examples based on good categories. The most outlandish was meditation. Remember that other people's brains are part of evidence, now witness quite a few people who have just spent the last few days on activities that convinced you they are pretty decent (compared to baseline, damn good) at doing their research, discarding bullshit, not strongly espousing ideas they don't strongly hold, examining the ideas they do hold, etc. etc... witness them say with a straight face that meditation, which you (I) assumed was a crock of mystic religion that just took a different turn than the Western religions you're familiar with... witness them say that meditation is super-useful. Then watch your brain say "Bull! Wait, they're good at things. Maybe not bull? Hey, argument from authority, bull after all! Wait, argument from authority is evidence... :S I... have to take this seriously..."

IFS, NVC, nootropics? Guess I have to take them seriously too.

(I exaggerate slightly, but my feelings were stronger than I think they should have been, so that story is in line with how I felt, if not precisely what my beliefs were)

If you're still in doubt, go anyway. I put the probability of self-damage due to attending minicamp at extremely low, compared to self-damage from attending your standard college level economics lecture or a managerial business skills improvement workshop. It doesn't even blip on a radar calibrated to the kind of self-damage you could do speculatively attending religious retreats.

What about the cost? I would not call spending $1500 in a week insignificant. And as a baseline, I believe that being surrounded for a week by a group of people who believe strongly in some collection of ideas is a risk at least an order of magnitude higher than an economics lecture. I certainly expect that it would have a much stronger effect on me (as it seems it has had on you) than the lecture would, and I would most certainly not take a risk of this magnitude if I have any non-negligible doubts.

Here are some points, as I think of them.

The Good

• It was fun. On par with the best one-week vacations I've had, but less fun than Hawaii.
• I look significantly better, directly caused by the fashion workshops. My sister was briefly jealous because while I usually won in the raw-g department, she had always handily won in the looking-good department, and this is no longer true.
• I took to heart Try Things, (hypothesis) directly caused by in-person admonitions by high-status instructors. Previously I had focused far too much on exploitation over exploration. Concrete example: I went to a code retreat with Arlo Belshee last weekend and my roommate did not, while any normal reference class would have said he was more likely to go, and it was super-useful.
• I actually applied (and am applying) Gendlin and Tarski to the scary object of my inner mental life. I recommend Internal Family Systems as very useful though I have no direct comparisons I can make. If it turns out that it's actively harmful or even significantly worse than average mainstream psychotherapy I will update strongly towards standard-retreat-rather-than-awesome.
• Directly after the minicamp, I made several time-management changes to my lifestyle that have persisted until now, giving me many effective extra hours per week. Concrete example: noticing that the marginal return on playing Dominion online was negative past about the first 10% of my time spent, and actually cutting way back.
• Noticing when I'm being too abstract that I should try to provide or ask for a concrete example. ;) This was very much caused by admonitions during minicamp and some supporting reminders afterwards by John.

The Bad

• The minicamp became more and more disorganized toward the end. IIRC this was because the instructors were updating as they went and thought they discovered that they could do better in the second half than originally planned, but the fact of disorganization stands.
• Several of the exercises we did felt artificial and did not feel like they hit their intended target. Concrete example: when we played the game of arguing for an answer instead of reasoning towards an answer, to feel what rationalization felt like, I just felt like I was playing a game, nothing like when I actually notice rationalization.
• I felt like there was too much stuff to learn and not enough practicing stuff. By a wide enough margin that I'd list this here.

The Ugly

• I came unsettlingly close to not looking for counterevidence like this when presented with investment advice. Not actually too close, but it was still unsettling. Constant vigilance!

Edit: formatting, also the investment advice did not come from SI, but from one of the attendees.

I attended the 2011 minicamp.

It's been almost a year since I attended. The minicamp has greatly improved me along several dimensions.

1. I now dress better and have used techniques provided at minicamp to become more relaxed in social situations. I'm more aware of how I'm expressing my body language. It's not perfect control and I've not magically become an extrovert, but I'm better able to interact in random social situations successfully. Concretely: I'm able to sit and stand around people I don't know and feel and present myself as relaxed. I dress better and people have noticed and I've received multiple comments to that effect. I've chosen particular ways to present myself and now I get comments like 'you must play the guitar' (this has happened five times since minicamp haha). This is good since it loads the initial assumptions I want the person to load.

2. I've intentionally hacked my affectation towards various things to better reach my goals. For years I never wanted to have children. My wife said (earlier this year, after minicamp) that she wanted to have kids. I was surprised and realized that given various beliefs (love for wife, more kids good for society, etc) I needed to bring my emotions and affectations in line with those goals. I did this by maximizing positive exposure to kids and focusing on the good experiences...and it worked. I'm sure nature helped, but I came to a change of emotional reaction that feels very stable. TMI: I had my vasectomy reversed and am actively working on building kid version 1.0

3. Minicamp helped me develop a better mental language for reasoning around rationalist principles. I've got tools for establishing mental breakpoints (recognizing states of surprise, rationalization, etc) and a sense for how to improve on weak areas in my reasoning. I have a LOT of things I still need to improve. Many of my actions still don't match my beliefs. The up side is that I'm aware of many of the gaps and can make progress toward solving them. There seems to be only so much I can change at once, so I've been prioritizing everything out.

4. I've used the more concise, direct reasoning around rationality at my job at Valve Software. I use it to help make better decisions, concretely: when making decisions around features to add to DOTA 2 I've worked particularly hard at quickly relinquishing failed ideas that I generated. I have developed litanies like 'my ideas are a product, not a component of my identity.' Before I enter into interactions I pause and think 'what is my goal for this interaction? The reasoning tools from minicamp have helped me better teach and interpret the values of my company (which are very similar). I helped write a new employee guide that captures Valve values, but uses tools such as Anna Salamon's "Litany for Simplified Bayes" to cut straight to the core concepts. "If X is true, what would the world look like?" "If X is not true, what would the world look like?" "What does the world look like?" I've been influential in instituting predictions meetings before we launch new features.

5. I've been better able to manage my time, because I'm more aware of the biases and pitfalls that lie before me. I think more about what 'BrandonReinhart2020' wants than what the current me wants. (Or at least, my best guess as to what I think he would want...like not being dead, and being a bad ass guitar shredder, etc). This has manifested itself concretely in my self-education around the guitar. When I went to minicamp I had only just started learning guitar. Since then I've practiced 415 hours (I work full time, so this is all in my spare time) and have developed entirely new skills. I can improv, write songs, etc. Minicamp provided some inspiration, yes, but there were also real tools that I've employed. A big one was coming home and doing research on human learning and practice. This helped me realize that my goals were achievable. Luke gave sessions on how to do efficient research. Critch gave a session on hacking your affectations. I used this to make practice something I really, really like doing (I listened to music I liked before practicing, I would put objects like role-playing books or miniatures that I liked around my practice area -- nerdy yes, but it worked for me -- and I would drink a frosty beer after practicing three hours in a row. Okay so that last one shows that my health beliefs and goals may not be entirely in line, but it served an objective here). Now I can easily practice for 3 hours and enjoy every moment of it. (This is important, before I would use that time for World of Warcraft and other pursuits that just wasted time and didn't improve me.)

I've been in the Less Wrong orbit for a long time and have had the goal of improving my rationality for a long time. I've read Yudkowsky's writing since the old SL4 days. I followed Overcoming Bias from the beginning. I can't say that I had a really good grasp on which concepts were the most important until after minicamp. There's huge value in being able to ask questions, debate a point, and just clarify your confusion quickly.

I have also been an SIAI skeptic. Both myself and John Salvatier thought that SIAI might be a little religion-like. Our mistake. The minicamp was a meeting of really smart people who wanted to help each other win more. The minicamp was genuinely about mental and social development and the mastery of concepts that seem to lead to a better ability to navigate complex decision trees toward desired outcomes.

While we did talk about existential risk, the SIAI never went deep into high shock level concepts that might alienate attendees. It wasn't an SIAI funding press. It wasn't a AGI press. In fact, I thought they almost went too light on this subject (but I came to modern rationality from trans/posthumanism and most people in the future will probably get to trans/posthumanism from modern rationality, so discussions about AGI and such feels normal to me). Point being if you have concerns about this you'll feel a lot better as you attend.

I would say the thing that most discomforted me during the event was the attitude toward meditation. I realized, though, that this was an indicator about my preconceptions about meditation and not necessarily due to facts about meditation. After talking to several people about meditation, I learned that there wasn't any funky mysticism inherent to meditation, just closely associated to meditation. Some people are trying to figure out if it can be used as a tool and are trying to figure out ways to experiment around it, etc. I updated away from 'meditation is a scary religious thing' toward 'meditation might be another trick to the bag.' I decided to let other people bear the burden/risk of doing the research there, though. :)

Some other belief shifts related to minicamp: I have greatly updated toward the Less Wrong style rationality process as being legitimate tools for making better decisions. I have updated a great deal toward the SIAI being a net good for humanity. I have updated a great deal toward the SIAI being led by the right group of people (after personal interactions with Luke, Anna, and Eliezer).

Comparing minicamp to a religious retreat seems odd to me. There is something exciting about spending time with a bunch of very smart people, but it's more like the kind of experience you'd have at a domain-specific research summit. The experience isn't to manipulate through repeated and intense appeals to emotion, guilt, etc (I was a Wesleyan Christian when I was younger and went to retreats like Emaeus and I still remember them pressing a nail sharply into my palm as I went to the altar to pray for forgiveness). It's more accurate to think of minicamp as a rationality summit, with the instructors presenting findings, sharing techniques for the replication of those findings, and there being an ongoing open discussion of the findings and the process used to generate findings. And like any good Summit there are parties.

If you're still in doubt, go anyway. I put the probability of self-damage due to attending minicamp at extremely low, compared to self-damage from attending your standard college level economics lecture or a managerial business skills improvement workshop. It doesn't even blip on a radar calibrated to the kind of self-damage you could do speculatively attending religious retreats.

If you're a game developer, you would probably improve your ability to make good decisions around products more by attending SIAI Minicamp than you would by attending GDC (of course, GDC is still valuable for building a social network within the industry).

My guess as to why cryonics is the best method currently available is that it's the best bet for keeping you potentially alive until better methods are developed.

Speaking just for myself, I'm not convinced that any method can give immortality (Murphy happens), but there's a good bit of hope for greatly extended lifespans.

Cryonics is useful for preserving your body until a method for immortality is developed. It is not, on it's own such a method.

If you die now, cryonics is the only method available to give you a chance at immortality.

Cryonics is either not an answer, or the only answer, depending on what exactly you mean by the question.

Your company plan sounds very much like how Valve is structured. You may find it challenging to maintain your desired organizational structure, given that you also plan to be dependent on external investment. Also, starting a company with the express goal of selling it as quickly as possible conflicts with several ways you might operate your company to achieve a high degree of success. Many of the recent small studios that have gone on to generate large amounts of revenue (relative to their size) (Terraria / Minecraft / etc) are independently owned and build 'service based' software that seeks to keep the community engaged.

Alexei, I would suggest an alternative and encourage you to apply to Valve. 1) It wouldn't take much of your time to try. 2) It may help you reach your goals more quickly. 3) You don't have to invest in building a rational organization, (which is costly and hard) since one already exists.

It would be a career oriented decision (few people leave Valve once they start there) and I know you are interested in applying yourself to existential risk as completely as you can, but you should consider the path of getting really good at satisfying the needs of customers and then through that, directing resources towards the existential risk problem. It may feel like you are less engaged, because you aren't there solving the hard problems yourself -- and if you have a high degree of confidence that you are the one to solve those problems then maybe you should pursue a direct approach -- but it is a path you should give serious thought to.

I wouldn't advise you to go work at any random company. Most game companies -- particularly large ones -- are structured in a way that doesn't mean you'd have a good chance of individual success (versus working anywhere else or doing something else).

Valve has one of the highest profits per employees in the world and is wholly owned by the employees. The company compensates very well. So my advice is specific to considering an application to Valve.

Carl Shulman has been convincing that I should do nothing directly (in terms of labor) on the problem on AI risks, instead become successful elsewhere, and then direct resources as I am able toward the problem.

However I believe I should 1) continue to educate myself on the topic 2) try to learn to be a better rationalist so when I do have resources I can direct them effectively 3) work toward being someone who can gain access to more resources 4) find ways to better optimize my lifestyle.

At one point I seriously considered running off to San Fran to be in the thick of things, but I now believe that would have been a strictly worse choice. Sometimes the best thing you can do is to do what you already do well and hope to direct the proceeds towards helping people. Even when it feels like this is selfish, disengaged, or remote.

I don't know why you would assume that it's "probably as a result of trying harder to avoid cultishness." My prior is that they just don't seem cultish because academics are often expected to hold unfamiliar positions.