All of Mass_Driver's Comments + Replies

I just came here to point out that even nuclear weapons were a slow takeoff in terms of their impact on geopolitics and specific wars. American nuclear attacks on Hiroshima and Nagasaki were useful but not necessarily decisive in ending the war on Japan; some historians argue that the Russian invasion of Japanese-occupied Manchuria, the firebombing of Japanese cities with massive conventional bombers, and the ongoing starvation of the Japanese population due to an increasingly successful blockade were at least as influential in the Japanese decision to sur... (read more)

I suspect we're talking about two different things. 

If you just naively program a super-intelligent AI to satisfice a goal, then, sure, most of the candidate pathways to satisfice will involve accruing a lot of some type of power, because power is useful for achieving goals. That's a valid point, and it's important to understand that merely switching from optimizers to satisficers won't adequately protect us against overly ambitious AIs.

However, that doesn't mean that it's futile to explicitly penalize most (but not literally all) of the paths that th... (read more)

Sure, the metaphor is strained because natural selection doesn't have feelings, so it's never going to feel satisfied, because it's never going to feel anything. For whatever it's worth, I didn't pick that metaphor; Eliezer mentions contraception in his original post.

As I understand it, the point of bringing up contraception is to show that when you move from one level of intelligence to another, much higher level of intelligence, then the more intelligent agent can wind up optimizing for values that would be anathema to the less intelligent agents, even i... (read more)

Right, I'm not claiming that AGI will do anything like straightforwardly maximize human utility. I'm claiming that if we work hard enough at teaching it to avoid disaster, it has a significant chance of avoiding disaster.

The fact that nobody is artificially mass-producing their genes is not a disaster from Darwin's point of view; Darwin is vaguely satisfied that instead of a million humans there are now 7 billion humans. If the population stabilizes at 11 billion, that is also not a Darwinian disaster. If the population spreads across the galaxy, mostly in... (read more)

I think we're doing a little better than I predicted. Rationalists seem to be somewhat better able than their peers to sift through controversial public health advice, to switch careers (or retire early) when that makes sense, to donate strategically, and to set up physical environments that meet their needs (homes, offices, etc.) even when those environments are a bit unusual. Enough rationalists got into cryptocurrency early enough and heavy enough for that to feel more like successful foresight than a lucky bet. We're doing something at least partly rig... (read more)

I appreciate how much detail you've used to lay out why you think a lack of human agency is a problem -- compared to our earlier conversations, I now have a better sense of what concrete problem you're trying to solve and why that problem might be important. I can imagine that, e.g., it's quite difficult to tell how well you've fit a curve if the context in which you're supposed to fit that curve is vulnerable to being changed in ways whose goodness or badness is difficult to specify. I look forward to reading the later posts in this sequence so that I can... (read more)

Hmm. Nobody's ever asked me to try to teach them that before, but here's my advice:

1. Think about what dimensions or components success at the task will include. E.g., if you're trying to play a song on the guitar, you might decide that a well-played song will have the correct chords played with the correct fingering and the correct rhythm.
2. Think about what steps are involved in each of the components of success, with an eye toward ordering those steps in terms of which steps are easiest to learn and which steps are logical prerequisites for the others. E.g.,

I'm curious about the source of your intuition that we are obligated to make an optimal selection. You mention that the utility difference between two plausibly best meals could be large, which is true, especially when we drop the metaphor and reflect on the utility difference between two plausibly best FAI value schemes. And I suppose that, taken literally, the utilitarian code urges us to maximize utility, so leaving any utility on the table would technically violate utilitarianism.

On a practical level, though, I'm usually not in the habit of nitpicking ... (read more)

Thank you for sharing this; there are several useful conceptual tools in here. I like the way you've found crisply different adjectives to describe different kinds of freedom, and I like the way you're thinking about the computational costs of surplus choices. 

Building on that last point a bit, I might say that a savvy agent who has already evaluated N choices could try to keep a running estimate of their expected gains from choosing the best option available after considering X more choices and then compare that gain to their cost of computing the op... (read more)

I agree with this post. I'd add that from what I've seen of medical school (and other high-status vocational programs like law school, business school, etc.), there is still a disproportionate emphasis on talking about the theory of the subject matter vs. building skill at the ultimate task. Is it helpful to memorize the names of thousands of arteries and syndromes and drugs in order to be a doctor? Of course. Is that *more* helpful than doing mock patient interviews and mock chart reviews and live exercises where you try to diagnose a tumor or a... (read more)

I like the style of your analysis. I think your conclusion is wrong because of wonky details about World War 2. 4 years of technical progress at anything important, delivered for free on a silver platter, would have flipped the outcome of the war. 4 years of progress in fighter airplanes means you have total air superiority and can use enemy tanks for target practice. 4 years of progress in tanks means your tanks are effectively invulnerable against their opponents, and slice through enemy divisions with ease. 4 years of progress in manufacturing means you... (read more)

1) I agree with the very high-level point that there are lots of rationalist group houses with flat / egalitarian structures, and so it might make sense to try one that's more authoritarian to see how that works. Sincere kudos to you for forming a concrete experimental plan and discussing it in public.

2) I don't think I've met you or heard of you before, and my first impression of you from your blog post is that you are very hungry for power. Like, you sound like you would really, really enjoy being the chief of a tribe, bossing people around, having peopl... (read more)

And if you think you can explain the concept of "systematically underestimated inferential distances" briefly, in just a few words, I've got some sad news for you...

"I know [evolution] sounds crazy -- it didn't make sense to me at first either. I can explain how it works if you're curious, but it will take me a long time, because it's a complicated idea with lots of moving parts that you probably haven't seen before. Sometimes even simple questions like 'where did the first humans come from?' turn out to have complicated answers."

I am always trying to cultivate a little more sympathy for people who work hard and have good intentions! CFAR staff definitely fit in that basket. If your heart's calling is reducing AI risk, then work on that! Despite my disappointment, I would not urge anyone who's longing to work on reducing AI risk to put that dream aside and teach general-purpose rationality classes.

That said, I honestly believe that there is an anti-synergy between (a) cultivating rationality and (b) teaching AI researchers. I think each of those worthy goals is best pursued separately.

Yeah, that pretty much sums it up: do you think it's more important for rationalists to focus even more heavily on AI research so that their example will sway others to prioritize FAI, or do you think it's more important for rationalists to broaden their network so that rationalists have more examples to learn from?

Shockingly, as a lawyer who's working on homelessness and donating to universal income experiments, I prefer a more general focus. Just as shockingly, the mathematicians and engineers who have been focusing on AI for the last several years prefe... (read more)

Well, like I said, AI risk is a very important cause, and working on a specific problem can help focus the mind, so running a series of AI-researcher-specific rationality seminars would offer the benefit of (a) reducing AI risk, (b) improving morale, and (c) encouraging rationality researchers to test their theories using a real-world example. That's why I think it's a good idea for CFAR to run a series of AI-specific seminars.

What is the marginal benefit gained by moving further along the road to specialization, from "roughly half our efforts these d... (read more)

I dislike CFAR's new focus, and I will probably stop my modest annual donations as a result.

In my opinion, the most important benefit of cause-neutrality is that it safeguards the integrity of the young and still-evolving methods of rationality. If it is official CFAR policy that reducing AI risk is the most important cause, and CFAR staff do almost all of their work with people who are actively involved with AI risk, and then go and do almost all of their socializing with rationalists (most of whom also place a high value on reducing AI risk), then ther... (read more)

Does anyone know what happened to TC Chamberlin's proposal? In other words, shortly after 1897, did he in fact manage to spread better intellectual habits to other people? Why or why not?

Thank you! I see that some people voted you down without explaining why. If you don't like someone's blurb, please either contribute a better one or leave a comment to specifically explain how the blurb could be improved.

Sure!

Again, fair point -- if you are reading this, and you have experience designing websites, and you are willing to donate a couple of hours to build a very basic website, let us know!

Sounds good to me. I'll keep an eye out for public domain images of the Earth exploding. If the starry background takes up enough of the image, then the overall effect will probably still hit the right balance between alarm and calm.

A really fun graphic would be an asteroid bouncing off a shield and not hitting Earth, but that might be too specific.

Great! Pick one and get started, please. If you can't decide which one to do, please do asteroids.

It would go to the best available charity that is working to fight that particular existential risk. For example, the 'donate' button for hostile AI might go to MIRI. The donate button for pandemics might go the Center for Disease Control, and the donate button for nuclear holocaust might go to the Global Threat Reduction Initiative. If we can't agree on which agency is best for a particular risk, we can pick one at random from the front-runners.

If you have ideas for which charities are the best for a particular risk, please share them here! That is part of the work that needs to get done.

Hi Dorikka,

Yes, I am also concerned that the banner is too visually complicated -- it's supposed to be a scene of a flooded garage workshop, suggesting both major problems and a potential ability to fix them, but the graphic is not at all iconic. If you have another idea for the banner (or can recommend a particular font that would work better), please chime in.

I am not convinced that www.existential-risk.org is a good casual landing page, because (a) most of the content is in the form of an academic CV, (b) there is no easy-to-read summary telling the reader about existential risks, and (c) there is no donate button.

It's probably "Song of Light," or if you want a more literal translation, "Hymn to Light."

You might be wrestling with a hard trade-off between wanting to do as much good as possible and wanting to fit in well with a respected peer group. Those are both good things to want, and it's not obvious to me that you can maximize both of them at the same time.

I have some thoughts on your concepts of "special snowflake" and "advice that doesn't generalize." I agree that you are not a special snowflake in the sense of being noticeably smarter, more virtuous, more disciplined, whatever than the other nurses on your shift. I'll concede t... (read more)

OK, but why is "chair" shorter than "furniture"? Why is "blue" shorter than "color"? Furniture and color don't strike me as words that are so abstract as to rarely see use in everyday conversation.

I'm confused. What makes "chair" the basic category? I mean, obviously more basic categories will have shorter words -- but who decided that "solid object taking up roughly a cubic meter designed to support the weight of a single sitting human" was a basic category?

That's an important warning, and I'm glad you linked me to the post on ethical inhibitions. It's easy to be mistaken about when you're causing harm, and so allowing a buffer in honor of the precautionary principle makes sense. That's part of why I never mention the names of any of my clients in public and never post any information about any specific client on any public forums -- I expect that most of the time, doing so would cause no harm, but it's important to be careful.

Still, I had the sense when I first read your comment six weeks ago that it's not a... (read more)

It seems to me that educated people should know something about the 13-billion-year prehistory of our species and the basic laws governing the physical and living world, including our bodies and brains. They should grasp the timeline of human history from the dawn of agriculture to the present. They should be exposed to the diversity of human cultures, and the major systems of belief and value with which they have made sense of their lives. They should know about the formative events in human history, including the blunders we can hope not to repeat. They

You're...welcome? For what it's worth, mainstream American legal ethics try to strike a balance between candor and advocacy. It's actually not OK for lawyers to provide unabashed advocacy; lawyers are expected to also pay some regard to epistemic accuracy. We're not just hired mercenaries; we're also officers of the court.

In a world that was full of Bayesian Conspiracies, where people routinely teased out obscure scraps of information in the service of high-stakes, well-concealed plots, I would share your horror at what you describe as "disclosing pe... (read more)

Is that revelation grounds for a lawsuit, a criminal offense or merely grounds for disbarment?

None of the above, really, unless you have so few murder cases that someone could plausibly guess which one you were referring to. I work with about 100 different plaintiffs right now, and my firm usually accepts any client with a halfway decent case who isn't an obvious liar. Under those conditions, it'd be alarming if I told you that 100 out of 100 were telling the truth -- someone's bound to be at least partly faking their injury. I don't think it undermines... (read more)

I'm confused about how this works.

Suppose the standard were to use 80% confidence. Would it still be surprising to see 60 of 60 studies agree that A and B were not linked? Suppose the standard were to use 99% confidence. Would it still be surprising to see 60 of 60 studies agree that A and B were not linked?

Also, doesn't the prior plausibility of the connection being tested matter for attempts to detect experimenter bias this way? E.g., for any given convention about confidence intervals, shouldn't we be quicker to infer experimenter bias when a set of stu... (read more)

Yes, Voldemort could probably teach DaDA without suffering from the curse, and a full-strength Voldemort with a Hogwarts Professorship could probably steal the stone.

I'm not sure either of those explains how Voldemort got back to full-strength in the first place, though. Did Voldemort fake the charred hulk of his body? And Harry forgot that apparent charred bodies aren't perfectly reliable evidence of a dead enemy because his books have maxims like "don't believe your enemy is dead until you see the body?" But then what was Voldemort doing between 1975 and 1990? He was winning the war until he tackled Harry; why would he suddenly decide to stop?

Puzzle:

Who is ultimately in control of the person who calls himself Quirrell?

• Voldemort

If Voldemort is possessing the-person-pretending-to-be-Quirrell using the path Dumbledore & co. are familiar with, or for that matter by drinking unicorn blood, then why isn't Voldy's magic noticeably weaker than before? Quirrell seems like he could at least hold his own against Dumbledore, and possibly defeat him.

If Voldemort took control of the-person-pretending-to-be-Quirrell's body outright using incredibly Dark magic, then why would Quirrell openly suggest th... (read more)

Is there more to the Soylent thing than mixing off-the-shelf protein shake powder, olive oil, multivitamin pills, and mineral supplement pills and then eating it?

Isn't there a very wide middle ground between (1) assigning 100% of your mental probability to a single model, like a normal curve and (2) assigning your mental probability proportionately across every conceivable model ala Solomonoff?

I mean the whole approach here sounds more philosophical than practical. If you have any kind of constraint on your computing power, and you are trying to identify a model that most fully and simply explains a set of observed data, then it seems like the obvious way to use your computing power is to put about a quarter of yo... (read more)

What's the percent chance that I'm doing it wrong?

I once heard a story about the original writer of the Superman Radio Series. He wanted a pay rise, his employers didn't want to give him one. He decided to end the series with Superman trapped at the bottom of a well, tied down with kryptonite and surrounded by a hundred thousand tanks (or something along these lines). It was a cliffhanger. He then made his salary demands. His employers refused and went round every writer in America, but nobody could work out how the original writer was planning to have Superman escape. Eventually the radio guys had to go

Ironically, this is my most-upvoted comment in several months.

OK, so how else might we get people to gate-check the troublesome, philosophical, misleading parts of their moral intuitions that would have fewer undesirable side effects? I tend to agree with you that it's good when people pause to reflect on consequences -- but then when they evaluate those consequences I want them to just consult their gut feeling, as it were. Sooner or later the train of conscious reasoning had better dead-end in an intuitively held preference, or it's spectacularly unlikely to fulfill anyone's intuitively held preferences. (I, of cou... (read more)

Given at least moderate quality, upvotes correlate much more tightly with accessibility / scope of audience than quality of writing. Remember, the article score isn't an average of hundreds of scalar ratings -- it's the sum of thousands of ratings of [-1, 0, +1] -- and the default rating of anyone who doesn't see, doesn't care about, or doesn't understand the thrust of a post is 0. If you get a high score, that says more about how many people bothered to process your post than about how many people thought it was the best post ever.

OK, let's say you're right, and people say "awesome" without thinking at all. I imagine Nyan_Sandwich would view that as a feature of the word, rather than as a bug. The point of using "awesome" in moral discourse is precisely to bypass conscious thought (which a quick review of formal philosophy suggests is highly misleading) and access common-sense intuitions.

I think it's fair to be concerned that people are mistaken about what is awesome, in the sense that (a) they can't accurately predict ex ante what states of the world they will w... (read more)

To say that something's 'consequentialist' doesn't have to mean that it's literally forward-looking about each item under consideration. Like any other ethical theory, consequentialism can look back at an event and determine whether it was good/awesome. If you going white-water rafting was a good/awesome consequence, then your decision to go white-water rafting and the conditions of the universe that let you do so were good/awesome.

Also, this book was a horrible agglomeration of irrelevant and un-analyzed factoids. If you've already read any two Malcolm Gladwell books or Freakonomics, It'd be considerably more educational to skip this book and just read the cards in a Trivial Pursuit box.

The undergrad majors at Yale University typically follow lukeprog's suggestion -- there will be 20 classes on stuff that is thought to constitute cutting-edge, useful "political science" or "history" or "biology," and then 1 or 2 classes per major on "history of political science" or "history of history" or "history of biology." I think that's a good system. It's very important not to confuse a catalog of previous mistakes with a recipe for future progress, but for the same reasons that general hi... (read more)