This was also my reaction, better stated than I would have done.

I think there's a version of this argument that says that most people would not reflectively endorse the animal suffering they cause, if they truly understood themselves and their own values in a CEV-like sense. I don't know if that version is true either.

If there was a button that would kill me with a 60% probability and transport me into a utopia for billions of years with a 15% probability, I would feel very scared to press that button

This is because the correct answer is option three: try to modify the button to lower the 60 and raise the 15, until such time as a 1-in-5 chance of survival is a net improvement relative to your default situation. I'd be much more likely to press that button if I'd just jumped out of an airplane without a parachute. Or if there was a hundred mile wide asteroid near-guaranteed to hit Earth next Tuesday.

Also, this is the first year where the people close to me are cognizant enough of AI that I can talk to them about life plan derailment expectations and not be dismissed as crazy. I can tell my parents to try to really attend to their health more than they have in the past, and why. I can explain to my wife that hey, we should both expect start surfing a wave of frequent job changes until the concept of a job stops making sense. It's been honestly very freeing to be able to discuss these things somewhere other than this community. I'm still a little hesitant to openly talk to my sisters about what their children's futures might look like, but even that is starting to change.

Excellent post, thank you! Two thoughts:

1. When I talk to less knowledgeable people about AI, they tend not to distinguish capabilities/incompetence from poor implementation choices. For example, at the grocery store yesterday the cashier and bagger were complaining about the new AI camera monitoring system tracking everything they do. It's easy to blame the AI because it's new, but at some level they're really mad about a boss demonstrating that he doesn't trust his employees to do their jobs but does trust the outputs of a still-error-prone automated system without sufficient human review.
2. "They can't hold a whole job, let alone self-improve, without help" - this kind of talk (that I hear regularly)  feels like we're at the "craftsmen balk when asked to work the first assembly line" stage. After the development of cottage industry, factories, and the assembly line, and almost 70 years from "I, Pencil," we still haven't collectively internalized the idea that "whole job" is not a coherent concept, and tasks can be refactored, and too bad for the humans trying to keep up and learn the right skills and mindset.

Much appreciated, and a concept (on the physical not digital side) I've had trouble explaining to people in the past. I've mentioned elsewhere on LW that I sold my house and lived in an RV for the past 4 years. It was very freeing and good for my mental health to downsize and learn to live minimally. We always had the intention to eventually build a house when we decided to stop traveling, and apparently a lot of people were very surprised that we were planning on building a non-tiny house. We had to explain that the point was never to have less space for its own sake. It's to make sure the space and things we have serve a meaningful purpose in our lives

I think it's important, on the GDP questions, to distinguish nominal vs real (aka output) growth. Nominal growth has been very stable, and in fact via policy we can keep it very stable under most circumstances. Real growth is harder to measure accurately, and the way we do so has some issues with handling the value generated by products that increase in quality for a given cost or decrease in cost for a given quality over time. If AI generates a lot of value, let's say dectupling total output, and then the AI falls in cost by a factor of 1000, what's the effect on (nominal) GDP? Is that the right question for understanding the impact on people's lives?

Economically, I agree with your calculations on power. However, the US has worked itself into a corner where we frequently tie our own hands, refusing to let the people who want to build the energy production facilities do so where they're needed on anything like a reasonable timescale.

Ok, fair example. I still maintain that "the nation's entire drinking water supply" is not actually a coherent, relevant concept. There are good reasons to build data centers in Chile - cheap wind and solar potential, for example. Could they really not have forced Google to commit to building a desal plant and associated power generation to offset their own water demand? That seems like a pretty clear negotiation failure but not necessarily Google's responsibility. Or if the government honestly believes the water cost is worth it, are they wrong? Or was there actual corruption involved?

Sorry, not trying to derail a post that I actually liked and think is important. It just read to me like all the other misleading claims about data center water usage.

extract resources from the Global South (such as potable water

 

Well, that turned me off from wanting to read it. What's the argument there? Water is almost always a local resource, especially at scale, not like fuels and food and minerals. Unless they're trucking water in from thousands of miles away or building data centers in those regions without the infrastructure needed to supply themselves? In which case the local opposition to data center construction on water demand grounds makes even less sense. But I'd be baffled if they were doing that. 

The median voter theorem applies to particular methods of deciding outcomes. The decision making processes in the EU, its institutions, and member states are sufficiently complex and diverse that I'd be very surprised if anything like it applied.

I agree with a lot of what you're saying, and it made me realize I left out some of my reasoning that's maybe more central than I realized.

Namely, what is the rate-limiting step in getting improved outcomes for people, health-wise? I would say the limiter is regulatory, in ways I don't see current or near term AI significantly altering. In other words, under OpenAI's own claimed timelines, I wouldn't expect AI-assisted health innovation to generate real world results before close-enough-to-AGI-to-be-really-dangerous gets developed. Of course we should be using AI to advance medicine faster as soon as we can do so. But I don't see why we need a non-profit to fund that, when it will also be very profitable to the companies that will use it. Conversely, an additional $25B invested in making future AI safer doesn't have a whole lot of other funders lining up to make it happen.