LESSWRONG
LW

All of lukedrago's Comments + Replies

The Intelligence Curse
lukedrago70

That's a choice, though. AGI could, for example, look like a powerful actor in its own right, with its own completely nonhuman drives and priorities, and a total disinterest in being directed in the sort of way you'd normally associate with a "resource".

My claim is that the incentives AGI creates are quite similar to the resource curse, not that it would literally behave like a resource. But:

If by "intent alignment" you mean AGIs or ASIs taking orders from humans, and presumably specifically the humans who "own" them, or are in charge of the "powerful acto

... (read more)
Reply
The Intelligence Curse
lukedrago10

Glad you enjoyed it! 

Could you elaborate on your last paragraph? Presuming a state overrides its economic incentives (ie establishes a robust post-AGI welfare system), I'd like to see how you think the selection pressures would take hold.

For what it's worth, I don't think "utopian communism" and/or a world without human agency are good. I concur with Rudolf entirely here -- those outcomes miss agency what has so far been an core part of the human experience. I want dynamism to exist, though I'm still working on if/how I think we could achieve that. I'll save that for a future post.

Reply
1tangerine
I don't know how selection pressures would take hold exactly, but it seems to me that in order to prevent selection pressures, there would have to be complete and indefinite control over the environment. This is not possible because the universe is largely computationally irreducible and chaotic. Eventually, something surprising will occur which an existing system will not survive. Diverse ecosystems are robust to this to some extent, but that requires competition, which in turn creates selection pressures.
The Intelligence Curse
lukedrago51

I appreciate this concern, but I disagree. An incognito google search of "intelligence curse" didn't yield anything using this phrase on the front page except for this LessWrong post. Adding quotes around it or searching for the full phrase ("the intelligence curse") showed this post as the first result. 

A quick twitter search in recent shows the phrase "the intelligence curse" before this post:

  • In 24 tweets in total
  • With the most recent tweet on Dec 21, 2024
  • Before that, in a tweet from August 30, 2023
  • In 10 tweets since 2020
  • And all other mentions pre-20
... (read more)
Reply
3Seth Herd
I agree that the intelligence curse isn't a common phrase. But I think the intelligence resource curse is more memorable because it encapsulates the whole idea.
The Intelligence Curse
lukedrago72

I agree. To add an example: the US government's 2021 expanded child tax credit lifted 3.7 million children out of poverty, a near 50% reduction. Moreover, according to the NBER's initial assessment: "First, payments strongly reduced food insufficiency: the initial payments led to a 7.5 percentage point (25 percent) decline in food insufficiency among low-income households with children. Second, the effects on food insufficiency are concentrated among families with 2019 pre-tax incomes below $35,000". 

Despite this, Congress failed to renew the program.... (read more)

Reply