= 27d567bc2d48d9f40e9ccc20ea370b15)
ata comments on The Importance of Self-Doubt - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (726)
Yeah, "people who think Singularitarians think that" is what I meant.
I've actually met exactly one something-like-a-Singularitarian who did think something-like-that — it was at one of the Bay Area meetups, so you may or may not have talked to him, but anyway, he was saying that only people who invent or otherwise contribute to the development of Singularity technology would "deserve" to actually benefit from a positive Singularity. He wasn't exactly saying he believed that the nonbelievers would be left to languish when cometh the Singularity, but he seemed to be saying that they should.
Also, I think he tried to convert me to Objectivism.
Technological progress has increased weath inequality a great deal so far.
Machine intelligence probably has the potential to result in enormous weath inequality.
How, in a post-AGI world, would you define wealth? Computational resources? Matter?
I don't think there's any foundation for speculation on this topic at this time.
Unless we get a hard-takeoff singleton, which is admittedly the SIAI expectation, there will be massive inequality, with a few very wealthy beings and average income barely above subsistence. Thus saith Robin Hanson, and I've never seen any significant holes poked in that thesis.
Robin Hanson seems to be assuming that human preferences will, in general, remain in their current ranges. This strikes me as unlikely in the face of technological self-modification.
I've never gotten that impression. What I've gotten is that evolutionary pressures will, in the long term, still exist--even if technological self-modification leads to a population that's 99.99% satisfied to live within strict resource consumption limits, unless they harshly punish defectors the .01% with a drive for replication or expansion will overwhelm the rest within a few millenia, until the average income is back to subsistence. This doesn't depend on human preferences, just the laws of physics and natural selection.
What evolutionary pressures? Even making the incredible assumption that we will continue to use sequences of genes as a large part of our identities, what's to stop a singleton of some variety from eliminating drives for replication or expansion entirely?
I feel uncomfortable speculating about a post-machine-intelligence future even to this extent; this is not a realm in which I am confident about any proposition. Consequently, I view all confident conclusions with great skepticism.
You're still not getting the breadth and generality of Hanson's model. To use recent LW terminology, it's an anti-prediction.
It doesn't matter whether agents perpetuate their strategies by DNA mixing, binary fission, cellular automata, or cave paintings. Even if all but a tiny minority of posthumans self-modify not to want growth or replication, the few that don't will soon dominate the light-cone. A singleton, like I'd mentioned, is one way to avert this. Universal extinction and harsh, immediate punishment of expansion-oriented agents are the only others I see.
You (or Robin, I suppose) are just describing a many-agent prisoner's dilemma. If TDT agents beat the dilemma by cooperating with other TDT agents, then any agents that started out with a different decision theory will have long since self-modified to use TDT.
Alternately, if there is no best decision theoretical solution to the prisoner's dilemma, then we probably don't need to worry about surviving to face this problem.
Now, there's a generalized answer. It even covers the possibility of meeting aliens--finding TDT is a necessary condition for reaching the stars. Harsh punishment of inconsiderate expanders might still be required, but there could be a stable equilibrium without ever actually inflicting that punishment. That's a new perspective for me, thanks!
Your point is spot on. Competition can not be relied on to produce adaptation if someone wins the competition once and for all.
Control, owned by preferences.
I wasn't trying to make an especially long-term prediction:
"We saw the first millionaire in 1716, the first billionaire in 1916 - and can expect the first trillionaire within the next decade - probably before 2016."
Inflation.
The richest person on earth currently has a net worth of $53.5 billion.
The greatest peak net worth in recorded history, adjusted for inflation, was Bill Gates' $101 billion, which was ten years ago. No one since then has come close. A 10-fold increase in <6 years strikes me as unlikely.
In any case, your extrapolated curve points to 2116, not 2016.
I am increasingly convinced that your comments on this topic are made in less than good faith.
Yes, the last figure looks wrong to me too - hopefully I will revisit the issue.
Update 2011-05-30: yes: 2016 was a simple math mistake! I have updated the text I was quoting from to read "later this century".
Anyway, the huge modern wealth inequalities are well established - and projecting them into the future doesn't seem especially controversial. Today's winners in IT are hugely rich - and tomorrow's winners may well be even richer. People thinking something like they will "be on the inside track when the singularity happens" would not be very surprising.
Projecting anything into a future with non-human intelligences is controversial. You have made an incredibly large assumption without realizing it. Please update.
If you actually want your questions answered, then money is society's representation of utility - and I think there will probably be something like that in the future - no matter how far out you go. What you may not find further out is "people". However, I wasn't talking about any of that, really. I just meant while there are still money and people with bank accounts around.
A few levels up, you said,
My dispute is with the notion that people with bank accounts and machine intelligence will coexist for a non-trivial amount of time.
We have been building intelligent machines for many decades now. If you are talking about something that doesn't yet exist, I think you would be well advised to find another term for it.