ideal reasoners are not supposed to disagree

My ideal thinkers do disagree, even with themselves. Especially about areas as radically uncertain as this.

Nick, do you see a fault in how I've been carrying on our discussions as well? Because you've also left several of our threads dangling, including:

• How likely is it that an AGI will be created before all of its potential economic niches have been filled by more specialized algorithms?
• How much hope is there for "security against malware as strong as we can achieve for symmetric key cryptography"?
• Does "hopelessly anthropomorphic and vague" really apply to "goals"?

(Of course it's understandable if you're just too busy. If that's the case, what kind of projects are you working on these days?)

Law is very highly evolved, the work of millions of people as smart or smarter than Yudkoswky over more than a millenium,

That seems pretty harsh! The Bureau of Labor Statistics reports 728,000 lawyers in the U.S., a notably attorney-heavy society within the developed world. The SMPY study of kids with 1 in 10,000 cognitive test scores found (see page 722) only a small minority studying law. The 90th percentile IQ for "legal occupations" in this chart is a little over 130. Historically populations were much lower, nutrition was worse, legal education or authority was only available to a small minority, and the Flynn Effect had not occurred. Not to mention that law is disproportionately made by politicians who are selected for charisma and other factors in addition to intelligence.

and tested empirically against the real world of real agents with a real diversity of values every day. It's not something you can ever come close to competing with by a philosophy invented from scratch.

It's hard to know what to make of this.

Perhaps that the legal system is good at creating incentives that closely align the interests of those it governs with the social good, and that this will work on new types of being without much dependence on their decisionmaking processes?

Contracts and basic property rights certainly do seem to help produce wealth. On the other hand, financial regulation is regularly adjusted to try to nullify new innovation by financiers that poses systemic risks or exploits government guarantees, but the financial industry still frequently outmaneuvers the legal system. And of course the legal system depends on the loyalty of the security forces for enforcement, and makes use of ideological agreement among the citizenry that various things are right or wrong.

Restraining those who are much weaker is easier than restraining those who are strong. A more powerful analogy would be civilian control over military and security forces. There do seem to have been big advances in civilian control over the military in the developed countries (fewer coups, etc), but they seem to reflect changes in ideology and technology more than law.

If it is easy to enforce laws on new AGI systems, then the situation seems fairly tractable, even for AGI systems with across-the-board superhuman performance which take action based on alien and inhumane cost functions. But it doesn't seem guaranteed that it will be easy to enforce such laws on smart AGIs, or that the trajectory of development will be "all narrow AI, all the time," given the great economic value of human generality.

I only have time for a short reply:

(1) I'd rephrase the above to say that computer security is among the two most important things one can study with regard to this alleged threat.

(2) The other important thing is law. Law is the "offensive approach to the problem of security" in the sense I suspect you mean it (unless you mean something more like the military). Law is very highly evolved, the work of millions of people as smart or smarter than Yudkoswky over more than a millenium, and tested empirically against the real world of real agents with a real diversity of values every day. It's not something you can ever come close to competing with by a philosophy invented from scratch.

(3) I stand by my comment that "AGI" and "friendliness" are hopelessly anthropomorphic, infeasible, and/or vague.

(4) Computer "goals" are only usefully studied against actual algorithms, or clearly defined mathemetical classes of algorithms, not vague and imaginary concepts. Perhaps you can make some progress by for example advancing the study of postconditions, which seem to be the closest analog to goals in the software engineering world. One can imagine a world where postconditions are always checked, for example, and other software ignores the output of software that has violated one of its postconditions.

His early works, such as The Selfish Gene, were actually really good books for convincing somebody of an alternative to creationism or guided creation, however. (Which isn't the same as convincing somebody of atheism, but does give somebody paralyzed by the question of where complex life came from a much-needed line of retreat.)

When some day some people (or some things) build an AGI [...] Humans will already have been obsoleted for all jobs except, probably, those that for emotional reasons require interaction with another human

To rephrase my question, how confident are you of this, and why? It seems to me quite possible that by the time someone builds an AGI, there are still plenty of human jobs that have not been taken over by specialized algorithms due to humans not being smart enough to have invented the necessary specialized algorithms yet. Do you have a reason to think this can't be true?

ETA: My reply is a bit redundant given Nesov's sibling comment. I didn't see his when I posted mine.

When some day some people (or some things) build an AGI, human-like or otherwise, it will at that time be extremely inferior to then-existing algorithms for any particular task (including any kind of learning or choice, including learning or choice of algorithms). Culture, including both technology and morality, will have changed beyond any of our recognitions long before that. Humans will already have been obsoleted for all jobs except, probably, those that for emotional reasons require interaction with another human (there's already a growth trend in such jobs today).

The phrasing suggests a level of certainty that's uncalled for for a claim that's so detailed and given without supporting evidence. I'm not sure there is enough support for even paying attention to this hypothesis. Where does it come from?

(Obvious counterexample that doesn't seem unlikely: AGI is invented early, so all the cultural changes you've listed aren't present at that time.)

If uploads are infeasible, what about other possible ways to build AGIs? In any case, I'm responding to Nick's argument that we do not have have to worry about extreme consequences from AGIs because "specialized algorithms are generally far superior to general ones", which seems to be a separate argument from whether AGIs are feasible.

It can be summarized as follows: for basic reasons of economics and computer science, specialized algorithms are generally far superior to general ones.

I don't understand your reasoning here. If you have a general AI, it can always choose to apply or invent a specialized algorithm when the situation calls for that, but if all you have is a collection of specialized algorithms, then you have to try to choose/invent the right algorithm yourself, and will likely do a worse (possibly much worse) job than the general AI if it is smarter than you are. So why do we not have to worry about "extreme consequences from general AI"?

Seems like there is more going on than just "Do transhumanists endorse Pascalian bargains?" Because of confidence levels inside and outside an argument, the fact that someone (e.g. SI) makes an argument that a particular risk has a non-negligible probability does not mean that someone examining this claim should assign a non-negligible probability. It's very possible for someone thinking about (e.g.) AI risk to assign low probabilities and thus find themselves in a Pascalian situation even if SI argues that the probability of AI risk is high.