I wonder how current cryonics organizations will adapt if/as plastination becomes viable, especially if plastination does well on the critical scanning demonstrations where vitrification is largely unproven.

Can cryonics orgs like Alcor adapt quickly? What about those currently paying insurance premiums based on higher vitrification prices?

Exciting times regardless!

This post is cute, but there are several flaws/omissions that can lead to compound propagating errors in typical interpretations.

Any two AI designs might be less similar to each other than you are to a petunia.

Cute. The general form of this statement:

(Any two X might be less similar to each other than you are to a petunia) is trivially true if our basis of comparison is based solely on genetic similarity.

This leads to the first big problem with this post: The idea that minds are determined by DNA. This idea only makes sense if one is thinking of a mind as a sort of potential space.

Clone Einstein and raise him with wolves and you get a sort of smart wolf mind inhabiting a human body. Minds are memetic. Petunias don't have minds. I am my mind.

The second issue (more of a missing idea really) is that of functional/algorithmic equivalence. If you take a human brain, scan it, and sufficiently simulate out the key circuits, you get a functional equivalent of the original mind encoded in that brain. The substrate doesn't matter, and nor even do the exact algorithms, as any circuit can be replaced with any algorithm that preserves the input/output relationships.

Functional equivalence is another way of arriving at the "minds are memetic" conclusion.

As a result of this, the region of mindspace which we can likely first access with AGI designs is some small envelop around current human mindspace.

The map of mindspace here may be more or less correct, but whats-anything-but-clear is how distinct near term de novo AGI actually is from say human uploads, given: functional equivalence, bayesian brain, no free lunch in optimization, and the mind is memetic.

For example, if the most viable route to AGI turns out to be brain-like designs, then it is silly not to anthropomorphize AGI.

What does that have to do with whether an AI will need living human beings?

The AI will need to simulate its history as a natural necessary component of its 'thinking'. For a powerful enough AI, this will entail simulation down to the level of say the Matrix, where individual computers and human minds are simulated at their natural computational scale level.

It seems like there is an unstated premise that living humans are equivalent to simulated humans. That's a defensible position, but implicitly asserting the position is not equivalent to defending it.

Yes. I'm assuming most people here are sufficiently familiar with this position such that it doesn't require my defense in a comment like this.

Yes. I'm surprised this isn't brought up more. AIXI formalizes the idea that intelligence involves predicting the future through deep simulation, but human brains use something like a Monte Carlo sim like approach as well.

Given enough computation the best way to generate accurate generative probabilistic models is to run lots of detailed Monte Carlo simulations. AIXI like models do this, human brains do it to a limited extent.

This is a dialogue between two prominent AI researchers.

What? This was a dialog between Pei and lukeprog, right?

Of course, I could be completely misinterpreting things. I thought I would share my thought process after I came to the same conclusion as you did.

I'm curious about what you mean by the appellation "prominent AI researcher" that you would apply it to lukeprog, and whether he considers himself as a member of that category.

The amount of Less Wrong jargon, links to Less Wrong posts explaining that jargon, and the Yudkowsky "proclamation" in this paragraph is all a bit squicky, alienating and potentially condescending.

Yes. Well said. The deeper issue though is the underlying causes of said squicky, alienating paragraphs. Surface recognition of potentially condescending paragraphs is probably insufficient.

Anyway, biting Pei's bullet for a moment, if building an AI isn't safe, if it's, like Pei thinks, similar to educating a child (except, presumably, with a few orders of magnitude more uncertainty about the outcome) that sounds like a really bad thing to be trying to do.

Its unclear that Pei would agree with your presumption that educating an AGI will entail "a few orders of magnitude more uncertainty about the outcome". We can control every aspect of an AGI's development and education to a degree unimaginable in raising human children. Examples: We can directly monitor their thoughts. We can branch successful designs. And perhaps most importantly, we can raise them in a highly controlled virtual environment. All of this suggests we can vastly decrease the variance in outcome compared to our current haphazard approach of creating human minds.

But we're terrible at educating children.

Compared to what? Compared to an ideal education? Your point thus illustrates the room for improvement in educating AGI.

Children routinely grow up to be awful people.

Routinely? Nevertheless, this only shows the scope and potential for improvement. To simplify: if we can make AGI more intelligent, we can also make it less awful.

And this one lacks the predictable, well-defined drives and physical limits that let us predict how most humans will eventually act

An unfounded assumption. To the extent that humans have these "predictable, well-defined drives and physical limits" we can also endow AGI's with these qualities.

Pei's argument is a grand rebuttal of the proposal that humanity spend more time on AI safety (why fund something that isn't possible?) but no argument at all against the second part of the proposal-- defund AI capabilities research.

Which doesn't really require much of an argument against. Who is going to defund AI capabilities research such that this would actually prevent global progress?

This seems based around limited views of what sort of AI minds are possible or likely, such as an anthropomorphized baby which can be taught and studied similar to human children.

The key difference between AI and other software is learning, and even current narrow AI systems require large learning/training times and these systems are only learning specific narrow functionalities.

Considering this, many (perhaps most?) AGI researchers believe that any practical human-level AGI will require an educational process much like human children do.

That a de novo AGI will be nothing like a human child in terms of how to make it safe is an antiprediction in that it would take a tremendous amount of evidence to suggest otherwise, and yet Wang just assumes this without having any evidence at all.

I think this single statement summarizes the huge rift between the narrow specific LW/EY view of AGI and other more mainstream views.

For researchers who are trying to emulate or simulate brain algorithms directly, its self-evidently obvious that the resulting AGI will start like a human child. If they succeed first your 'antiprediction' is trivially false. And then we have researchers like Wang or Goertzel who are pursuing AGI approaches that are not brain-like at all and yet still believe the AGI will learn like a human child and specifically use that analogy.

You can label anything an "antiprediction" and thus convince yourself that you need arbitrary positive evidence to disprove your counterfactual, but in doing so you are really just rationalizing your priors/existing beliefs.

For so many people, "building a safe AI" just pattern-matches to "raising a child so he becomes a good citizen", even though these tasks have nothing to do with each other.

Suppose you build an AI that exactly replicates the developmental algorithms of an infant brain, and you embody it in a perfect virtual body. For this particular type of AI design, the analogy is perfectly exact, and the AI is in fact a child exactly equivalent to a human child.

A specific human brain is a single point in mindspace, but the set of similar architectures extends out into a wider region which probably overlaps highly with much of the useful, viable, accessible space of AGI designs. So the analogy has fairly wide reach.

As an analogy, its hard to see how comparing a young AI to a child is intrinsically worse than comparing the invention of AI to the invention of flight, for example.