A note on how this post was produced:

Eliezer brain-dumped his thoughts on this open problem to Facebook, and replied to questions there for several hours. Then Robby spent time figuring out how to structure a series of posts that would more clearly explain the open problem, and wrote drafts of those posts. Several people, including Eliezer, commented heavily on various drafts until they reached a publishable form. Louie coordinates the project.

After discussion of the posts on Less Wrong, we may in some cases get someone to write up journal article expositions of some of the ideas in the posts.

The aim is to write up open problems in Friendly AI using as little Eliezer-time as possible. It seems to be working so far.

Brain dump of a quick idea:

A sufficiently complex bridge law might say that the agent is actually a rock which, through some bizarre arbitrary encoding, encodes a computation[1]. Meanwhile the actual agent is somewhere else. Hopefully the agent has some adequate Occamian prior and he never assigns this hypothesis any relevance because of the high complexity of the encoding code.

In idea-space, though, there is a computation which is encoded by a rock using a complex arbitrary encoding, which, by virtue of having a weird prior, concludes that it actually is that rock, and to whom them breaking of the rock would mean death. We usually regard this computation as irrelevant for moral purposes - only computations corresponding to "natural" interpretations of physical phenomena count. But this distinction between natural and arbitrary criterions of interpretation seems, well, arbitrary.

We regard a person's body as "executing" the computation that is a person that thinks she is in that body. But we do not regard the rock as actually "executing" the computation that is a weird agent that thinks it's the computation encoded by the rock (through some arbitrary encoding).

Why?

The pragmatic reason is obvious: you can't do anything to help the rock-computation in anyway, and whatever you do, you'd be lowering utility for some other ideal computation.

But maybe the kind of reasoning the rock-computation has to make to keep seeing itself as the rock is relevant.

A rock-as-computation hypothesis (I am this rock in this universe = my phenomenological experiences correspond to the states os atoms in several points of this rock as translated by this [insert very long bridge law, full of specifics] bridge law) is doomed to fail at the few next steps in the computation. Because the bridge law is so ad hoc, it won't correctly predict the next phenomena perceived by the computation (in the ideal or real world where it actually executes). So if the rock-computation does induction at all, it will have to change the bridge law, and give up on being that rock.

In other words, if we built a robot with a prior that privileges the bridge law hypothesis that it's a computation encoded in a rcok though some bizarre encoding, it would have to abandon that hypothesis very very soon. And, as phenomelogical data came in, it would approach the correct hypothesis that it's the robot. Unless it's doing some kind of privileged-hypothesis anti-induction where it keeps adopting increasingly complex bridge laws to keep believing it is that one rock.

So, proposal: a substrate is regarded to embody a computation-agent if that computation-agent is one that, by doing induction in the same sense we do to find out about ourselves, will eventually arrive at the correct bridge law that it is being executed in said substrate.

--

[1] Rock example is from DRESCHER, Gary, Good and Real, 2006, p. 55.

The ULH suggests that most everything that defines the human mind is cognitive software rather than hardware: the adult mind (in terms of algorithmic information) is 99.999% a cultural/memetic construct.

I think a distinction worth tracing here is the diferrence between "learning" in the neural-net-sense and "learning" in the human pedagogical/psychological sense.

The "learning" done by a piece of cortex becoming a visual cortex after receiving neural impulses from the eye isn't something you can override by teaching a person (in the usual sense o the word "teaching") - you'd need to rewire their brain. I don't think you can call it cultural/memetic because this neural learning does not (seem to) occur through the mechanism(s) that deals with concepts, ideas and feelings, which is involved in learning a language or a social custom or a scientific theory.

In the same way, maybe the availability heuristic isn't genetically coded, but is learned through the type of data certain parts of the brain have to work with. That would mean you could fix it through some input rewiring during gestation, but doesn't mean you can change it through a new human education system - it may be too low level, like a generic part of the cortex becoming the visual cortex. If that's the case, I wouldn't say it's a cultural/memetic construct (although it is an environmental construct).

I meant not hardcoding values or ethics.

The lesson is: do not hard-code important functionality into your AGI without proving it correct. 

That's two lessons. Not hardcoding iat all is under explored round here.

At last. Wouldn't miss it.

I actually watched his TED talk last night. Will look more into his stuff.

The main issues I'm facing are understanding the math behind combining estimates and actually making the program right now. However, he definitely seems to be one of the top world experts on actually making these kinds of models.

How is that translation coming along? I could help with German.

Hi, and thanks for the awesome job! Will you keep a public record of changes you make to the book? I'm coordinating a translation effort, and that would be important to keep it in sync if you change the actual text, not just fix spelling and hyperlinking errors.

Edit: Our translation effort is for Portuguese only, and can be found at http://racionalidade.com.br/wiki .

I asked judge Richard Posner (one of my dissertation advisers) if he would be willing to use lotteries as a judge and he said no, it would get him impeached.