A. Solve the Problem of Meaning-in-General in advance, and program it to follow our instructions' real meaning. Then just instruct it 'Satisfy my preferences', and wait for it to become smart enough to figure out my preferences.

That problem has got to be solved somehow at some stage, because something that couldn't pass a Turing Test is no AGI.

But there are a host of problems with treating the mere revelation that A is an option as a solution to the Friendliness problem. 1. You have to actually code the seed AI to understand what we mean. Y

Why is that a problem? Is anyone suggesting AGI can be had for free?

1. The Problem of Meaning-in-General may really be ten thousand heterogeneous problems, especially if 'semantic value' isn't a natural kind. There may not be a single simple algorithm that inputs any old brain-state and outputs what, if anything, it 'means'; it may instead be that different types of content are encoded very differently.

Ok. NL is hard. Everyone knows that. But its got to be solved anyway.

3... On the face of it, programming an AI to fully understand 'Be Friendly!' seems at least as difficult as just programming Friendliness into it, but with an added layer of indirection.

Yeah, but it's got to be done anyway.

[more of the same snipped]

It's clear that building stable preferences out of B or C would create a Friendly AI.

Yeah. But it wouldn't be an AGI or an SI if it couldn't pass a TT.

The genie — if it bothers to even consider the question — should be able to understand what you mean by 'I wish for my values to be fulfilled.' Indeed, it should understand your meaning better than you do. But superintelligence only implies that the genie's map can compass your true values. Superintelligence doesn't imply that the genie's utility function has terminal values pinned to your True Values, or to the True Meaning of your commands.

The issue of whether the SI's UF contains a set of human values is irrelevant. In a Loosemore architecture, an AI needs to understand and follow the directive "be friendly to humans", and those are all the goals it needs-- to understand, and to follow;

When you write the seed's utility function, you, the programmer, don't understand everything about the nature of human value or meaning. That imperfect understanding remains the causal basis of the fully-grown superintelligence's actions, long after it's become smart enough to fully understand our values.

The UF only needs to contain "understand English, and obey this directive". You don't have to code semantics into the UF. You do of course, have to code it in somewhere,

Instead, we have to give it criteria we think are good indicators of Friendliness, so it'll know what to self-modify toward

A problem which has been solved over and over by humans. Humans don't need to be loaded apriori with what makes other humans happy, they only need to know general indicators, like smiles and statements of approval.

Yes, the UFAI will be able to solve Friendliness Theory. But if we haven't already solved it on our own power, we can't pinpoint Friendliness in advance, out of the space of utility functions. And if we can't pinpoint it with enough detail to draw a road map to it and it alone, we can't program the AI to care about conforming itself with that particular idiosyncratic algorithm.

Why would that be necessary? In the Loosemore architecture, the AGI has the goals of understanding English and obeying the Be Friendly directive. It eventually gets a detailed, extensional, understanding of Friendliness from pursuing those goals, Why would it need to be preloaded with a detailed, extensional unpacking of friendliness? It could fail in understanding English, of course. But there is no reason to think it is unlikely to fail at understanding "friendliness" specifically, and its competence can be tested as you go along.

And if we can't pinpoint it with enough detail to draw a road map to it and it alone, we can't program the AI to care about conforming itself with that particular idiosyncratic algorithm.

I don't see the problem. In the Loosemore architecture, the AGI will care about obeying "be friendly", and it will arrive at the detailed expansion, the idiosyncracies, of "friendly" as part of its other goal to understand English. It cares about being friendly, and it knows the detailed expansion of friendliness, so where's the problem?

Yes, the UFAI will be able to self-modify to become Friendly, if it so wishes. But if there is no seed of Friendliness already at the heart of the AI's decision criteria, no argument or discovery will spontaneously change its heart.

Says who? It has the high level directive, and another directive to understand the directive. It's been Friendly in principle all along, it just needs to fill in the details.

Unless we ourselves figure out how to program the AI to terminally value its programmers' True Intentions,

Then we do need to figure out how to program the AI to terminally value its programmers' True Intentions. That is hardly a fatal objection. Did you think the Loosemore architecture was one that bootstraps itself without any basic goals?

And if we do discover the specific lines of code that will get an AI to perfectly care about its programmer's True Intentions, such that it reliably self-modifies to better fit them — well, then that will just mean that we've solved Friendliness Theory.

No. The goal to understand English is not the same as a goal to be friendly in every way, it is more constrained.

Solving Friendliness, in the MIRI sense, means preloading a detailed expansion of "friendly". That is not what is happening in the Loosemore architecture. So it is not equivalent to solving the same problem.

The clever hack that makes further Friendliness research unnecessary is Friendliness.

Nope.

Intelligence on its own does not imply Friendliness.

That is an open question.

It's true that a sufficiently advanced superintelligence should be able to acquire both abilities. But we don't have them both, and a pre-FOOM self-improving AGI ('seed') need not have both. Being able to program good programmers is all that's required for an intelligence explosion; but being a good programmer doesn't imply that one is a superlative moral psychologist or moral philosopher.

Then hurrah for the Loosemore architecture, which doesn't require humans to"solve" friendliness in the MIRI sense.

You need non-cyclical reasoning. Which would generally be something where you aren't the one having to explain people that the achievement in question is profound.

This bit confuses me.

That aside:

You think Yudkowsky is not a crank, so you think the folks that play that silly game with him are intelligent and rational

Non sequitur. From the posts they make, everyone on this site seems to me to be sufficiently intelligent as to make "selling snake oil" impossible, in a cut-and-dry case like the AI box. Yudowsky's own credibility doesn't enter into it.

The argument against p-zombies is that there is no physical difference that could explain the difference in consciousness. That does not extend to silicon WBEs or AIs

Two things. 1) that the same electronic functioning produces consciousness if implemented on biological goo but does not if implemented on silicon seems unlikely, what probability would you assign that this is the meaningful difference? 2) if it is biological goo we need to have consciousness, why not build an AI out of biological goo? Why not synthesize neurons and stack and connect them in the appropriate ways, and have understood the whole process well enough that either you assemble it working or you know how to start it? It would still be artificial, but made from materials that can produce consciousness when functioning.

Mary isn't a normal human.

If this is the case then, as I said before, my intuition that she would not understand qualia disappears.

“When you’re young, you look at television and think, There’s a conspiracy. The networks have conspired to dumb us down. But when you get a little older, you realize that’s not true. The networks are in business to give people exactly what they want. That’s a far more depressing thought. Conspiracy is optimistic! You can shoot the bastards! We can have a revolution! But the networks are really in business to give people what they want. It’s the truth.”

-Steve Jobs, [Wired, February 1996]

You... disagree? Do you mean your own intuition is different, or do you mean you have some special insight into my psychology that tells you that I'm misunderstanding or misrepresenting my own intuitions?

I mean my intuition is different.

I don't feel that mental states are simple! Yet the Mary hunch persists. You seem to be hopping back and forth between the explanations 'qualia seem irreducible because we don't know enough about them yet' and 'qualia seem irreducible because we don't realize how complicated they are'.

Alright, I'll try to stop hopping and nail down what I'm saying:

1. I think the most likely reason that qualia seem irreducible is because of some kind of software problem in the brain that makes it extremely difficult, if not impossible, for us to translate the sort of "experiential knowledge" found in the unconscious "black box" parts of the brain into the sort of verbal, propositional knowledge that we can communicate to other people by language. The high complexity of our minds probably compounds the difficulty even further.

2. I think this problem goes both ways. So even if we could get some kind of AI to translate the knowledge into verbal statements for us, it would be impossible, or very difficult, for anything resembling a normal human to gain "experiential knowledge" just by reading the verbal statements.

3. In addition to making qualia seem irreducible, this phenomenon explains other things, such as the fact that many activities are easier to learn to do by experience.

I've never actually read any Denett, except for short summaries of some of his criticisms written by other people. One person who has influenced me a lot is Thomas Sowell, who frequently argues that the most important knowledge is implicit and extremely difficult, if not impossible, to articulate into verbal form. He does this in terms of economics, but when I started reading about the ineffability of qualia I immediately began to think "This probably has a similar explanation."

She isn't generating Red, she's generating a memory of the feeling Red generates without generating Red. She now knows what emotional state Red would make her feel, but hasn't actually made herself see red. So when she goes outside she doesn't say "Wow" she says "Oh, those feelings again, just as I suspected."

So she's bound and gagged, with no ability to use her knowledge? Seems implausible, but OK. (Did she get this knowledge by dictation, or by magically reaching out to the Aristotelian essences of neurons?)

In any case, at least two of us have linked to orthonormal's mini-sequence on the matter. Those three posts seem much better than ESR's attempt at the quest.

Ex hypothesi, Mary knows all the relevant third-person specifiable color facts. Our inability to simulate her well doesn't change that fact.

It does if our inability to simulate her well messes with our intuitions. If, as I conjectured, we tend to translate "omniscient person" with "scholar with lots of book-learning" then our intuitions will reflect that, and will hence be wrong.

Consider the Marianna variant.....But she still lacks the relevant items of knowledge about what other people experience

Is Marianna omniscient about light and neuroscience like Mary? If she is, she'd be able to figure out which color is which fairly easily.

If it's merely a matter of qualia being complicated, then shouldn't all other complicated systems yield relevantly identical Hard Problem intuitions?

It's not just a matter of qualia being complicated, it's a matter of the human brain being bad at communicating certain things, of which qualia are only one thing of many. And this isn't just an issue of processing power and the complexity of something being processed, it's an issue of software problems. There are certain problems we have trouble processing regardless of what level of power we have, because of our mind's internal architecture. Wei Dei puts it well when he says:

...a quale is like a handle to a kernel object in programming. Subconscious brain corresponds to the OS kernel, and conscious brain corresponds to user-space. When you see red, you get a handle to a "redness" object, which you can perform certain queries and operations on, such as "does this make me feel hot or cold", or "how similar is this color to this other color" but you can't directly access the underlying data structure. Nor can the conscious brain cause the redness object to be serialized into a description that can be deserialized in another brain to recreate the object. Nor can Mary instantiate a redness object in her brain by studying neuroscience.

Furthermore, there are in fact other things that humans have a lot of difficulty communicating besides qualia. For instance, it's common knowledge that people with a few days of job experience are much better at doing jobs than people who have spent months reading about the job.

My intuition is that making Mary superhuman doesn't change that experiencing red seems to narrow down the possibilities for her.

I disagree. If Mary was a superhuman she could study what functions of the brain cause us to experience "qualia," and then study the memories these processes generated. She could then generate such memories in her own brain, giving her the knowledge of what qualia feel like without ever experiencing them. She would see red and not be surprised at all.

If qualia were not a physical part of the brain, duplicating the memories of someone who had experienced them would not have this effect. However, I think it very likely that doing so would have this effect.

Can you explain why this intuition persists for me, when (as far as I can tell) it doesn't for any other complex system?

Because, as I said before, our emotions are "black boxes" that humans are very bad at understanding and explaining. Their Kolmogorov complexity is extraordinarily high, but we feel like they are simple because of our familiarity with them.

Maybe, but in that case the challenge is to explain, at least schematically, what superhuman power Mary obtains that lets her solve the Hard Problem.

I think the ability to study and modify her own source code and memory, as well as the source code and memory of others is probably all she'd need, but I could be wrong.

Kidney dyalisis machines don't need nephrons, but that doens't mean nephrons are causally idle in kidneys.

You keep bringing up that argument, but kidney dialysis machines are built specifically to replace the functionality of kidneys ("deliberately replacing them with a substitute"). If you built a kidney-dialysis machine by a 1:1 mapping and forgot some cell type that is causally active in kidneys, the machine would not actually work. If it did, you should question if that cell type actually does anything in kidneys.

Changing the physical substrate could remove the qualia, but to claim it could remove the qualia while keeping talk of qualia alive, by sheer coincidence - implying that there's a separate, unrelated reason why the replacement neurons talk of qualia, that has nothing to do with qualia, that was not deliberately engineered - that stretches belief past the breaking point. You're saying, essentially: "qualia cause talk of qualia in my meatbrain, but talk of qualia is not any indication of qualia in any differently built brain implementing the same spec". Then why are you so certain that your talk of qualia is caused by your supposed qualia, and not the neural analogue of what causes talk of qualia in WBE brains? It really does sound like your qualia are either superfluous or bizarre.

[edit] Actually, I'm still not sure I understand you. Are you proposing that it's impossible to build a straight neuron substitute that talks of qualia, without engineering purposeful qualia-talk-emulation machinery? Is that what you mean by "functional equivalent"? I'm having serious trouble comprehending your position.

[edit] I went back to your original comment, and I think we're using "functional equivalence" in a very different sense. To you, it seems to indicate "a system that behaves in the same way despite having potentially hugely different internal architecture". To me, it indicates a 1:1 neuron computational replacement; keeping the computational processes while running them on a different substrate.

I agree that there may conceivably exist functionally equivalent systems that don't have qualia, even though I have difficulty seeing how they could compute "talk of qualia" without running a sufficient-fidelity qualia simulation internally, which would again correspond to our qualia. However, I find it unlikely that anybody who is not a very very bored deity would ever actually create such a system - the qualia-talk machinery seems completely pointless to its function, as well as probably much more computationally expensive. (This system has to be self-deluding in a way consistent with a simpler system that it is not allowed to emulate) Why not just build a regular qualia engine, by copying the meat-brain processes 1:1? That's what I'd consider the "natural" functional-equivalence system.