I think this is an open question, but certainly one approach is to follow the brain's lead and make a system that learns its ethics and high level goals dynamically, through learning.

In that type of design, the initial motivation gets imprinting queues from the parents.

This seems like a non-answer to me.

You can't just say 'learning' as if all possible minds will learn the same things from the same input, and internalize the same values from it.

There is something you have to hardcode to get it to adopt any values at all.

your algorithms converge on some asymptotic limit for the hardware.

Well, what is that limit?

It seems to me that an imaginary perfectly efficient algorithm would read process and output data as fast as the processor could shuffle the bits around, which is probably far faster than it could exchange data with the outside world.

Even if we take that down 1000x becsaue this is an algorithm that's doing actual thinking, you're looking at an easy couple of million bytes per second. And that's superintelligently optimized structured output based on preprocessed efficient input. Because this is AGI, we don't need to count in say, raw video bandwidth, because that can be preprocessed by a system that is not generally intelligent.

So a conservatively low upper limit for my PC's intelligence is outputting a million bytes per second of compressed poetry, or viral genomes, or viral genomes that write poetry.

If the first Superhuman AGI is only superhuman by an order of magnitude or so, or must run on a vastly more powerful system, then you can bet that it's algorithms are many orders of magnitude less efficient than they could be.

Because FOOM is just exponential growth

No.

Why couldn't your supercomputer AGI enter into a growth phase higher than exponential?

Example: If not-too-bright but technological aliens saw us take a slow general purpose computer, and then make a chip that worked 100 times faster, but they didn't know how to put algorithms on a chip, then it would look like our technology got 1000 times better really quickly. But that's just because they didn't already know the trick. If they learned the trick, they could make some of their dedicated software systems work 1000 times faster.

"Convert algorithm to silicon." is just one procedure for speeding things up that an agent can do, or not yet know how to do. You know it's possible, and a superintelligence would figure it out, but how do you rule out a superintelligence figureing out twelve trick like that, which each provide a 1000x speedup. In it's first calendar month?

This should do it. Put a copy of jquery-1.4.2.min.js in the same directory (jQuery is a Javascript library for keeping browser compatibility bugs at bay).

<script type="text/javascript" src="jquery-1.4.2.min.js"></script>
<script>
function setColorScheme(colorScheme)
{
    setCookie("colorScheme", colorScheme, 30);

    if(colorScheme=="whiteOnBlack")
    {
        $("body").css({
            "background-color": "black",
            "color": "white"
        });
    }
    else if(colorScheme=="blackOnWhite")
    {
        $("body").css({
            "background-color": "white",
            "color": "black"
        });
    }
}

$(document).ready(function()
{
    var cookie = getCookie("colorScheme");
    if(cookie)
        setColorScheme(cookie);
});

function setCookie(c_name,value,expiredays)
{
    var exdate=new Date();
    exdate.setDate(exdate.getDate()+expiredays);
    document.cookie=c_name+ "=" +escape(value)+
    ((expiredays==null) ? "" : ";expires="+exdate.toUTCString());
}

function getCookie(c_name)
{
    if (document.cookie.length>0)
    {
        c_start=document.cookie.indexOf(c_name + "=");
        if (c_start!=-1)
        {
            c_start=c_start + c_name.length+1;
            c_end=document.cookie.indexOf(";",c_start);
            if (c_end==-1) c_end=document.cookie.length;
            return unescape(document.cookie.substring(c_start,c_end));
        }
    }
    return "";
}
</script>

Background color: <a href="javascript:setColorScheme('blackOnWhite')">white</a> | <a href="javascript:setColorScheme('whiteOnBlack')">black</a>

This stores the color scheme in a cookie, so it's preserved when you move between chapters. The setCookie and getCookie functions came from here. Given the example, this should be pretty easy to extend for things like changing text size and margins, if you are so inclined.

Nice catch.

Good show, reduced my Harry withdrawal a bit, though not hitting all the same achy spots.

Here's to Rosalie totally having thought of offing Bella and conveniently appropriating, err, adopting, her eggs.

Jacob, I am the only one replying to your replies to me (and no one is voting me up). I choose to take that as a sign that this thread is insufficiently interesting to sufficient numbers of LWers for me to continue.

Note that doing so is not a norm of this community although I would like it if it were and it was IIRC one of the planks or principles of a small movement on Usenet in the 1990s or very early 2000s.

We are clearly talking past each other, and it does not seem to me that it would be productive to continue.

For example, I have repeatedly responded to your claim (not explanation!) that the 2a utility function is not susceptible to "revealed preference". You have never acknowledged my response, but continue claiming that you have explained it to me.

I simply don't understand your theory well enough to criticize it. ... what is the decision theory you end up with?

It is a class of DTs, the kind that count acausal benefits (SAMELs) on par with causal ones.

I have to interpret that as a policy of using some other kind of "surgery" for counterfactuals. Something other than the standard kind of surgery used in causal decision theory (CDT). So the obvious questions become, "So, what kind of surgery do you advocate?" and "How do you know when to use this strange surgery rather than the one Pearl suggests?".

The SAMELs need not be consciously recognized as such but they do need to feel different to motivate the behavior.

That sentence may mean something to you, but I can't even tell who is doing the feeling, what that feeling is different from, and what (or who) is doing the motivating.

... You more than made up for it with the Parfit's robot idea, though.

It wasn't my idea. It was timtyler's. Maybe you will have better luck explaining your ideas to him. He was patient enough to explain the robot to me twice.

But I've noticed in many people (myself included) a definite tendency to overvalue intelligence relative to practice.

I'm not sure I agree with that. In what areas do you see overvalue of intelligence relative to practice and why do you think there really is overvalue in those areas?

I've noticed for example that people's abilities to make good comments on LW do not seem to improve much with practice and feedback from votes (beyond maybe the first few weeks or so). Does this view represent an overvalue of intelligence?

Programming is the art of figuring out what you want so precisely that you can tell even a machine how to do it.

No, you're right, that's in there. I'm not familiar with the source material, I suppose.

Hypothetically, would it cause a problem if a human somehow disambiguated her entire preference graph?

If conscious processing is required to do that, you probably don't want to disambiguate all possible tortures where you're not really sure which one is worse, exactly.

(I mean, unless the choice is actually going to come up, is there really a reason to know for sure which kind of pliers you'd prefer to have your fingernails ripped out with?)

Now, if you limit that preference graph to pleasant experiences, that would at least be an improvement. But even then, you still get the subjective experience of a lifetime of doing nothing but making difficult decisions!

These problems go away if you leave the preference graph ambiguous (wherever it's currently ambiguous), because then you can definitely avoid simulating conscious experiences.

(Note that this also isn't a problem if all you want to do is get a rough idea of what positive and/or negative reactions someone will initially have to a given world state, which is not the same as computing their totally ordered preference over some set of possible world states.)

I'd prefer Sunday or Saturday (9/5 would work for me.)

No, she had control in canon too. Luminous!Bella is only a touch more self-possessed. My explanation - maybe I haven't played it up enough? - is that if you know something about what to expect, you will do a better job of handling it. This is borne out by Carlisle being immune to wanting to nom humans, and having known vampires existed before he was bitten. The others have more trouble, and the one who has the most trouble is Jasper - who also was turned the most unexpectedly and traumatically, while the others at least expected to die of injury/sickness or had amnesia.

Why? Do you agree with him? :)

Yvain's post suggested it; I just stuck it in my cache.

Ok, I accept your argument that Occam is neutral between you and I.

Hold on -- that's not what I said. I said that it was neutral on the issue of including "they can only use decision theories that could survive natural selection". I claim it is not neutral on the supposition of additional terms in the utility function, as 2a does.

SAMELs aren't involved at decision time in 2b, just as "Inclusive fitness" and "Hamilton's rule" aren't involved at decision time in 2a.

It doesn't matter. They (inclusive fitness and Hamilton's rule) have to be assumed (or implied by something that has to be assumed) anyway, because we're dealing with people, so they'll add the same complexity to both explanations.

I will point out though, since we are looking only at the present, that the utility function in 2a can, in principle, be examined using "revealed preference", whereas your purely selfish child-neutral utility function is a theoretical construct which would be hard to measure, even in principle.

As I've explained to you several times, looking at actions does not imply a unique utility function, so you can't claim that you've measured it just by looking at their actions. The utility functions "I care about myself and my child" and "I care about myself" can produce the same actions, as I've demonstrated, because certain (biologically plausible) decision theories can output the action "care for child at expense of self", even in the absence of a causal benefit to the self.

I simply don't understand your theory well enough to criticize it. ... what is the decision theory you end up with?

It is a class of DTs, the kind that count acausal benefits (SAMELs) on par with causal ones. The SAMELs need not be consciously recognized as such but they do need to feel different to motivate the behavior.

However, I could be more helpful if you asked specific questions about specific passages. Previously, you claimed that after reading it, you didn't see how natural selection is like Omega, even after I pointed to the passage. That made me a sad panda.

You more than made up for it with the Parfit's robot idea, though :-)

I want to second RolfAndreassen' viewpoint below.

The problem with this entire train of thought is that you completely skip past the actual real difficulty, which is constructing any type of utility function even remotely as complex as the one you propose.

Your hypothetical utility function references undefined concepts such as "taking control of", "cooperating", "humans", and "self", etc etc

If you actually try to ground your utility function and go through the work of making it realistic, you quickly find that it ends up being something on the order of complexity of a human brain, and its not something that you can easily define in a few pages of math.

I'm skeptical then about the entire concept of 'utility function filters', as it seems their complexity would be on the order of or greater than the utility function itself, and you need to keep constructing an endless sequence of such complex utility function filters.

A more profitable route, it seems to me, is something like this:

Put the AI's in a matrix-like sim (future evolution of current computer game & film simulation tech) and get a community of a few thousand humans to take part in a Truman Show like experiment. Indeed, some people would pay to spectate or even participate, so it could even be a for profit venture. A hierarchy of admins and control would ensure that potential 'liberators' were protected against. In the worst case, you can always just rewind time. (something the Truman Show could never do - a fundamental advantage of a massive sim)

The 'filter function' operates at the entire modal level of reality: the AI's think they are humans, and do not know they are in a sim. And even if they suspected they were in a sim (ie by figuring out the simulation argument), they wouldn't know who were humans and who were AI's (and indeed they wouldn't know which category they were in). As the human operators would have godlike monitoring capability over the entire sim, including even an ability to monitor AI thought activity, this should make a high level of control possible.

They can't turn against humans in the outside world if they don't even believe it exists.

This sounds like a science fiction scenario (and it is), but it's also feasible, and I'd say far more feasible than approaches which directly try to modify, edit, or guarantee mindstates of AI's who are allowed to actually know they are AIs.

You're right. Her control falls squarely under the single-cause rule.

Her control as a newborn is her luminosity, so far as I can determine - she has a set of generalized techniques to influence her desires and emotions, and takes advantage of the specific knowledge which she acquired from the others in her coven to focus these techniques on the precise alterations she needs to be a well-controlled newborn.

I hadn't thought about it like that, but on reflection, I think that it does qualify for the pass. It helps, too, that it's not a general immunity to witches -- she's still vulnerable to Alice and Jasper, for example.

What about her control as a newborn? Does that count?

No, no website-- it's just me right now, and work started about a week ago, so it'll be a while yet. Calling it a "startup" is just a way to reassure my parents that I'm doing something with my time :)

The basic premises behind my approach to game-creation software are:

• The game must always be runnable
• The game must be easily shareable to OSX and Windows users
• The user cannot program program (variables, threading, even loops and math should be as scarce as possible)
• Limitations must be strict (don't let users try to create blockbuster-level games, or they'll become discouraged and stop trying)

I'd like to get a working prototype up, send it around to a few testers, and iterate before getting sidetracked into web design. I've found that I can sink a distressing amount of time into getting my CSS "just right". I'll definitely put you on my list for v.0.5 if you PM me an email address.

I see you did a game startup for some years; any tips for someone just starting out? And does your current venture have a website?