What happens if an AI manages to game the system despite the n layers of abstraction?
Your argument would be stronger if you provided a citation. I've only skimmed CEV, for instance, so I'm not fully familiar with Eliezer strongest arguments in favour of goal structure tending to be preserved (though I know he did argue for that) in the course of intelligence growth. For that matter, I'm not sure what your arguments for goal stability under intelligence improvement are. Nevertheless, consider the following:
...In poetic terms, our coherent extrapolated volition is our wish if we knew more, thought faster, were more the people we wished we wer
We might be approaching a point of diminishing returns as far as improving cultural transmission is concerned. Sure, it would be useful to adopt a better language, e.g. one less ambiguous, less subject to misinterpretation, more revealing of hidden premises and assumptions. More bandwidth and better information retrieval would also help. But I don't think these constraints are what's holding AI back.
Bandwidth, storage, and retrieval can be looked at as hardware issues, and performance in these areas improves both with time and with adding more hardware. What AI requires are improvements in algorithms and in theoretical frameworks such as decision theory, morality, and systems design.
I think it will prove computationally very expensive, both to solve protein folding and to subsequently design a bootstrapping automaton. It might be difficult enough for another method of assembly to come out ahead cost-wise.
You're right, that is more realistic. Even so, I get the feeling that the human would have less and less to do as time goes on. I quote:
“He just loaded up on value stocks,” says Mr. Fleiss, referring to the AI program. The fund gained 41% in 2009, more than doubling the Dow’s 19% gain.
As another data point, a recent chess contest between a chess grandmaster (Daniel Naroditsky) working together with an older AI (Rybka, rated ~3050) and the current best chess AI (Stockfish 5, rated 3290) ended with a 3.5 - 0.5 win for Stockfish.
While not exactly investment, consider the case of an AI competing with a human to devise a progressively better high-frequency trading strategy. An AI would probably:
I expect the AI's superior capacity to "drink from the fire hose" together with its faster response time to yield a higher exponent for the growth function than that resulting from the human's iterative improvement.
The effectiveness of learning hyper-heuristics for other problems, i.e. how much better algorithmically-produced algorithms perform than human-produced algorithms, and more pertinently, where the performance differential (if any) is heading.
As an example, Effective learning hyper-heuristics for the course timetabling problem says: "The dynamic scheme statistically outperforms the static counterpart, and produces competitive results when compared to the state-of-the-art, even producing a new best-known solution. Importantly, our study illustrates that ...
Only problem is cooking. Eats up like 4 hours a week.
This article by Roger Ebert on cooking is, I suspect, highly relevant to your interests. Mine too, as a matter of fact.
For example, consider a system that takes seriously the idea of souls. One might very well decide that all that matters is whether an entity has a soul, completely separate from its apparent intelligence level. Similarly, a sufficiently racist individual might assign no moral weight to people of some specific racial group, regardless of their intelligence.
Right you are. I did not express myself well above. Let me try and restate, just for the record.
Assuming one does not assign equal rights to all autonomous agents (for instance, if we take the position...
I think it would be difficult to construct an ethical system where you give no consideration to cognitive capacity. Is there a practical reason for said superintelligence to not take into account humans' cognitive capacity? Is there a logical reason for same?
Not to make light of a serious question, but, "Equal rights for bacteria!"? I think not.
Aside: I am puzzled as to the most likely reason Esar's comment was downvoted. Was it perhaps considered insufficiently sophisticated, or implying that its poster was insufficiently well-read, for LW?
I'm watching this dialogue now, I'm 45 (of 73) minutes in. I'd just like to remark that:
Aside: what is the LW policy on commenting on old threads? All good? Frowned upon?
Indeed it is. But the way you fight "memetic infection" in the real world is to take a look at the bad stuff and see where it goes wrong, not to isolate yourself from harmful ideas.
Thankfully for Mr. Pratchett, you can't influence the genetic lottery or the luck fairy, so his is still valid advice. In fact, one could see "trust in yourself" et al. as invitations to "do or do not, there is no try", whereas "work hard, learn hard and don't be lazy" supports the virtue of scholarship as well as that of "know when to give up". Miss Tick is being eminently practical, and "do or do not", while also an important virtue, requires way more explanation before the student can understand it.
Hello LessWrong,
I've been reading the website for at least the past two years. I like the site, I admire the community, and I figured I should start commenting.
I like to think of myself as a rationalist. LW, along with other sources (Bertrand Russell, Richard Dawkins) has contributed heavily (and positively) to my mental models. Still, I have a lot of work to do.
I like to learn. I like to discuss. I used to like to engage in heated debates, but this seems to have lost some of its appeal recently--either someone is wrong or isn't, and I prefer to figure out...
Let me clarify why I asked. I think the "multiple layers of abstraction" idea is essentially "build in a lot of 'manual' checks that the AI isn't misbehaving", and I don't think that is a desirable or even possible solution. You can write n layer of checks, but how do you know that you don't need n+1?
The idea being--as has been pointed out here on LW--that what you really want and need is a mathematical model of morality, which the AI will implement and which moral behaviour will fall out of without you having to specify it explicitly. ... (read more)