= 27d567bc2d48d9f40e9ccc20ea370b15)
I don't see how you can build a human-level intelligence without making it at least somewhat consequentialist. If it doesn't decide actions based on something like expected utility maximization, how does it decide actions?
What I was referring to is the difference between:
A) An AI that accepts an instruction from the user, thinks about how to carry out the instruction, comes up with a plan, checks that the user agrees that this is a good plan, carries it out, then goes back to an idle loop.
B) An AI that has a fully realized goal system that has some variant of 'do what I'm told' implemented as a top-level goal, and spends its time sitting around waiting for someone to give it a command so it can get a reward signal.
Either AI will kill you (or worse) in some unexpected way if it's a full-blown superintelligence. But option B has all sorts of failure modes that don't exist in option A, because of that extra complexity (and flexibility) in the goal system. I wouldn't trust a type B system with the IQ of a monkey, because it's too likely to find some hilariously undesirable way of getting its goal fulfilled. But a type A system could probably be a bit smarter than its user without causing any disasters, as long as it doesn't unexpectedly go FOOOM.
Of course, there's a sense in which you could say that the type A system doesn't have human-level intelligence no matter how impressive its problem-solving abilities are. But if all you're looking for is an automated problem-solving tool that's not really an issue.
The standard solution to that is to be altruistic to some group of people as they existed at time T, and the standard problem with that is it doesn't allow moral progress, and the standard solution to that is to be altruistic to some idealized or extrapolated group of people. So we just have to make the heuristics-based FAI understand the concept of CEV (or whatever the right notion of "idealized" is), which doesn't seem impossible. What does seem impossible is to achieve high confidence that it understands the notion correctly, but if provably-Friendly AI is just too slow or unfeasible, and we're not trying to achieve 100% safety...
I thought that too until I spent a few hours thinking about how to actually implement CEV, after which I realized that any AI capable of using that monster of an algorithm is already a superintelligence (and probably turned the Earth into computronium while it was trying to get enough CPU power to bootstrap its goal system).
Anyone who wants to try a "build moderately smart AGI to help design the really dangerous AGI" approach is probably better off just making a genie machine (i.e. an AI that just does whatever its told, and doesn't have explicit goals independent of that). At least that way the failure modes are somewhat predictable, and you can probably get to a decent multiple of human intelligence before accidentally killing everyone.
I've been confused for a while by the idea that an AI should be able to modify itself at all. Self-modifying systems are difficult to reason about. If an AI modifies itself stupidly, there's a good chance it will completely break. If a self-modifying AI is malicious, it will be able to ruin whatever fancy safety features it has.
A non-self-modifying AI wouldn't have any of the above problems. It would, of course, have some new problems. If it encounters a bug in itself, it won't be able to fix itself (though it may be able to report the bug). The only way it would be able to increase its own intelligence is by improving the data it operates on. If the "data it operates on" includes a database of useful reasoning methods, then I don't see how this would be a problem in practice.
I can think of a few of arguments against my point:
- There's no clear boundary between a self-modifying program and a non-self-modifying program. That's true, but I think the term "non-self-modifying" implies that the program cannot make arbitrary changes to its own source code, nor cause its behavior to become identical to the behavior of an arbitrary program.
- The ability to make arbitrary calculations is effectively the same as the ability to make arbitrary changes to one's own source code. This is wrong, unless the AI is capable of completely controlling all of its I/O facilities.
- The AI being able to fix its own bugs is really important. If the AI has so many bugs that they can't all be fixed manually, and it is important that these bugs be fixed, and yet the AI does run well enough that it can actually fix all the bugs without introducing more new ones... then I'm surprised.
- Having a "database of useful reasoning methods" wouldn't provide enough flexibility for the AI to become superintelligent. This may be true.
- Having a "database of useful reasoning methods" would provide enough flexibility for the AI to effectively modify itself arbitrarily. It seems like it should be possible to admit "valid" reasoning methods like "estimate the probability of statement P, and, if it's at least 90%, estimate the probability of Q given P", while not allowing "invalid" reasoning methods like "set the probability of statement P to 0".
The last item on your list is an intractable sticking point. Any AGI smart enough to be worth worrying about is going to have to have the ability to make arbitrary changes to an internal "knowledge+skills" representation that is itself a Turing-complete programming language. As the AGI grows it will tend to create an increasingly complex ecology of AI-fragments in this way, and predicting the behavior of the whole system quickly becomes impossible.
So "don't let the AI modify its own goal system" ends up turning into just anther way of saying "put the AI in a box". Unless you have some provable method of ensuring that no meta-meta-meta-meta-program hidden deep in the AGI's evolving skill set ever starts acting like a nested mind with different goals than its host, all you've done is postpone the problem a little bit.
You'll want to read an earlier Yvain blog post, then, explaining "many reasons to expect that arguments for socially dominant beliefs (which correlate highly with truth) to be worse than the arguments for fringe beliefs (which probably correlate highly with falsehood)".
Why would you expect the social dominance of a belief to correlate with truth? Except in the most trivial cases, society has no particular mechanism that selects for true beliefs in preference to false ones.
The Darwinian competition of memes selects strongly for those that provide psychological benefits, or are politically useful, or serve the self-interest of large segments of the population. But truth is only relevant if the opponents of a belief can easily and unambiguously disprove it, which is only possible in rare cases.
In response to
[Link] Economists' views differ by gender
If true, this is fairly strong evidence that the effort to turn the study of economics into a science has failed. If the beliefs of professional economists about their field of study are substantially affected by their gender, they obviously aren't arriving at those beliefs by a reliable objective process.
Censorship is generally not a wise response to a single instance of any problem. Every increment of censorship you impose will wipe out an unexpectedly broad swath of discussion, make it easier to add more censorship later, and make it harder to resist accusations that you implicitly support any post you don't censor.
If you feel you have to Do Something, a more narrowly-tailored rule that still gets the job done would be something like: "Posts that directly advocate violating the laws of <jurisdiction in which Less Wrong staff live> in a manner likely to create criminal liability will be deleted."
Because, you know, it's just about impossible to talk about specific wars, terrorism, criminal law or even many forms of political activism without advocating real violence against identifiable groups of people.
In response to
comment
by
on
Harry Potter and the Methods of Rationality discussion thread, part 18, chapter 87
You're missing my point by a long ways. I'm not complaining about the main character. I keep explicitly saying "even if you don't count the protagonist". I'm mostly examining how the not-protagonists stack up against each other. Your remark about variance might be on point, except I'm complaining not only about the ratio of competent males to competent females, but also about the specific sorts of insufficiently varied flaws that are depressing the female characters' abilities/badassery.
In response to
comment
by
on
Harry Potter and the Methods of Rationality discussion thread, part 18, chapter 87
I was commenting specifically about the end of your previous comment, not the whole topic. Sorry if that wasn't clear. But as to this new point, why should an author feel obligated to gender-balance the complexity of the flaws they assign to minor characters?
Yes, I'm aware that there's a fairly common intellectual position claiming that authors should devote vast amounts of energy to worrying about that sort of thing. I just think that's a deeply misguided enterprise. A good author will naturally come to a pretty reasonable balance in the natural course of writing a story, and any major tweaking beyond that point is more likely to make the story worse than better.
Do you really think HP:MoR would be a better story if EY had spent a few weeks listing all the characters by gender, and trying to tweak the plot and insert details to 'balance' things? As opposed to, say, working out plot complications or dreaming up new moments of awesome?
In response to
comment
by
on
Harry Potter and the Methods of Rationality discussion thread, part 18, chapter 87
I agree with the "undifferentiated gossiping mass" bit.
Any specific example has a corresponding counterexample. Padma Patil, for example, gets nonzero development and, IIRC, perspective time, which could go a ways to counter "undifferentiated gossiping mass" - but a male character on about her tier of importance, like Blaise Zabini, gets to enact plot and is more distinct as a single person than she is. Even Ron, who is of negligible relevance, has a named skill that differentiates him from the background. Does Padma? As far as I can recall Padma is just sort of generically informedly bright. Hermione's intelligence, gratuitous perfect recall, and magical prowess can go a ways to counter "female characters are less competent" - but the most competent characters, even if you don't count the protagonist, are all male.
In response to
comment
by
on
Harry Potter and the Methods of Rationality discussion thread, part 18, chapter 87
So what?
From a storytelling perspective, authors are not obligated to make their main characters (or even 50% of main characters) female. Considering the way the whole SF&F genre has been taken over by gritty female urban fantasy vampire hunters in recent years, finding a decent story with a male lead is actually a nice change.
From the perspective of realism, the fact that the most competent characters are male is to be expected. That really is the way the world works, thanks to the fact that males have a flatter bell curve with longer tails on just about every measure of ability. It isn't the result of an evil male conspiracy, and there's nothing wrong with an author depicting this elementary fact of (current) human nature accurately.
So I'm left wondering how your comments amount to anything more than "I'm unhappy because you aren't writing the story the way I would have done it."
In response to
newcomb's altruism
Knowing that philosophers are the only people who two-box on Newcomb's problem, and they constitute a vanishingly small fraction of Earth's population, I confidently one-box. Then I rush out to spend my winnings as quickly as possible, before the inevitable inflation hits.
Telling me what X is will have no effect on my action, because I already have that information. Making copies of me has no effect on my strategy, for the same reason.
= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d){kind=logo}
Are you sure it would have to be able to make arbitrary changes to the knowledge representation? Perhaps there's a way to filter out all of the invalid changes that could possibly be made, the same way that computer proof verifiers have a way to filter out all possible invalid proofs.
I'm not sure what you're saying at all about the Turing-complete programming language. A programming language is a map from strings onto computer programs; are you saying that the knowledge representation would be a computer program?
Yes, I'm saying that to get human-like learning the AI has to have the ability to write code that it will later use to perform cognitive tasks. You can't get human-level intelligence out of a hand-coded program operating on a passive database of information using only fixed, hand-written algorithms.
So that presents you with the problem of figuring out which AI-written code fragments are safe, not just in isolation, but in all their interactions with every other code fragment the AI will ever write. This is the same kind of problem as creating a secure browser or Java sandbox, only worse. Given that no one has ever come close to solving it for the easy case of resisting human hackers without constant patches, it seems very unrealistic to think that any ad-hoc approach is going to work.