= 27d567bc2d48d9f40e9ccc20ea370b15)
I don't see how you can build a human-level intelligence without making it at least somewhat consequentialist. If it doesn't decide actions based on something like expected utility maximization, how does it decide actions?
What I was referring to is the difference between:
A) An AI that accepts an instruction from the user, thinks about how to carry out the instruction, comes up with a plan, checks that the user agrees that this is a good plan, carries it out, then goes back to an idle loop.
B) An AI that has a fully realized goal system that has some variant of 'do what I'm told' implemented as a top-level goal, and spends its time sitting around waiting for someone to give it a command so it can get a reward signal.
Either AI will kill you (or worse) in some unexpected way if it's a full-blown superintelligence. But option B has all sorts of failure modes that don't exist in option A, because of that extra complexity (and flexibility) in the goal system. I wouldn't trust a type B system with the IQ of a monkey, because it's too likely to find some hilariously undesirable way of getting its goal fulfilled. But a type A system could probably be a bit smarter than its user without causing any disasters, as long as it doesn't unexpectedly go FOOOM.
Of course, there's a sense in which you could say that the type A system doesn't have human-level intelligence no matter how impressive its problem-solving abilities are. But if all you're looking for is an automated problem-solving tool that's not really an issue.
The standard solution to that is to be altruistic to some group of people as they existed at time T, and the standard problem with that is it doesn't allow moral progress, and the standard solution to that is to be altruistic to some idealized or extrapolated group of people. So we just have to make the heuristics-based FAI understand the concept of CEV (or whatever the right notion of "idealized" is), which doesn't seem impossible. What does seem impossible is to achieve high confidence that it understands the notion correctly, but if provably-Friendly AI is just too slow or unfeasible, and we're not trying to achieve 100% safety...
I thought that too until I spent a few hours thinking about how to actually implement CEV, after which I realized that any AI capable of using that monster of an algorithm is already a superintelligence (and probably turned the Earth into computronium while it was trying to get enough CPU power to bootstrap its goal system).
Anyone who wants to try a "build moderately smart AGI to help design the really dangerous AGI" approach is probably better off just making a genie machine (i.e. an AI that just does whatever its told, and doesn't have explicit goals independent of that). At least that way the failure modes are somewhat predictable, and you can probably get to a decent multiple of human intelligence before accidentally killing everyone.
I've been confused for a while by the idea that an AI should be able to modify itself at all. Self-modifying systems are difficult to reason about. If an AI modifies itself stupidly, there's a good chance it will completely break. If a self-modifying AI is malicious, it will be able to ruin whatever fancy safety features it has.
A non-self-modifying AI wouldn't have any of the above problems. It would, of course, have some new problems. If it encounters a bug in itself, it won't be able to fix itself (though it may be able to report the bug). The only way it would be able to increase its own intelligence is by improving the data it operates on. If the "data it operates on" includes a database of useful reasoning methods, then I don't see how this would be a problem in practice.
I can think of a few of arguments against my point:
- There's no clear boundary between a self-modifying program and a non-self-modifying program. That's true, but I think the term "non-self-modifying" implies that the program cannot make arbitrary changes to its own source code, nor cause its behavior to become identical to the behavior of an arbitrary program.
- The ability to make arbitrary calculations is effectively the same as the ability to make arbitrary changes to one's own source code. This is wrong, unless the AI is capable of completely controlling all of its I/O facilities.
- The AI being able to fix its own bugs is really important. If the AI has so many bugs that they can't all be fixed manually, and it is important that these bugs be fixed, and yet the AI does run well enough that it can actually fix all the bugs without introducing more new ones... then I'm surprised.
- Having a "database of useful reasoning methods" wouldn't provide enough flexibility for the AI to become superintelligent. This may be true.
- Having a "database of useful reasoning methods" would provide enough flexibility for the AI to effectively modify itself arbitrarily. It seems like it should be possible to admit "valid" reasoning methods like "estimate the probability of statement P, and, if it's at least 90%, estimate the probability of Q given P", while not allowing "invalid" reasoning methods like "set the probability of statement P to 0".
The last item on your list is an intractable sticking point. Any AGI smart enough to be worth worrying about is going to have to have the ability to make arbitrary changes to an internal "knowledge+skills" representation that is itself a Turing-complete programming language. As the AGI grows it will tend to create an increasingly complex ecology of AI-fragments in this way, and predicting the behavior of the whole system quickly becomes impossible.
So "don't let the AI modify its own goal system" ends up turning into just anther way of saying "put the AI in a box". Unless you have some provable method of ensuring that no meta-meta-meta-meta-program hidden deep in the AGI's evolving skill set ever starts acting like a nested mind with different goals than its host, all you've done is postpone the problem a little bit.
You'll want to read an earlier Yvain blog post, then, explaining "many reasons to expect that arguments for socially dominant beliefs (which correlate highly with truth) to be worse than the arguments for fringe beliefs (which probably correlate highly with falsehood)".
Why would you expect the social dominance of a belief to correlate with truth? Except in the most trivial cases, society has no particular mechanism that selects for true beliefs in preference to false ones.
The Darwinian competition of memes selects strongly for those that provide psychological benefits, or are politically useful, or serve the self-interest of large segments of the population. But truth is only relevant if the opponents of a belief can easily and unambiguously disprove it, which is only possible in rare cases.
In response to
[Link] Economists' views differ by gender
If true, this is fairly strong evidence that the effort to turn the study of economics into a science has failed. If the beliefs of professional economists about their field of study are substantially affected by their gender, they obviously aren't arriving at those beliefs by a reliable objective process.
Censorship is generally not a wise response to a single instance of any problem. Every increment of censorship you impose will wipe out an unexpectedly broad swath of discussion, make it easier to add more censorship later, and make it harder to resist accusations that you implicitly support any post you don't censor.
If you feel you have to Do Something, a more narrowly-tailored rule that still gets the job done would be something like: "Posts that directly advocate violating the laws of <jurisdiction in which Less Wrong staff live> in a manner likely to create criminal liability will be deleted."
Because, you know, it's just about impossible to talk about specific wars, terrorism, criminal law or even many forms of political activism without advocating real violence against identifiable groups of people.
In response to
newcomb's altruism
Knowing that philosophers are the only people who two-box on Newcomb's problem, and they constitute a vanishingly small fraction of Earth's population, I confidently one-box. Then I rush out to spend my winnings as quickly as possible, before the inevitable inflation hits.
Telling me what X is will have no effect on my action, because I already have that information. Making copies of me has no effect on my strategy, for the same reason.
What is the evidence that 2 is out? Suppose there are five available effective means to some end. If I take away one of them, doesn't that reduce the availability of effective means to that end? Is the idea supposed to be that the various means are all so widely available that overall availability of means to the relevant end is not affected by eliminating (or greatly reducing) availability of one of them? Seems contentious to me. Moreover, what you say after the claim that 2 is out seems rather to support the claim that 2 is basically correct: poison, bombs, and knives are either practically less effective for one reason or another (hard to use, hard to practice, less destructive -- in the case of knives) or practically less available for one reason or another (poisons not widely available).
I think you have a point here, but there's a more fundamental problem - there doesn't seem to be much evidence that gun control affects the ability of criminals to get guns.
The problem here is similar to prohibition of drugs. Guns and ammunition are widely available in many areas, are relatively easy to smuggle, and are durable goods that can be kept in operation for many decades once acquired. Also, the fact that police and other security officials need them means that they will continue to be produced and/or imported into an area with even very strict prohibition, creating many opportunities for weapons to leak out of official hands.
So gun control measures are much better at disarming law-abiding citizens than criminals. Use of guns by criminals does seem to drop a bit when a nation adopts strict gun control policies for a long period of time, but the fact that the victims have been disarmed also means criminals don't need as many guns. If your goal is disarming criminals it isn't at all clear that this is a net benefit.
In response to
Gun Control: How would we know?
if that was answerable we as a culture wouldn't still be arguing about it.
Who is "we", white man? ;)
Here in France there seems to be very little public debate around firearm legislation - there probably is some, somewhere, but it gets very little media attention.
The existence of public debate around an issue doesn't seem to be very strong evidence on whether the question can be answered conclusively given the available evidence, since public debate around issues varies from country to country, and in some places there are public debates around issues where anybody with half a brain should be able to tell which side is right. Heck, sometimes there's a lack of public debate because everybody agrees on the wrong conclusion.
Agreed. Presence or absence of debate on an issue gives information about a nation's culture, but very little about how hard it is to discover the facts of the matter. This is especially true in matters of social science, where the available evidence is never going to be strong enough to convince someone who has already made up his mind.
= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d){kind=logo}
Are you sure it would have to be able to make arbitrary changes to the knowledge representation? Perhaps there's a way to filter out all of the invalid changes that could possibly be made, the same way that computer proof verifiers have a way to filter out all possible invalid proofs.
I'm not sure what you're saying at all about the Turing-complete programming language. A programming language is a map from strings onto computer programs; are you saying that the knowledge representation would be a computer program?
Yes, I'm saying that to get human-like learning the AI has to have the ability to write code that it will later use to perform cognitive tasks. You can't get human-level intelligence out of a hand-coded program operating on a passive database of information using only fixed, hand-written algorithms.
So that presents you with the problem of figuring out which AI-written code fragments are safe, not just in isolation, but in all their interactions with every other code fragment the AI will ever write. This is the same kind of problem as creating a secure browser or Java sandbox, only worse. Given that no one has ever come close to solving it for the easy case of resisting human hackers without constant patches, it seems very unrealistic to think that any ad-hoc approach is going to work.