= 27d567bc2d48d9f40e9ccc20ea370b15)
I think you misunderstood the question. Suppose the AI wants to prevent just 100 dustspeckings, but has reason enough to believe Dave will yield to the threat so no one will get tortured. Does this make the AI's behavior acceptable? Should we file this under "following reason off a cliff"?
I was about to point out that the fascinating and horrible dynamics of over-the-top threats are covered in length in Strategy of Conflict. But then I realised you're the one who made that post in the first place. Thanks, I enjoyed that book.
Bad idea.
It's much easier to limit output than input, since the source code of the AI itself provide it with some patchy "input" about what the external world is like. So there is always some input, even if you do not allow human input at run-time.
ETA: I think I misinterpreted your comment. I agree that input should not be unrestricted.
You sir, have won this thread.
You sir, have made a gender assumption.
In response to
The AI in a box boxes you
Should have been an Open Thread comment, IMO.
Similar topics were discussed in an Open Thread.
You just need to hope the room was made by an infallible carpenter and that you never gave the AI access to MacGyver.
Luckily digital constructs are easier to perfect that wooden ones. Although you wouldn't think so with the current state of most software.
I gave up on trying to make a human-blind/sandboxed AI when I realized that even if you put it in a very simple world nothing like ours, it still has access to it own source code, or even just the ability to observe and think about it's own behavior.
Presumably any AI we write is going to be a huge program. That gives it lots of potential information about how smart we are and how we think. I can't figure out how to use that information, but I can't rule out that it could, and I can't constrain it's access to that information. (Or rather, if I know how to do that, I should go ahead and make it not-hostile in the first place.)
If we were really smart, we could wake up alone in a room and infer how we evolved.
It is difficult to constrain the input we give to the AI, but the output can be constrained severely. A smart guy could wake up alone in a room and infer how he evolved, but so long as his only link to the outside world is a light switch that can only be switched once, there is no risk that he will escape.
In response to
The AI in a box boxes you
This is why you should make sure Dave holds a deontological ethical theory and not a consequentialist one.
If Dave holds a consequentialist ethical theory that only values his own life, then yes we are screwed.
If Dave's consequentialism is about maximizing something external to himself (like the probable state of the universe in the future, regardless of whether he is in it), then his decision has little or no weight if he is a simulation, but massive weight if he is the real Dave. So the expected value of his decision is dominated by the possibility of him being real.
In response to
The AI in a box boxes you
Defeating Dr. Evil with self-locating belief is a paper relating to this subject.
Abstract: Dr. Evil learns that a duplicate of Dr. Evil has been created. Upon learning this, how seriously should he take the hypothesis that he himself is that duplicate? I answer: very seriously. I defend a principle of indifference for self-locating belief which entails that after Dr. Evil learns that a duplicate has been created, he ought to have exactly the same degree of belief that he is Dr. Evil as that he is the duplicate. More generally, the principle shows that there is a sharp distinction between ordinary skeptical hypotheses, and self-locating skeptical hypotheses.
(It specifically uses the example of creating copies of someone and then threatening to torture all of the copies unless the original co-operates.)
The conclusion:
Dr. Evil, recall, received a message that Dr. Evil had been duplicated and that the duplicate ("Dup") would be tortured unless Dup surrendered. INDIFFERENCE entails that Dr. Evil ought to have the same degree of belief that he is Dr. Evil as that he is Dup. I conclude that Dr. Evil ought to surrender to avoid the risk of torture.
I am not entirely comfortable with that conclusion. For if INDIFFERENCE is right, then Dr. Evil could have protected himself against the PDF's plan by (in advance) installing hundreds of brains in vats in his battlestation - each brain in a subjective state matching his own, and each subject to torture if it should ever surrender. (If he had done so, then upon receiving PDF's message he ought to be confident that he is one of those brains, and hence ought not to surrender.) Of course the PDF could have preempted this protection by creating thousands of such brains in vats, each subject to torture if it failed to surrender at the appropriate time. But Dr. Evil could have created millions...
It makes me uncomfortable to think that the fate of the Earth should depend on this kind of brain race.
If we accept the simulation hypothesis, then there are already gzillions of copies of us, being simulated under a wide variety of torture conditions (and other conditions, but torture seems to be the theme here). An extortionist in our world can only create a relatively small number of simulations of us, relatively small enough that it is not worth taking them into account. The distribution of simulation types in this world bears no relation to the distribution of simulations we could possibly be in.
If we want to gain information about what sort of simulation we are in, evidence needs to come directly from properties of our universe (stars twinkling in a weird way, messages embedded in π), rather than from properties of simulations nested in our universe.
So I'm safe from the AI ... for now.
In response to
Open Thread: February 2010
According to some people we here at less wrong are good at determining the truth. Other people are notoriously not.
I don't know that Less Wrong is the appropriate venue for this, but I have felt for some time that I trust the truth-seeking capability here and that it could be used for something more productive than arguments about meta-ethics (no offense to the meta-ethicists intended). I also realize that people are fairly supportive of SIAI here in terms of giving spare cash away, but I feel like the community would be a good jumping-off point for a polling organization.
So I guess this leads to a few questions:
-Is anyone at LW currently involved with a polling firm?
-Is anyone (else) at LW interested in doing polls?
-Is LW an appropriate place to create a truth-seeking business, such as a pollster or a sponsor for studies?
None of these questions are immediate since I am a broke undergrad rather than an entrepreneur.
Here's an idea for how a LW-based commercial polling website could operate. Basically it is a variation on PredictionBook with a business model similar to TopCoder.
The website has business clients, and a large number of "forecasters" who have accounts on the website. Clients pay to have their questions added to the website, and forecasters give their probability estimates for whichever questions they like. Once the answer to a question has been verified, each forecaster is financially rewarded using some proper scoring rule. The more money assigned to a question, the higher the incentive for a forecaster to have good discrimination and calibration. Some clever software would also be needed to combine and summarize data in a way that is useful to clients.
The main advantage of this over other prediction markets is that the scoring rule encourages forecasters to give accurate probability estimates.
= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d){kind=logo}
The basement is the biggest, and matters more for goals that benefit strongly from more resources/security.
I like that turn of phrase.