All of anon85's Comments + Replies
If you have to speak "carefully enough" then you're taking a big risk though you may luck out and get what you want, they're not safe.
If your argument is that unless a powerful being is extremely safe, then they're not extremely safe, this is true by definition. Obviously, if a genie sometimes doesn't give you what you want, there is some risk that the genie won't give you what you want. I thought a more substantial argument was being made, though - it sounded like EY was claiming that saying "I wish for whatever I should wish for" i...
Examples of what? Of hypothetical intelligent minds? I feel like there are examples all over fiction; consider genies themselves, which often grant wishes in a dangerous way (but you can sometimes get around it by speaking carefully enough). Again, I agree that some genies are never safe and some are always safe, but it's easy to imagine a genie which is safe if and only if you specify your wish carefully.
Anyway, do you concede the point that EY's article contains no arguments?
I'm making 2 points:
His metaphor completely fails conceptually, because I'm perfectly capable of imagining genies that fall outside the three categories.
Perhaps the classification works in some other setting, such as AIs. However, the article never provided any arguments for this (or any arguments at all, really). Instead, there was one single example (seriously, just one example!) which was then extrapolated to all genies.
At age 5 you could safely wish for "I wish for you to do what I should wish for" and at worst you'd be a little disappointed if what she came up with wasn't as fun as you'd have liked.
I would have gotten the wrong flavor of ice cream. It was strictly better to specify the flavor of ice cream I preferred. Therefore, the statement about the 3 types of genies is simply false. It might be approximately true in some sense, but even if it is, the article never gives any arguments in favor of that thesis, it simply gives one example.
That sounds pretty similar to a Deist's God, which created the universe but does not interfere thereafter. Personally, I'd just shave it off with Ocam's razor.
Also, it seems a little absurd to try to infer things about our simulators, even supposing they exist. After all, their universe can be almost arbitrarily different from ours.
Does the simulation hypothesis have any predictive power? If so, what does it predict? Is there any way to falsify it?
Oh, yes, me too. I want to engage in one-shot PD games with entirelyuseless (as opposed to other people), because he or she will give me free utility if I sell myself right. I wouldn't want to play one-shot PDs against myself, in the same way that I wouldn't want to play chess against Kasparov.
By the way, note that I usually cooperate in repeated PD games, and most real-life PDs are repeated games. In addition, my utility function takes other people into consideration; I would not screw people over for small personal gains, because I care about their happiness. In other words, defecting in one-shot PDs is entirely consistent with being a decent human being.
Cool, so in conclusion, if we met in real life and played a one-shot PD, you'd (probably) cooperate and I'd defect. My strategy seems superior.
I never liked that article. It says "there are three types of genies", and then, rather than attempting to prove the claim or argue for it, it just provides an example of a genie for which no wish is safe. I mean, fine, I'm convinced that specific genie sucks. But there may well be other genies that don't know what you want but have the ability to give it to you if you ask (when I was 5 years old, my mom was such a genie).
But since you're making it clear that your code is quite different, and in a particular way, I would defect against you.
You don't know who I am! I'm anonymous! Whoever you'd cooperate with, I might be that person (remember, in real life I pretend to have a completely different philosophy on this matter). Unless you defect against ALL HUMANS, you risk cooperating when facing me, since you don't know what my disguise will be.
You can see which side of the room you are on, so you know which one you are.
If I can do this, then my clone and I can do different things. In that case, I can't be guaranteed that if I cooperate, my clone will too (because my decision might have depended on which side of the room I'm on). But I agree that the cloning situation is strange, and that I might cooperate if I'm actually faced with it (though I'm quite sure that I never will).
...People don't actually have the same code, but they have similar code. If the code in some case is similar enough tha
Yes. The universe is deterministic. Your actions are completely predictable, in principle. That's not unique to this thought experiment. That's true for every thing you do. You still have to make a choice. Cooperate or defect?
Um, what? First of all, the universe is not deterministic - quantum mechanics means there's inherent randomness. Secondly, as far as we know, it's consistent with the laws of physics that my actions are fundamentally unpredictable - see here.
Third, if I'm playing against a clone of myself, I don't think it's even a valid PD. Can th...
Well there is no causal influence. Your opponent is deterministic. His choice may have already been made and nothing you do will change it. And yet the best decision is still to cooperate.
If his choice is already made and nothing I do will change it, then by definition my choice is already made and nothing I do will change it. That's why my "decision" in this setting is not even well-defined - I don't really have free will if external agents already know what I will do.
The most obvious example of cooperating due to acausal dependence is making two atom-by-atom-identical copies of an agent and put them in a one-shot prisoner's dilemma against each other. But two agents whose decision-making is 90% similar instead of 100% identical can cooperate on those grounds too, provided the utility of mutual cooperation is sufficiently large.
I'm not sure what "90% similar" means. Either I'm capable of making decisions independently from my opponent, or else I'm not. In real life, I am capable of doing so. The clone situa...
I don't know enough about this to tell if (2) had more influence than (3) initially. I'm glad you agree that (2) had some influence, at least. That was the main part of my point.
How long did discussion of the Basilisk stay banned? Wasn't it many years? How do you explain that, unless the influence of (2) was significant?
Just FYI, if you want a productive discussion you should hold back on accusing your opponents of fallacies. Ironically, since I never claimed that you claimed Eliezer engages in habitual banning on LW, your accusation that I made a strawman argument is itself a strawman argument.
Anyway, we're not getting anywhere, so let's disengage.
What utility do you think is gained by discussing the basilisk?
An interesting discussion that leads to better understanding of decision theories? Like, the same utility as is gained by any other discussion on LW, pretty much.
Strawman. This forum is not a place where things get habitually banned.
Sure, but you're the one that was going on about the importance of the mindset and culture; since you brought it up in the context of banning discussion, it sounded like you were saying that such censorship was part of a mindset/culture that you approve of.
I agree that resolving paradoxes is an important intellectual exercise, and that I wouldn't be satisfied with simply ignoring an ontological argument (I'd want to find the flaw). But the best way to find such flaws is to discuss the ideas with others. At no point should one assign such a high probability to ideas like Roko's basilisk being actually sound that one refuses to discuss them with others.
It seems unlikely that they would, if their gun is some philosophical decision theory stuff about blackmail from their future. I don't expect that gun to ever fire, no matter how many times you click the trigger.
Probably not a quick fix, but I would definitely say Eliezer gives significant chances (say, 10%) to there being some viable version of the Basilisk, which is why he actively avoids thinking about it.
If Eliezer was just angry at Roko, he would have yelled or banned Roko; instead, he banned all discussion of the subject. That doesn't even make sense as a "slashing out" reaction against Roko.
Somehow, blackmail from the future seems less plausible to me than every single one of your examples. Not sure why exactly.
If you are a programmer and think your code is safe because you see no way things could go wrong, it's still not good to believe that it isn't plausible that there's a security hole in your code.
Let's go with this analogy. The good thing to do is ask a variety of experts for safety evaluations, run the code through a wide variety of tests, etc. The think NOT to do is keep the code a secret while looking for mistakes all by yourself. If you keep your code out of the public domain, it is more likely to have security issues, since it was not scrutinized by the public. Banning discussion is almost never correct, and it's certainly not a good habit.
1Ideas that aren't proven to be impossible are possible. They don't have to be plausible.
That even seems to be false in Eliezer's case, and Eliezer definitely isn't 'LessWrong'.
It seems we disagree on this factual issue. Eliezer does think there is a risk of acausal blackmail, or else he wouldn't have banned discussion of it.
I'm not the person you replied to, but I mostly agree with (a) and reject (b). There's no way you can could possibly know enough about a not-yet-existing entity to understand any of its motivations; the entities that you're thinking about and the entities that will exist in the future are not even close to the same. I outlined some more thoughts here.
I think saying "Roko's arguments [...] weren't generally accepted by other Less Wrong users" is not giving the whole story. Yes, it is true that essentially nobody accepts Roko's arguments exactly as presented. But a lot of LW users at least thought something along these lines was plausible. Eliezer thought it was so plausible that he banned discussion of it (instead of saying "obviously, information hazards cannot exist in real life, so there is no danger discussing them").
In other words, while it is true that LWers didn't believe Roko...
my variation: choose the next candidate after 1/e trials that is better than 90% of existing trials. Why?: if you have a low number of candidates: worked solution - 10 candidates. you should (according to the secretary problem) interview 4 candidates, then select the next one that is better than the ones before.
Why n/e, and not some other number? Why 90%, and not some other amount? Come to think of it, shouldn't the value of the candidates matter, and not just the rank? For example, if I know my candidates' utility is sampled from [-1000,1000] and the f...
By changing the strategy from "first candidate better than the ones seen in the first n/e" to anything else, you lose all the rigorous mathematical backing that made the secretary problem cool in the first place. Is your solution optimal? Near-optimal? Who knows; it depends on your utility function and the distribution of candidates, and probably involves ugly integrals with no closed-form solution.
The whole point of the secretary problem is that a very precise way of stating the problem has a cool mathematical answer (the n/e strategy). But this precise statement of the problem is almost always useless in practice, so there's very little insight gained.
The secretary problem is way overused, and very rarely has any application in practice. This is because it maximizes the probability of finding the best match, and NOT the expectation over the utility of the match your get. This is almost never what you want in practice; in practice, you don't care much between a match with utility 1000 and a match with utility 999, you just want to avoid a match with utility -1000.
You might be interested in Aaronson's proposed theory for why it might be physically impossible to copy a human brain. He outlined it in "The Ghost in the Quantum Turing Machine": http://arxiv.org/abs/1306.0159
In that essay he discusses a falsifiable theory of the brain that, if true, would mean brain states are un-copyable. So Yudkowsky's counter-argument may be a little too strong: it is indeed consistent with modern physics for brain simulation to be impossible.
That might be Eliezer's stated objection. I highly doubt it's his real one (which seems to be something like "not releasing the logs makes me seem like a mysterious magician, which is awesome"). After all, if the goal was to make the AI-box escape seem plausible to someone like me, then releasing the logs - as in this post - helps much more than saying "nya nya, I won't tell you".
Okay, let's suppose for a second that I buy that teaching students to be goal oriented helps them significantly. That still leaves quite a few questions:
Many school boards already try to teach students to be goal oriented. Certainly "list out realistic goals" was said to me countless times in my own schooling. What do you plan to do differently?
There seems to be no evidence at all that LW material is better for life outcomes than any other self-help program, and some evidence that it's worse. Consider this post (again by Scott): http://lesswrong.com/lw/9p/extreme_rationality_its_not_that_great/
There's also quite a bit of reason to be skeptical of that evidence. Here's slatestarcodex's take: http://slatestarcodex.com/2015/03/11/too-good-to-be-true/
Do you have any evidence that LW materials help people refine and achieve their goals?
Helping people refine and achieve their goals is pretty damn difficult: school boards, psychiatrists, and welfare programs have been trying to do this for decades. For example, are you saying that teaching LW material in schools will improve student outcomes? I would bet very strongly against such a prediction.
Modern SGD mechanisms are powerful global optimizers.
They are heuristic optimizers that have no guarantees of finding a global optimum. It's strange to call them "powerful global optimizers".
Solomonoff induction is completely worthless - intractable - so you absolutely don't want to do that anyway.
I believe that was my point.
My goal is convincing people to have more clear and rational, evidence-thinking, as informed by LW materials.
Is there an objective measure by which LW materials inform more "clear and rational" thought? Can you define "clear and rational"? Or actually, to use LW terminology, can you taboo "clear" and "rational" and restate your point?
Regardless, as Brian Tomasik points out, helping people be more rational contributes to improving the world, and thus the ultimate goal of the EA movement.
But does it contribute to improving the world in an effective way?
If by "spreading rationality" you mean spreading LW material and ideas, then a potential problem is that it causes many people to donate their money to AI friendliness research instead of to malaria nets. Although these people consider this to be "effective altruism", as an AI skeptic it's not clear to me that this is significantly more effective than, say, donating money to cancer research (as non-EA people often do).
You cannot cherry pick a single year (a pretty non-representative year given the recession) in which the growth of a few sub-Saharan African countries was faster than the average growth of the stock market.
I didn't cherry-pick anything; that was the first google image result, so it's the one I linked to. I didn't think it's any different from a typical year. Is it? If so, what was special that year? If you're concerned that the US was in a recession, you can simply compare sub-Saharan Africa to the typical 6-7% stock market returns instead of comparing...
The rate of return on the stock market is around 10%
You didn't adjust for inflation; it's actually around 6 or 7%.
This is much faster than the rate of growth of sub-Saharan economies.
Depends on the country:
...Actually foreign aid might have a negative rate of return since most of the transfers are consumed rather than reinvested. Which isn't a problem per say - eventually you have to convert capital into QALYs even if that means you stop growing
The first is: more wasteful economically. This seems pretty robust, investments in sub-Saharan Africa have historically generated much less wealth than investments in other countries. Moreover wealth continues to grow via reinvestment.
It's not clear what you mean by this. Do you mean investments in Africa have generated less wealth for the investor? That might be true, but it doesn't mean they have generated less wealth overall. How would you measure this?
...he second is: more wasteful ethically. This is harder to defend, but I think it is a reasonable c
Yeah, you're probably right. I was probably just biased because the timeline is my main source of disagreement with AI danger folks.
I think point 1 is very misleading, because while most people agree with it, hypothetically a person might assign 99% chance of humanity blowing itself up before strong AI, and < 1% chance of strong AI before the year 3000. Surely even Scott Alexander will agree that this person may not want to worry about AI right now (unless we get into Pascal's mugging arguments).
I think most of the strong AI debate comes from people believing in different timelines for it. People who think strong AI is not a problem think we are very far from it (at least conceptually, but probably also in terms of time). People who worry about AI are usually pretty confident that strong AI will happen this century.
...I suppose the Bayesian answer to that is that a probability distribution is a description of one's knowledge, and that in principle, every state of knowledge, including total ignorance, can be represented as a prior distribution. In practice, one may not know how to do that. Fundamentalist Bayesians say that that is a weakness in our knowledge, while everyone else, from weak Bayesians to Sunday Bayesians, crypto-frequentists, and ardent frequentists, say it's a weakness of Bayesian reasoning. Not being a statistician, I don't need to take a view, although
I read all of your replies. What are you referring to? Also, this is uncharitable/insulting.
To be honest, I'm not sure what we're even disagreeing about. Like, sure, some genies are unsafe no matter how you phrase your wish. For other genies, you can just wish for ... (read more)