= 27d567bc2d48d9f40e9ccc20ea370b15)
Dmytry comments on Scenario analysis: semi-general AIs - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (66)
Are you truly reflecting on yourself though? I understand that the thing you're reflecting at, does map to same point, as the thing that i would describe as you, and so you are 'reflecting on yourself', but it is just a general feature of map vs territory problem where multiple things compress to a single map point.
One could say that the thing you are reflecting at, is some portion of yourself; that would perhaps be fair as 'limited reflection' goes. But then, it is not hard to code some example that looks at a small portion of itself.
We're certainly better at reflecting at some parts of our self than others. The ironic thing is, though, that when we look more closely and analyze just what it is that we are not reflecting on very well that we open up the can of worms that we had previously been avoiding.
It occurs to me that it may be a good thing that we are limited in this regard and are not yet able to reflect well enough to reproduce our intelligence in the form of a self reflective AI. If we could we'd probably have gone extinct already.
(I've been trying to avoid the uFAI=extinction thing lately, as there are various reasons that might not be the case. If we build uFAI we'll probably lose in some sense, maybe even to the extent that the common man will be able to notice we lost, but putting emphasis on the extinction scenario might be inaccurate. Killing all the humans doesn't benefit the AI much in most scenarios and can easily incur huge costs, both causally and acausally. Do you disagree that it's worth avoiding conflating losing and extinction?)
I agree that it's worth avoiding the conflation. There are losing scenarios that don't involve extinction - most highly concentrated in the realm of not-quite-friendly-AI. (And, naturally, excluding from the class under consideration all "uFAIs" that are insufficiently capable or insufficiently motivated to do anything much at all.)
For the purpose of this utterance it happens that I would make both the claims:
I expect I have different predictions regarding likely uFAI results and in particular different expectations regarding how acausal costs will be calculated.
FWIW acausal considerations don't figure as much into my calculations as straightforward "wow I hope there isn't an AI already chilling 'round these parts who will get pissed at me if I try to kill all the humans" considerations do.
It really seems to me that you'd have to be very, very confident that there were no gods around to punish you for you to think it was worth it to turn the humans into computronium. Like, there's an entire sun just sitting there for you to pluck, assuming Amon-Ra isn't already chilling in the center of it. I guess if the uFAI splintered a lot for whatever reason into AIs of differing power, and the AIs didn't cooperate, then you might end up with humans caught in the crossfire...?
You just Pascal-Mugged future superintelligences into letting humans live. I've got to admit. That's kind of baddass.
I just learned that von Neumann got... um, "wagered". Considering von Neumann was clearly a transhuman this establishes a lower bound on how smart you can be and still accept Pascal's wager. (Though I somewhat suspect that von Neumann's true reasons for returning to Catholicism late in life are more complicated than that.)
Gary Drescher once reminded me that von Neumann may have already suffered neurological damage from metastatic cancer at that point, so this lower bound may not be as high as you think (unless that's what you were alluding to by "true reasons").
Von Neumann had been a practicing Catholic earlier in life, so it's not that strange that he would return to Catholicism near the end. By "true reasons" I didn't mean brain damage, though thanks for bringing up that possibility. He was still transhumanly intelligent up until the end, but maybe not quite as transhumanly intelligent. But I guess I just meant that Pascal's wager surely wasn't his only consideration.
To expand on that, the difference between the let humans live pascal's wager, and the ordinary pascal's wager, is that letting humans live is the status quo.
Consider the 'don't eat me, my daddy is a cop' argument. It is hardly a pascal's wager. It is a rational thing to consider, especially when there's no shortage of food. It is more plausible that the status quo is product of actions of ultra powerful being, than 'you must give all money you got to me or the god will be pissed off' .
It is quite fascinating how the belief in God does oscillate between intelligence (rationality?) levels. Chimpanzees are naturally atheistic. Average humans are religious. Above average humans are usually atheistic. High IQ individuals like Eliezer Yudkowsky tend to be agnostic, in the sense that they assign a nonzero probability to the existence of God and believe in the existence of natural or artificial Gods. And people on the verge of posthumanism like John von Neumann, of whom was said that "only he was fully awake", are again leaning towards theism. I wonder if a truly posthuman AI would oscillate back to atheism while conjecturing that Omega is probably a theist.
Well, Omega would have to have a rather strange mind design not to believe in itself.
Like AIXI.
I've never even seen a shred of evidence suggesting I should believe there's a deity prepared to punish you if you kill people or do something immoral (or anything else for that matter), so it's on the exact same level as worrying about the possibility that snapping your fingers more than 300 times throughout the course of your lifetime may lead to an afterlife of eternal torture.
There's absolutely no reason why one should consider it more likely that there's a deity waiting to judge you after your death than really anything else at all. Maybe playing disc golf even once is bound to lead to an afterlife of hell. Ever played? If not, you may still have a chance!
If you consider it from a sound epistemic point of view, it becomes obvious that in our current state of knowledge, Pascal's Wager is equally apt to everything, and thus utterly useless or even simply meaningless. It's only a particular vulnerability in human brain hardware that makes it seem any different.
A lot of people still believe the whole god hypothesis, and many more have believed it over the course of human history, but that doesn't mean it's any more useful than an equally ridiculous hypothesis that I could invent right now: Learning to type with proper mechanics may help you in the short term (your mortal life), but beware, for God may not approve, and He doesn't mess around with his revenge.
We as humans are designed to think in groups, and it takes a special sort of introspection to pull yourself out of the mess and realize that the system is broken, and that a large percentage of the output is not to be trusted. That much you should know by virtue of having spent more than an hour reading Less Wrong, but perhaps you don't see this particular application.
I understand the pull of Pascal's Wager. I've felt it, and I still do. It feels like there's some special evidence for God's existence, and like we should privilege this hypothesis over another, but that's only because of how many people have believed in religion in the past, or rather how many times you've heard it espoused in an approving way. We're simply wired that way.
AI is thinking more straight than you. It doesn't need to believe in God. It can assign some uncertainty. The only good argument for non-existence of God is Occam's razor, and Occam's razor doesn't say the complex explanation is impossible, merely unlikely (and shouldn't be privileged).
Imagine the AI that knows precisely how much more complex are the rules that lead to universe full of Gods who create sims of our universe, than rules for our universe. This AI will tell you exactly how likely it is that it is sitting in a simulation, instead of making some hacks for the pascal's wager. This AI will have atheist predictor with weight p and theist predictor with weight 1-p . The theist predictor likes status quo. The theist predictor, however, won't necessarily respond to claims "you must give me all your money or else God will punish you", because it has no evidence towards this statement about God over the converse "you must not give me any of your money or else God will punish you".
What's the point of speculating about what something literally defined as having more knowledge than us would believe? All I know is that in our current state of knowledge, there's no more evidence for the existence of a God prepared to punish you with an afterlife in hell if you murder somebody than one who would do the same if you play disc golf.
I certainly don't think I have an argument for the non-existence of God, nor am I looking for one. Try disproving the idea that having a lifetime average of chewing on your left side between 60-70% will leave you screwed in the afterlife. Of course you can't disprove it, but then again I don't think I need to make this point on LW.
Why exactly would one have to be "very, very confident that there were no gods around to punish you for you to think it was worth it to turn the humans into computronium" (the original quote from Will)? Since there's no evidence for this hypothesis, you might as well spend your time worrying about making left turns, or chewing on the wrong side of your mouth, or really anything at all.
Precisely. No point. But people been speculating a lot about how it would behave, talking themselves into certainty that it would eat us. Those people need speculative antidote. If you speculate about one thing too much, but not about anything else, you start taking speculation as weak evidence, and deluding yourself.
edit: Also, try eating random unknown chemicals if you truly believe you should not worry about unknowns. One absolutely SHOULD worry about changing the status quo.
It may be because I haven't slept in 30 hours, but I'm having a hard time interpreting your writing. I've seen you make some important insights elsewhere, and I occasionally see exactly what you're saying, but my general impression of you is that you're not very good at judging your audience and properly managing the inferential distance.
You seem to agree with me to some extent in this discussion, or at least we don't seem to have a crucial disagreement, and this topic doesn't seem very important anyway, so I'm not necessarily asking you to explain yourself if that would take a long time, but perhaps this can serve as some constructive criticism thrown at you in a dark corner of a random thread.
As a meta question, would this sort of reply do better as a PM? What are the social considerations (signaling etc) with this sort of response? I don't know where to even start in that train of thought.
To respond to the edit, I simply don't see the analogy.
Your wording makes it sound analogous because you could describe what I'm saying as "don't worry about unknowns" (i.e., you have no evidence for whether God exists or not, so don't worry about it), and you could also describe your reductio the same way (i.e., you have no evidence for whether some random chemical is safe, so don't worry about it), but when I try to visualize the situation I don't see the connection.
A better analogy would be being forced to take one of five different medications, and having absolutely no evidence at all for their safety, or any hope of getting such evidence, and knowing that the only possible unsafe side effects would come only far in the future (if at all). In such a situation, you would of course forget about choosing based on safety, and simply choose based on other practical considerations such as price, how easy they are to get down, etc.
One should worry about changing the status quo only if there was a useful, reliable market test in place beforehand that had anything to do with why that status quo was like it was, and especially in the case that you don't have overwhelming evidence that (1) it was a known hardware or software vulnerability that led to what became the status quo, and (2) it's obvious that remaining a part of that status quo is extremely epistemically hazardous (being religious is certainly an epistemic hazard--feel free to ask for elaboration).
You do realize that "gods" means "other AGIs", right?
Yes, or rather I realize that in the sense that I do remember seeing you write that somewhere, but I'm not sure whether I had it sufficiently in mind during my replies. If you see anything suggesting that I didn't have it in mind such that it invalidated what I said as irrelevant to your position, let me know.
I should mention though that it may be epistemically unsanitary to use the term "god" (or "God)" when you really mean AIs, considering how long and winding the history of such theistic terminology has been. If your goal is clear communication, I would suggest switching to a term with less baggage.
Even though I did know that's what you meant in the sense that I saw you define it earlier, I might easily have fallen into pattern-matching and ended up largely criticizing a position irrelevant to yours.
Your goal seems to be to identify as a theist though, so using the term "God" (and the other standard theistic terminology) may be necessary for that purpose, in which case you may either (1) want to make sure to take extra care to compensate for the historical baggage and ambiguity, or (2) simply forget you ever read this comment.
Other peoples' beliefs are evidence. Many people believe in God. No one believes that disc golf causes eternal torture. The two hypotheses should not be assigned equal probability.
So you do not believe that others' beliefs are evidence?
It's sometimes (or even very often) evidence, but not when (1) there's not even a shred of evidence elsewhere, and (2) there's a convincing, systematic explanation for how a particular cluster of epistemic vulnerabilities in human brain hardware led to its widespread adoption.
In other words, a large portion of society believing something is evidence only if the memetic market test for the adoption of the idea at hand is intact. But our hardware and factory settings are so ridiculously mal-adapted to the epistemic environment of the modern world that this market test is extremely often utterly broken and useless.
If you want to make use of the societal thoughts on an issue, you must first appraise the health of the market test for the adoption of the ideas. Is it likely that competition in this area of the memetic environment will lead to ever more sound beliefs, or is there a wrench in the system that is bound to lead to a systematic spiral to ever more ridiculous or counterproductive dogmas?
Our hardware is just so riddled with epistemic problems that it would be a huge mistake to consider societal conclusions at face value. If the market test for meme propagation were intact, and the trial-and-error system for weeding out less useful beliefs in favor of more useful ones ran smoothly, large-scale acceptance of a position would of course be plenty of evidence--no further questions asked.
But we live in a different world--one where this trial-and-error system is in utter disrepair in an absolutely staggering number of cases. In such a world, one must always start with the question, "Is the memetic market test intact in this case, or must I go this epistemic journey myself?"
Of course the market test is better or worse from one place to the next, and I hang out here because the Less Wrong community certainly has one of the best belief propagation systems out there. If everybody on here seems to believe something with a lot of conviction, that to me is strong evidence.
In case the point was lost in the length, I should state it concisely. Whether the beliefs of others are evidence is a contextual question. It depends what the market test is, specifically for the propagation of the belief under scrutiny. If there's reason to believe that the market test is corrupted because of a particular hardware or software vulnerability, then there's reason to dismiss the widespread acceptance, and declare it no evidence at all.
If you accept all that, this of course brings us to the all-important question of why I think the memetic market test for the propagation of religion is broken enough to explain such widespread adoption despite how epistemically insane I consider it, but I don't think I need to (try to) answer that. You've probably heard it all before on here, in writing on memetics, from Dawkins, etc.
I more or less accept your reasoning as far as it goes, but:
If this is true, then why have so much confidence in your own personal appraisal of who to trust and who to write off as deluded? It is of course true that nearly everyone believes what they do for non-truth-tracking reasons, but "nearly everyone" isn't everyone, and there are many people, both theist and atheist, who believe what they do even despite strong memetic pressures to the contrary. Take me, for example; my theism doesn't win me any points with anyone, at least not as many points as it loses. And there are many theists like me. Knowing what you know about how easily humans fall into delusion, how can you be so confident that it's the other side that is deluded, and not your own? To return to the point, can you really be confident enough to disregard Pascal's wager? If so, how did so many at-least-nominally-truth-seeking people, from Plato to Pascal to Kant to me, end up disagreeing with you? How did we fall into such an obvious error?
Based on the comments of yours I've read, I think the only way you can call yourself a theist is by redefining most theistic terminology. Tell me if I'm wrong, but I don't think you agree with the object-level claims made by an average theist, as that theist would understand them. I'm not sure what you should call yourself...
We seem to be getting into some potentially very important territory, and I would certainly like to continue this discussion, but I'm running out of time for now and may be busy for up to 24 hours.
Before I go though, I should say at least one thing. It's certainly not an obvious error, and I could well be the one who's wrong. The discussions about rationality on Less Wrong are extremely useful for a basic reason: it's an extremely difficult and intricate epistemic journey to compensate for our mal-adapted hardware and software, and LW does it better than any other place at the moment (as far as I can see).
So yeah, your questions are certainly important, and they perhaps get to the essence of the issue. I look forward to trying to answer those questions, and seeing where it leads us in the discussion (assuming you think this is useful too). Feel free to write anything else in the meantime, or not.
A belief can be evidence for its stipulated meaning (this holds often), but could also be counterevidence, or irrelevant. What is a belief evidence for? Not at all automatically its stipulated meaning.
Yes, that's the response I was fishing for, so I could spring my trap. But the bait wasn't meant for you, so now I have to make pointless commentary. Alas.
Would it be possible for you to go ahead and spring your trap anyway? I'm very curious what you had in mind.
In the context of the original post - suppose that SGAI is logging some of the internal state into a log file, and then gains access to reading this log file, and reasons about it in same way as it reasons about the world - noticing correlation between it's feelings and state with the log file. Wouldn't that be the kind of reflection that we have? Is SGAI even logically possible without hard-coding some blind spot inside the AI about itself?
Or maybe we're going to go extinct real soon now, because we lack ability to reflect like this, and consequently didn't have couple thousands years to develop effective theory of mind for FAI before we make the hardware.
Having the ability to design and understand AI for a couple of thousand of years but somehow the inability to actually implement it sounds just about perfect. If only!
That is one idea for hacking friendliness: "Become the AI we would make if there were no existential threats and we didn't have the hardware to implement it for a few thousand years, and flaming letters appeared on the moon saying 'thou shall focus on designing Frienly AI' "
Havn't bothered typing it out before because it falls in the reference class of trying to cheat on FAI, wich is always a bad idea, but it seemed relevant here.
Well, it wouldn't be AI, it'd be simply I, as in "I think therefore I am." but not stopping at this period.
edit: I mean, look at the SIAI; what do exactly they do right now which they couldn't do in ancient Greece? If we could reflect on our mind better, and if our mind is physical in nature, then the idea of thinking machine would've been readily apparent, yet the microchips would still require very, very long time.
By this logic we'd have discovered all there is to know about math (including computer science) by Roman times at the latest.
Would anyone downvoting me care to explain why they disagree? Look at Newton or Turing: what exactly did they do which couldn't have been done in ancient Greece, and why is there no analogous counterexample for the SIAI?
Isn't "why weren't the Greeks working on Calculus" a far less silly question than "why weren't the ancient Greeks working on AI"?