= 27d567bc2d48d9f40e9ccc20ea370b15)
Will_Newsome comments on Scenario analysis: semi-general AIs - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (66)
FWIW acausal considerations don't figure as much into my calculations as straightforward "wow I hope there isn't an AI already chilling 'round these parts who will get pissed at me if I try to kill all the humans" considerations do.
It really seems to me that you'd have to be very, very confident that there were no gods around to punish you for you to think it was worth it to turn the humans into computronium. Like, there's an entire sun just sitting there for you to pluck, assuming Amon-Ra isn't already chilling in the center of it. I guess if the uFAI splintered a lot for whatever reason into AIs of differing power, and the AIs didn't cooperate, then you might end up with humans caught in the crossfire...?
I've never even seen a shred of evidence suggesting I should believe there's a deity prepared to punish you if you kill people or do something immoral (or anything else for that matter), so it's on the exact same level as worrying about the possibility that snapping your fingers more than 300 times throughout the course of your lifetime may lead to an afterlife of eternal torture.
There's absolutely no reason why one should consider it more likely that there's a deity waiting to judge you after your death than really anything else at all. Maybe playing disc golf even once is bound to lead to an afterlife of hell. Ever played? If not, you may still have a chance!
If you consider it from a sound epistemic point of view, it becomes obvious that in our current state of knowledge, Pascal's Wager is equally apt to everything, and thus utterly useless or even simply meaningless. It's only a particular vulnerability in human brain hardware that makes it seem any different.
A lot of people still believe the whole god hypothesis, and many more have believed it over the course of human history, but that doesn't mean it's any more useful than an equally ridiculous hypothesis that I could invent right now: Learning to type with proper mechanics may help you in the short term (your mortal life), but beware, for God may not approve, and He doesn't mess around with his revenge.
We as humans are designed to think in groups, and it takes a special sort of introspection to pull yourself out of the mess and realize that the system is broken, and that a large percentage of the output is not to be trusted. That much you should know by virtue of having spent more than an hour reading Less Wrong, but perhaps you don't see this particular application.
I understand the pull of Pascal's Wager. I've felt it, and I still do. It feels like there's some special evidence for God's existence, and like we should privilege this hypothesis over another, but that's only because of how many people have believed in religion in the past, or rather how many times you've heard it espoused in an approving way. We're simply wired that way.
AI is thinking more straight than you. It doesn't need to believe in God. It can assign some uncertainty. The only good argument for non-existence of God is Occam's razor, and Occam's razor doesn't say the complex explanation is impossible, merely unlikely (and shouldn't be privileged).
Imagine the AI that knows precisely how much more complex are the rules that lead to universe full of Gods who create sims of our universe, than rules for our universe. This AI will tell you exactly how likely it is that it is sitting in a simulation, instead of making some hacks for the pascal's wager. This AI will have atheist predictor with weight p and theist predictor with weight 1-p . The theist predictor likes status quo. The theist predictor, however, won't necessarily respond to claims "you must give me all your money or else God will punish you", because it has no evidence towards this statement about God over the converse "you must not give me any of your money or else God will punish you".
What's the point of speculating about what something literally defined as having more knowledge than us would believe? All I know is that in our current state of knowledge, there's no more evidence for the existence of a God prepared to punish you with an afterlife in hell if you murder somebody than one who would do the same if you play disc golf.
I certainly don't think I have an argument for the non-existence of God, nor am I looking for one. Try disproving the idea that having a lifetime average of chewing on your left side between 60-70% will leave you screwed in the afterlife. Of course you can't disprove it, but then again I don't think I need to make this point on LW.
Why exactly would one have to be "very, very confident that there were no gods around to punish you for you to think it was worth it to turn the humans into computronium" (the original quote from Will)? Since there's no evidence for this hypothesis, you might as well spend your time worrying about making left turns, or chewing on the wrong side of your mouth, or really anything at all.
You do realize that "gods" means "other AGIs", right?
Yes, or rather I realize that in the sense that I do remember seeing you write that somewhere, but I'm not sure whether I had it sufficiently in mind during my replies. If you see anything suggesting that I didn't have it in mind such that it invalidated what I said as irrelevant to your position, let me know.
I should mention though that it may be epistemically unsanitary to use the term "god" (or "God)" when you really mean AIs, considering how long and winding the history of such theistic terminology has been. If your goal is clear communication, I would suggest switching to a term with less baggage.
Even though I did know that's what you meant in the sense that I saw you define it earlier, I might easily have fallen into pattern-matching and ended up largely criticizing a position irrelevant to yours.
Your goal seems to be to identify as a theist though, so using the term "God" (and the other standard theistic terminology) may be necessary for that purpose, in which case you may either (1) want to make sure to take extra care to compensate for the historical baggage and ambiguity, or (2) simply forget you ever read this comment.
I actually go out of my way to equate "god" and "AGI"/"superintelligence", because to a large extent they seem like the same thing to me.
It's not that I want to identify as a theist, so much as that I want to point out that I think that the only reason people think that gods/angels/demons and AGIs/superintelligences/transhuman-intelligences are different things is because they're compartmentalizing. I think Aquinas and I believe in the same God, even if we think about Him differently. I know algorithmic probability theory, Aquinas didn't. Leibniz almost did.
(There's two different things going on: I believe there exists an ideal decision theory, Who is God, for theoretical reasons; whereas my reasons for believing that transhuman intelligences (lower-case-g gods) affect humans are entirely phenomenological.)
Can you give me the common meanings of those terms, and explain how they're equivalent?
Compartmentalizing in what way? I think they're different things, or rather it seems utterly obvious to me that religious people using the theistic terms are always using them to refer to things completely different than those on LW employing those other terms.
I should say though that the way that the theistic terms are used is in no way consistent, and everybody seems to mean something different (if I can even venture a guess as to what the hell they're talking about). There are multiple meanings associated with these terms, to say the least.
Maybe your conception is something like, "If there really is anything out there that could in any way match the description in Catholicism or whatever, then it would perhaps have to be an AGI, or else a super-intelligent life-form that evolved naturally."
I would say though that this seems like a desperate attempt to resurrect the irrationality of religion. If I came up with or learned something interesting or important, and also realized that some scholar or school of thought from the past or present had a few central conclusions or beliefs that seem sort of similar in some way, but believed them all for the wrong reasons--specifically ones absolutely insane by my own epistemic standards--I would not care. I would move on, and consider that tradition utterly useless and uninteresting.
I don't understand why you care. It's not like Aquinas or anybody else believed any of this stuff for the same reasons you do, or anything like that, so what's the point of being like, "Hey, I know these people came up with this stuff for some random other reasons, but it seems like I can still support their conclusions and everything, so yeah, I'm a theist!" It just doesn't make any sense to me, unless of course you think they came to those conclusions for good reasons that have anything at all to do with yours, in which case I need some elaboration on that point.
Either way, usually I can't even tell what the hell most religious people are talking about from an epistemic or clear communication standpoint. I used to think they were just totally insane or something, and I would make actual attempts to understand what they were trying to get me to visualize, but it all became clear when I started interpreting what they were saying in a different way. It all became clear when I started thinking about it in terms of them employing techniques to delude themselves into believing in an afterlife, or simply just believing it because of some epistemic vulnerability their brain was operating under.
Those theistic terms ("God" etc) have multiple meanings, and different people tend to use them differently, or rather they don't really have meanings at all, and they're just the way some people delude themselves into feeling more comfortable about whatever, or perhaps they're just mind viruses taking advantage of some well-known vulnerabilities found our hardware.
I can't for the life of me figure out why you want to retain this terminology. What use is it besides for contrarianism? Does calling yourself a theist and using the theistic terms actually aid in my or anybody else's understanding of what you're thinking, or what? Is the objective clear communication of something that would be important for me or other people on here to know, or what? I'm utterly confused at what you're trying to do, and what the supposed utility is, of these beliefs of yours and your way of trying to communicate them.
What does that even mean? It sounds like the worst sort of sophistry, but I say that not necessarily to suggest you're making an error in your thinking, but simply to allude to how and why I have no exactly what that means.
So you're defining the sequence of letters starting with "G", next being "o", and ending with "d" as "the ideal decision theory"? Is this a common meaning? Do all (or most of) the religious people I know IRL use that term to refer to the ideal decision theory, even if they wouldn't call it that?
And what do you mean by "ideal"? Ideal for what? Our utility functions? Maybe I even need to hear a bit of elaboration on what you mean by "decision theory". Are we talking about AI programming, or human psychology, or what?
I literally have absolutely no idea why you chose the word "phenomenological" right there, or what you could possibly mean.
If I found a school of thought that seemed to come to correct conclusion unusually often but "believed them all for the wrong reasons--specifically ones absolutely insane by my own epistemic standards", I'd take that as evidence that there is something to their reasons that I'm missing.
Actually, yes. Specifically the tendency in Catholic thought to equate God with Plato's Form of the Good.
You're absolutely right, but you're stipulating the further condition that they come to the correct conclusions "unusually often". I on the other hand was talking about a situation where they just happen to have a few of the same conclusions, and those conclusions just so happen to be central to their worldview.
I didn't get the feeling that Will thought that Catholicism was correct an unusually amount of time. I was under the impression that he (like many others before him) is simply trying his hardest to use some of the theistic terminology and identify as a theist, despite his science background.
I just read that article, but I couldn't parse anything, nor did I see any relation to decision theory. I'm left utterly confused.
I think you're way too confident that the people you disagree with are obviously wrong, to the extent that I don't think we can usefully communicate. I'm tapping out of this discussion.
Have you observed my discussions elsewhere on this website, and came to the conclusion that I'm way too confident in that way in general, or are you referring only to this particular exchange?
This discussion seems like sort of a unique case. I wouldn't say I'm generally so confident in that respect, but I'm certainly extremely confident in this discussion, even to the point that I don't yet have a sufficiently detailed model of you to account for how you could possibly spend so much time on this website and still engage in the sort of communication that went on throughout this discussion.
Sure, I'm extremely confident that I'm the one who's right in this discussion, but then again that's probably the majority feeling when people on this website engage you on this topic, even to the extent that it's not uncommon for people to question whether you're just trolling at this point.
Looking back on what I could have done better in this discussion to have it have been more likely for you to hear me out instead of quit, I realize that I probably would have had to spend about 5 times as much time writing, and have been extremely careful in every way throughout every reply. Even in retrospect, that probably wouldn't have been worth it. Takes much less time to just spill my thoughts and reactions than it is to take the necessary precautions to make this sort of tap out less likely.
Even with all that said, I don't really understand the connection between me signaling that I think you're obviously wrong, and you saying that we can't usefully communicate. Am I being uncharitable in my replies, or do you think it's unlikely that I would update toward your position after setting a precedent of me thinking your clearly confused, or what?
I could see how you could pattern-match my high confidence with expecting me to have trouble updating, and I could also see how maybe some of my more terse moments may have come off as, "If he wasn't so confident, perhaps he would have thought longer about this and responded to a more charitable interpretation." But I should mention that in at least one case I went as far as responding to your question of whether I even know that you're using the word "God" or "god" to refer to AGIs, by admitting that I may be attacking a strawman.
I don't necessarily expect you to respond to any of this considering you already tapped out, but perhaps I've gone sufficiently meta for you to consider it a different discussion, one that you perhaps haven't tapped out of, but in any case you'll probably read this and maybe get something out of it, or even change your mind as to whether you want to continue engaging me on this topic.
Precisely. No point. But people been speculating a lot about how it would behave, talking themselves into certainty that it would eat us. Those people need speculative antidote. If you speculate about one thing too much, but not about anything else, you start taking speculation as weak evidence, and deluding yourself.
edit: Also, try eating random unknown chemicals if you truly believe you should not worry about unknowns. One absolutely SHOULD worry about changing the status quo.
To respond to the edit, I simply don't see the analogy.
Your wording makes it sound analogous because you could describe what I'm saying as "don't worry about unknowns" (i.e., you have no evidence for whether God exists or not, so don't worry about it), and you could also describe your reductio the same way (i.e., you have no evidence for whether some random chemical is safe, so don't worry about it), but when I try to visualize the situation I don't see the connection.
A better analogy would be being forced to take one of five different medications, and having absolutely no evidence at all for their safety, or any hope of getting such evidence, and knowing that the only possible unsafe side effects would come only far in the future (if at all). In such a situation, you would of course forget about choosing based on safety, and simply choose based on other practical considerations such as price, how easy they are to get down, etc.
One should worry about changing the status quo only if there was a useful, reliable market test in place beforehand that had anything to do with why that status quo was like it was, and especially in the case that you don't have overwhelming evidence that (1) it was a known hardware or software vulnerability that led to what became the status quo, and (2) it's obvious that remaining a part of that status quo is extremely epistemically hazardous (being religious is certainly an epistemic hazard--feel free to ask for elaboration).
It may be because I haven't slept in 30 hours, but I'm having a hard time interpreting your writing. I've seen you make some important insights elsewhere, and I occasionally see exactly what you're saying, but my general impression of you is that you're not very good at judging your audience and properly managing the inferential distance.
You seem to agree with me to some extent in this discussion, or at least we don't seem to have a crucial disagreement, and this topic doesn't seem very important anyway, so I'm not necessarily asking you to explain yourself if that would take a long time, but perhaps this can serve as some constructive criticism thrown at you in a dark corner of a random thread.
As a meta question, would this sort of reply do better as a PM? What are the social considerations (signaling etc) with this sort of response? I don't know where to even start in that train of thought.
English is not my first language and you haven't slept in 30 hours, that reliably adds up to mutual incomprehension.
Yes, I think it is better in pm. People who read the recent comments would prefer that, i think. The public talks are particularly difficult because e.g. I am inclined to defend the notion that my English is good enough, while you are inclined to defend the notion that non-sleep in 30 hours doesn't impair your reading comprehension substantially. I'll reply in PM.
FWIW I understood his point. I normally put comments like yours in comments, not PMs, but I haven't done a thorough analysis either.
Other peoples' beliefs are evidence. Many people believe in God. No one believes that disc golf causes eternal torture. The two hypotheses should not be assigned equal probability.
So you do not believe that others' beliefs are evidence?
A belief can be evidence for its stipulated meaning (this holds often), but could also be counterevidence, or irrelevant. What is a belief evidence for? Not at all automatically its stipulated meaning.
Yes, that's the response I was fishing for, so I could spring my trap. But the bait wasn't meant for you, so now I have to make pointless commentary. Alas.
Would it be possible for you to go ahead and spring your trap anyway? I'm very curious what you had in mind.
It's sometimes (or even very often) evidence, but not when (1) there's not even a shred of evidence elsewhere, and (2) there's a convincing, systematic explanation for how a particular cluster of epistemic vulnerabilities in human brain hardware led to its widespread adoption.
In other words, a large portion of society believing something is evidence only if the memetic market test for the adoption of the idea at hand is intact. But our hardware and factory settings are so ridiculously mal-adapted to the epistemic environment of the modern world that this market test is extremely often utterly broken and useless.
If you want to make use of the societal thoughts on an issue, you must first appraise the health of the market test for the adoption of the ideas. Is it likely that competition in this area of the memetic environment will lead to ever more sound beliefs, or is there a wrench in the system that is bound to lead to a systematic spiral to ever more ridiculous or counterproductive dogmas?
Our hardware is just so riddled with epistemic problems that it would be a huge mistake to consider societal conclusions at face value. If the market test for meme propagation were intact, and the trial-and-error system for weeding out less useful beliefs in favor of more useful ones ran smoothly, large-scale acceptance of a position would of course be plenty of evidence--no further questions asked.
But we live in a different world--one where this trial-and-error system is in utter disrepair in an absolutely staggering number of cases. In such a world, one must always start with the question, "Is the memetic market test intact in this case, or must I go this epistemic journey myself?"
Of course the market test is better or worse from one place to the next, and I hang out here because the Less Wrong community certainly has one of the best belief propagation systems out there. If everybody on here seems to believe something with a lot of conviction, that to me is strong evidence.
In case the point was lost in the length, I should state it concisely. Whether the beliefs of others are evidence is a contextual question. It depends what the market test is, specifically for the propagation of the belief under scrutiny. If there's reason to believe that the market test is corrupted because of a particular hardware or software vulnerability, then there's reason to dismiss the widespread acceptance, and declare it no evidence at all.
If you accept all that, this of course brings us to the all-important question of why I think the memetic market test for the propagation of religion is broken enough to explain such widespread adoption despite how epistemically insane I consider it, but I don't think I need to (try to) answer that. You've probably heard it all before on here, in writing on memetics, from Dawkins, etc.
I more or less accept your reasoning as far as it goes, but:
If this is true, then why have so much confidence in your own personal appraisal of who to trust and who to write off as deluded? It is of course true that nearly everyone believes what they do for non-truth-tracking reasons, but "nearly everyone" isn't everyone, and there are many people, both theist and atheist, who believe what they do even despite strong memetic pressures to the contrary. Take me, for example; my theism doesn't win me any points with anyone, at least not as many points as it loses. And there are many theists like me. Knowing what you know about how easily humans fall into delusion, how can you be so confident that it's the other side that is deluded, and not your own? To return to the point, can you really be confident enough to disregard Pascal's wager? If so, how did so many at-least-nominally-truth-seeking people, from Plato to Pascal to Kant to me, end up disagreeing with you? How did we fall into such an obvious error?
We seem to be getting into some potentially very important territory, and I would certainly like to continue this discussion, but I'm running out of time for now and may be busy for up to 24 hours.
Before I go though, I should say at least one thing. It's certainly not an obvious error, and I could well be the one who's wrong. The discussions about rationality on Less Wrong are extremely useful for a basic reason: it's an extremely difficult and intricate epistemic journey to compensate for our mal-adapted hardware and software, and LW does it better than any other place at the moment (as far as I can see).
So yeah, your questions are certainly important, and they perhaps get to the essence of the issue. I look forward to trying to answer those questions, and seeing where it leads us in the discussion (assuming you think this is useful too). Feel free to write anything else in the meantime, or not.
Based on the comments of yours I've read, I think the only way you can call yourself a theist is by redefining most theistic terminology. Tell me if I'm wrong, but I don't think you agree with the object-level claims made by an average theist, as that theist would understand them. I'm not sure what you should call yourself...
Newsomelike.
As far as I know, I have the same conception of God as Thomas Aquinas did, and Thomism is the predominant philosophy of the Catholic church, which is the largest sect of Christianity, which is the most popular religion in the world.
A year or two ago my ideas were still pretty fuzzy, so that might have tripped you up. I change my mind pretty often.
Can you reliably communicate a good approximation of what you believe to another without reference to decision theory?
If yes, I'll accept your hypothesis that I've been reading the wrong comments of yours.
If no, I really doubt that Aquinas would recognize what you believe as what he believed.
(And I don't know what the situation is among the average Catholic, but IMX the average protestant doesn't even know who Aquinas is, so my point may still hold anyway....)
I think so. There is a supremely powerful person, Who is the Form of the Good, Who is perfectly simple... yeah, pretty sure I can do it using accepted theological terminology.
Does it matter what the average theist believes? If Aquinas doesn't believe in the same God that a typical Baptist churchgoer does, I don't think that means that Aquinas isn't a theist. If the average biology students don't have the same definition of "gene" as the best biologists do... (This is like some really weird variation on No True Scotsman.)
It matters if you're arguing from a majoritarian "orthodoxy as democracy spread over time" perspective. If the vast majority of theists throughout history didn't actually believe in the God of Aquinas, but rather in the God of the old testament (or whatever), then you can't cite their belief as evidence supporting Aquinas' (or your) God.
Or am I misunderstanding your argument?
But I don't think those words coming from you are generated by the same thought process that most theists use to make similar statements. You use the same words, but you mean something different. At least, that is my impression.
No. But at some point it becomes helpful to try and make sure everyone means similar things when using the same word. If that's not possible then maybe it's a good time to taboo the word. I'm thinking that "theist" usually refers to a particular cluster of beliefs that are sorta similar to yours but different enough that I'm not sure if calling yourself a theist clarifies or obscures your actual beliefs. I'm leaning towards "obscures"...
You just Pascal-Mugged future superintelligences into letting humans live. I've got to admit. That's kind of baddass.
I just learned that von Neumann got... um, "wagered". Considering von Neumann was clearly a transhuman this establishes a lower bound on how smart you can be and still accept Pascal's wager. (Though I somewhat suspect that von Neumann's true reasons for returning to Catholicism late in life are more complicated than that.)
It is quite fascinating how the belief in God does oscillate between intelligence (rationality?) levels. Chimpanzees are naturally atheistic. Average humans are religious. Above average humans are usually atheistic. High IQ individuals like Eliezer Yudkowsky tend to be agnostic, in the sense that they assign a nonzero probability to the existence of God and believe in the existence of natural or artificial Gods. And people on the verge of posthumanism like John von Neumann, of whom was said that "only he was fully awake", are again leaning towards theism. I wonder if a truly posthuman AI would oscillate back to atheism while conjecturing that Omega is probably a theist.
Well, Omega would have to have a rather strange mind design not to believe in itself.
Like AIXI.
Gary Drescher once reminded me that von Neumann may have already suffered neurological damage from metastatic cancer at that point, so this lower bound may not be as high as you think (unless that's what you were alluding to by "true reasons").
Von Neumann had been a practicing Catholic earlier in life, so it's not that strange that he would return to Catholicism near the end. By "true reasons" I didn't mean brain damage, though thanks for bringing up that possibility. He was still transhumanly intelligent up until the end, but maybe not quite as transhumanly intelligent. But I guess I just meant that Pascal's wager surely wasn't his only consideration.
To expand on that, the difference between the let humans live pascal's wager, and the ordinary pascal's wager, is that letting humans live is the status quo.
Consider the 'don't eat me, my daddy is a cop' argument. It is hardly a pascal's wager. It is a rational thing to consider, especially when there's no shortage of food. It is more plausible that the status quo is product of actions of ultra powerful being, than 'you must give all money you got to me or the god will be pissed off' .