= 27d567bc2d48d9f40e9ccc20ea370b15)
Less Wrong Q&A with Eliezer Yudkowsky: Ask Your Questions
As promised, here is the "Q" part of the Less Wrong Video Q&A with Eliezer Yudkowsky.
The Rules
1) One question per comment (to allow voting to carry more information about people's preferences).
2) Try to be as clear and concise as possible. If your question can't be condensed to a few paragraphs, you should probably ask in a separate post. Make sure you have an actual question somewhere in there (you can bold it to make it easier to scan).
3) Eliezer hasn't been subpoenaed. He will simply ignore the questions he doesn't want to answer, even if they somehow received 3^^^3 votes.
4) If you reference certain things that are online in your question, provide a link.
5) This thread will be open to questions and votes for at least 7 days. After that, it is up to Eliezer to decide when the best time to film his answers will be. [Update: Today, November 18, marks the 7th day since this thread was posted. If you haven't already done so, now would be a good time to review the questions and vote for your favorites.]
Suggestions
Don't limit yourself to things that have been mentioned on OB/LW. I expect that this will be the majority of questions, but you shouldn't feel limited to these topics. I've always found that a wide variety of topics makes a Q&A more interesting. If you're uncertain, ask anyway and let the voting sort out the wheat from the chaff.
It's okay to attempt humor (but good luck, it's a tough crowd).
If a discussion breaks out about a question (f.ex. to ask for clarifications) and the original poster decides to modify the question, the top level comment should be updated with the modified question (make it easy to find your question, don't have the latest version buried in a long thread).
Update: Eliezer's video answers to 30 questions from this thread can be found here.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d){kind=logo}
Comments (682)
Could you (Well, "you" being Eliezer in this case, rather than the OP) elaborate a bit on your "infinite set atheism"? How do you feel about the set of natural numbers? What about its power set? What about that thing's power set, etc?
From the other direction, why aren't you an ultrafinitist?
Well obviously if a set is finite, no amount of taking its power set is going to change that fact.
I meant "the set of all natural numbers", IIRC, he's explicitly said he's not an ultrafinitist, so either he considers that as an acceptable infinite set, or he considers the natural numbers to exist, but not the set of them, or something.
I meant "if you accept countable infinities, where and how do you consider the whole cantor hierarchy to break down?"
What would it even mean for the natural numbers (the entire infinity of them) to "exist"?
What makes a set acceptable or not?
This question sounds weird to me.
I find it best not to speak about "existence", but speak instead of logical models that work. For example, we don't know if our concept of integers is consistent, but we have evolved a set of tools for reasoning about it that have been quite useful so far. Now we try to add new reasoning tools, new concepts, without breaking the system. For example, if we imagine "the set of all sets" and apply some common reasoning to it, we reach Russell's paradox; but we can't feed this paradox back into the integers to demonstrate their inconsistency, so we just throw the problematic concept away with no harm done.
<Wince>. Uh-oh; if he takes this up, I may finally have to write that post I promised back in June!
Earlier today, I pondered whether this infinite set atheism thing is something Eliezer merely claims to believe as some sort of test of basic rationality. It's a belief that, as far as I can tell, makes no prediction.
But here's what I predict that I would say if I had Eliezer's opinions and my mathematical knowledge: I'm a fan of thinking of ZFC as being its countably infinite model, in which the class of all sets is enumerable, and every set has a finite representation. Of course, things like the axiom of infinity and Cantor's diagonal argument still apply; it's just that "uncountably infinite set" simply means "set whose bijection with the natural numbers is not contained in the model".
(Yes, ZFC has a countable model, assuming it's consistent. I would call this weird, but I hate admitting that any aspect of math is counterintuitive.)
ZFC's countable model isn't that weird.
Imagine a computer programmer, watching a mathematician working at a blackboard. Imagine asking the computer programmer how many bytes it would take to represent the entities that the mathematician is manipulating, in a form that can support those manipulations.
The computer programmer will do a back of the envelope calculation, something like: "The set of all natural numbers" is 30 characters, and essentially all of the special symbols are already in Unicode and/or TeX, so probably hundreds, maybe thousands of bytes per blackboard, depending. That is, the computer programmer will answer "syntactically".
Of course, the mathematician might claim that the "entities" that they're manipulating are more than just the syntax, and are actually much bigger. That is, they might answer "semantically". Mathematicians are trained to see past the syntax to various mental images. They are trained to answer questions like "how big is it?" in terms of those mental images. A math professor asking "How big is it?" might accept answers like "it's a subset of the integers" or "It's a superset of the power set of reals". The programmer's answer of "maybe 30 bytes" seems, from the mathematical perspective, about as ridiculous as "It's about three feet long right now, but I can write it longer if you want".
The weirdly small models are only weirdly small if what you thought were manipulating was something other than finite (and therefore Godel-numberable) syntax.
Models are semantics. The whole point of models is to give semantic meaning to syntactic strings.
I haven't studied the proof of the Löwenheim–Skolem theorem, but I would be surprised if it were as trivial as the observation that there are only countably many sentences in ZFC. It's not at all clear to me that you can convert the language in which ZFC is expressed into a model for ZFC in a way that would establish the Löwenheim–Skolem theorem.
I have studied the proof of the (downward) Lowenheim-Skolem theorem - as an undergraduate, so you should take my conclusions with some salt - but my understanding of the (downward) Lowenheim-Skolem theorem was exactly that the proof builds a model out of the syntax of the first-order theory in question.
I'm not saying that the proof is trivial - what I'm saying is that holding Godel-numberability and the possibility of a strict formalist interpretation of mathematics in your mind provides a helpful intuition for the result.
I've said this before in many places, but I simply don't do that sort of thing. If I want to say something flawed just to see how my readers react to it, I put it into the mouth of a character in a fictional story; I don't say it in my own voice.
In 2000, you said this:
Would you stick by your assertion that in 2000, the republicans were "slightly better", and who or what did you mean they were slightly better for? From where I'm standing, albeit with the benefit of hindsight, it seems like eight years of al gore would've been "slightly better" than eight years of dick cheney for just about everyone and everything.
Cf. "Stop Voting for Nincompoops" (2008):
...and that's still my reply, but you could vote it up if you want me to repeat it on video. :)
I see. Were you similarly at a loss to distinguish the two parties in the 2004 and 2008 elections? It's one thing to say that you can't tell the difference and quite another to insist that nobody could have and we were all fools for voting. Maybe you just weren't paying attention.
Until Obama, I've never voted for anyone who won a presidential election. (I voted for Nader in 2000 and Kerry in 2004.)
No.
If you thought an AGI couldn't be built what would you dedicate your life to doing? Perhaps another formulation, or a related question: what is the most important problem/issue not directly related to AI.
That counterfactual seems like trouble. Do you mean literally impossible by the laws of physics (surely not)? Or highly improbable that humans will be able to build one? What counts as artificial intelligence - can we do human augmentation? What counts as "highly improbable" - can we really assume stupid or evil humans won't be able to build one eventually?
It seems to me that plugging all the holes and ways of building a general intelligence to spec would require messing with the laws of physics. We may want to specify a Cosmic Censor law.
Yeah, it is trouble. Thats why I offered the other formulation, thought that might be too vague. Basically, I just wanted to know what non-transhumanist Eliezer would be doing. I don't really care about the counterfactual some much as picking out a different topic area. Maybe the question should just be "If the idea of intelligence augmentation had never occurred to you and no one had ever shared it with you, what would you be doing with your life?"
At the Singularity Summit, this question (or one similar) was asked, and (if I remember correctly) EY answer was something like: If the world didn't need saving? Possibly writing science fiction.
Cool. But say the world does need saving, would there be a way to do it that doesn't involve putting something smarter than us in charge?
I have questions. You say we must have one question per comment. So, I will have to make multitple posts.
1) Is there a domain where rational analysis does not apply?
Improvisational theater. (I'm not Eliezer, I know.)
actually... http://greenlightwiki.com/improv/Status http://craigtovey.blogspot.com/2008/02/popular-comedy-formulas.html
learning this stuff allowed me (introvert) to successfully fake extroversion for my own benefit when I need to.
2) How does one affect the process of increasing the rationality of people who are not ostensibly interested in objective reasoning and people who claim to be interested but are in fact attached to their biases?
I find that question interesting because it is plain that the general capacity for rationality in a society can be improved over time. Once almost no one understood the concept of a bell curve or a standard deviation, but now the average person has a basic understanding of how these concepts apply to the real world.
It seems to me that we really are faced with the challenge of explaining the value of empirical analysis and objective reasoning to much of the world. Today the Middle East is hostile towards reason though they presumably don't have to be this way.
So again, my question is how do more rational people affect the reasoning capacity in less rational people, including those hostile towards rationality?
I suspect that, on the contrary, >50% of the population have very little idea what either term means.
How young can children start being trained as rationalists? And what would the core syllabus / training regimen look like?
Just out of curiosity, why are you asking this? And why is Yudkowsky's opinion on this matter relevant?
Could you please tell us a little about your brain? For example, what is your IQ, at what age did you learn calculus, do you use cognitive enhancing drugs or brain fitness programs, are you Neurotypical and why didn't you attend school?
During a panel discussion at the most recent Singularity Summit, Eliezer speculated that he might have ended up as a science fiction author, but then quickly added:
Shortly thereafter, Peter Thiel expressed a wish that all the people currently working on string theory would shift their attention to AI or aging; no disagreement was heard from anyone present.
I would therefore like to ask Eliezer whether he in fact believes that the only two legitimate occupations for an intelligent person in our current world are (1) working directly on Singularity-related issues, and (2) making as much money as possible on Wall Street in order to donate all but minimal living expenses to SIAI/Methuselah/whatever.
How much of existing art and science would he have been willing to sacrifice so that those who created it could instead have been working on Friendly AI? If it be replied that the work of, say, Newton or Darwin was essential in getting us to our current perspective wherein we have a hope of intelligently tackling this problem, might the same not hold true in yet unknown ways for string theorists? And what of Michelangelo, Beethoven, and indeed science fiction? Aren't we allowed to have similar fun today? For a living, even?
I don't know about Eliezer, but I would be able to sacrifice quite a lot; perhaps all of art. If humanity spreads through the galaxy there will be way more than enough time for all that.
It might. But their expected contribution would be much greater if they looked at the problem to see how they could contribute most effectively.
No one's saying that you're not allowed to do something. Just that it's suboptimal under their utility function, and perhaps yours.
My guess is that you overestimate how much of an altruist you are. Consider that lives can be saved using traditional methods for well under $1000. That means every time you spend $1000 on other things, your revealed preference is that having that stuff is more important to you than saving the life of another human being. If you're upset upon hearing this fact, then you're suffering from cognitive dissonance. If you're a true altruist, you'll be happy after hearing this fact, because you'll realize that you can be scoring much better on your utility function than you are currently. (Assuming for the moment that happiness corresponds with opportunities to better satisfy your utility function, which seems to be fairy common in humans.)
Regardless of whether you're a true altruist, it makes sense to spend a chunk of your time on entertainment and relaxation to spend the rest more effectively.
By the way, I would be interested to hear Eliezer address this topic in his video.
Somewhat related, AGI is such an enormously difficult topic, requiring intimate familiarity with so many different fields, that the vast majority of people (and I count myself among them) simply aren't able to contribute effectively to it.
I'd be interested to know if he thinks there are any singularity-related issues that are important to be worked on, but somewhat more accessible, that are more in need of contributions of man-hours rather than genius-level intellect. Is the only way a person of more modest talents can contribute through donations?
Do you vote?
There are a few things--voting, lotteries, the viability of picking up pennies off the ground--that draw way too much attention from rationalists. Not criticizing you here, I'm interested in them too!
Answered in this comment.
I don't think it is answered. The person who says they voted is not Eliezer, and his general comment about whom to vote for says nothing about whether he himself actually votes; just that he voted at one time in the past. (I can recommend that you wear a parachute while skydiving, but that doesn't mean I do or do not think skydiving is something worth doing & have done it myself & will do it again.)
Even if it were not, Eliezer followed up further down that comment tree suggesting that his choice was clear in the 2008 elections.
I figured "I was therefore obliged to vote for third parties wherever possible, to penalize the Republicrats for getting grabby" seemed like the kind of argument that doesn't apply to only a single election.
Perhaps I'm confused by "the person who says they voted is not Eliezer;" that quote, after all, was from this article he wrote.
I was referring to CronoDAS who said that they voted in '08. Eliezer didn't; 'whenever possible' is pretty slippery.
(Not that I think there's any point to asking how Eliezer votes, being that he lives in California, I thought. In a state like that, voting is a pure waste of time, unless someone has come up with a really ingenious version of Newcomb's Paradox showing you should vote even when you know your vote won't make a difference.)
Margins matter - politicians don't want to take a chance when they don't have to, and if it looks like some policy leads to weaker support or stronger support, they act accordingly.
I admit to being curious about various biographical matters. So for example I might ask:
What are your relations like with your parents and the rest of your family? Are you the only one to have given up religion?
Sticking with biography/family background:
Anyone who has read this poignant essay knows that Eliezer had a younger brother who died tragically young. If it is not too insensitive of me, may I ask what the cause of death was?
It's been discussed somewhere in the second half of this podcast:
http://www.speakupstudios.com/Listen.aspx?ShowUID=333035
What are the hazards associated with making random smart people who haven't heard about existential dangers more intelligent, mathematically inclined, and productive?
What is the background that you most frequently wish would-be FAI solvers had when they struck up conversations with you? You mentioned the Dreams of Friendliness series; is there anything else? You can answer this question in comment form if you like.
Who was the most interesting would-be FAI solver you encountered?
What was the most useful suggestion you got from a would-be FAI solver? (I'm putting separate questions in separate comments per MichaelGR's request.)
What's your advice for Less Wrong readers who want to help save the human race?
What criteria do you use to decide upon the class of algorithms / computations / chemicals / physical operations that you consider "conscious" in the sense of "having experiences" that matter morally? I assume it includes many non-human animals (including wild animals)? Might it include insects? Is it weighted by some correlate of brain / hardware size? Might it include digital computers? Lego Turing machines? China brains? Reinforcement-learning algorithms? Simple Python scripts that I could run on my desktop? Molecule movements in the wall behind John Searle's back that can be interpreted as running computations corresponding to conscious suffering? Rocks? How does it distinguish interpretations of numbers as signed vs. unsigned, or ones complement vs. twos complement? What physical details of the computations matter? Does it regard carbon differently from silicon?
That's 14 questions! ;-)
In one of the discussions surrounding the AI-box experiments, you said that you would be unwilling to use a hypothetical fully general argument/"mind hack" to cause people to support SIAI. You've also repeatedly said that the friendly AI problem is a "save the world" level issue. Can you explain the first statement in more depth? It seems to me that if anything really falls into "win by any means necessary" mode, saving the world is it.
This comes to mind:
— Protected From Myself
What was the significance of the wirehead problem in the development of your thinking?
Is there any published work in AI (whether or not directed towards Friendliness) that you consider does not immediately, fundamentally fail due to the various issues and fallacies you've written on over the course of LW? (E.g. meaningfully named Lisp symbols, hiddenly complex wishes, magical categories, anthropomorphism, etc.)
ETA: By AI I meant AGI.
Well, it appears that no published work in AI has ended in successful strong artificial intelligence.
It might be making visible progress, or failing that, at least not making basic fatal errors.
I assume this is to be interpreted as "published work in AGI". Plenty of perfectly good AI work around.
Yes, I meant AGI by AI. I don't consider any of the stuff outside AGI to be worth calling AI. The good stuff there is merely the more or less successful descendants of spinoffs of failed attempts to create AGI, and is good in direct proportion to its distance from that original vision.
You(EY)'ve mentioned moral beliefs from time to time, but I don't recall you addressing morality directly at length. A commonly expressed view in rationalist circles is that there is no such thing, but I don't think that is your view. What is a moral judgement, and how do you arrive at them?
ETA: As Psy-Kosh points out, he has, so scratch that unless EY has something more to say on the matter.
Actually, he has.
Could you give an uptodate estimate of how soon non-Friendly general AI might be developed? With confidence intervals, and by type of originator (research, military, industry, unplanned evolution from non-general AI...)
Your approach to AI seems to involve solving every issue perfectly (or very close to perfection). Do you see any future for more approximate, rough and ready approaches, or are these dangerous?
What are your current techniques for balancing thinking and meta-thinking?
For example, trying to solve your current problem, versus trying to improve your problem-solving capabilities.
Since you and most around here seem to be utilitarian consequentialists, how much thought have you put into developing your personal epistemological philosophy?
Worded differently, how have you come to the conclusion that "maximizing utility" is the optimized goal as opposed to say virtue seeking?
*blinks* I'm curious as to what it is you are asking. A utility function is just a way of encoding and organizing one's preferences/values. Okay, there're a couple additional requirements like internal consistency (if you prefer A to B and B to C, you'd better prefer A to C) and such, but other than that, it's just a convenient way of talking about one's preferences.
The goal isn't "maximize utility", but rather "maximizing utility" is a way of stating what it is you're doing when you're working to achieve your goals. Or did I completely misunderstand?
I think there has to be more to utility function talk than "convenience" - for one thing, it's not more convenient than preference talk, in general. Consider an economic utility function, valuing bundles of apples and oranges. If someone's preferences are summarizable by U(apples, oranges)=sqrt(apples*oranges), that might be convenient, but there's no free lunch. No compression can be achieved without assumptions about the prior distribution. Believing that preferences tend to have terse expressions in functional talk is a claim about the actual distribution of preferences in the world. The belief that maximizing utility is a perspicuous way of expressing "behave correctly" is something that one has to have evidence for.
My (very partial) understanding of virtue morality is that virtue ethicists believe that "behave correctly" is well expressed in terms of virtues.
I didn't mean convenient in the sense of compressibility, but convenient in the sense of representing our preference ordering in a form that lets one then talk about stuff like "how can I get the world into the best possible state, where 'best' is in terms of my values?" in terms of maximizing utility, and when combined with uncertainty, maximizing expected utility.
I just meant "utility doesn't automatically imply a specific set of values/virtues. It's more a way of organizing your virtues so that you can at least formally define optimal actions, giving you a starting point to look for ways to approximately compute such things, etc.."
Or did I misunderstand your point completely?
The phrase "how can I get the world into the best possible state" is explicitly consequentialist. Non-consequentialists (e.g. "The end does not justify the means") do not admit that correct behavior is getting the world into the best possible state.
Non-utilitarians probably perceive suggestions of maximizing utility, maximizing expected utility, and (in particular) approximating those two as very dangerous and likely to lead to incorrect behavior.
The original poster implied that there is a difference between seeking to maximize utility and (for example) virtue seeking. I'm trying to explain in what sense the original poster had a real point. Not everyone is a utilitarian, and saying "in principle, I could construct a utility function from your preferences" doesn't make everyone a utilitarian.
Really, the non-consequentialism can be rephrased as a consequentialist philosophy by simply including the means, ie, the history, as part of the "state"... ie, assigning lower value to getting to a certain state by bad methods vs good methods.
Or am I still not getting it?
Yes, it's possible to encode the nonconsequentialism or "nonutilitarianism" into the utility function. However, by doing so you're making the utility function inconvenient to work with. You can't simultaneously claim that the utility function is "simply" an encoding of people's preferences and ALSO that the utility function is convenient or preferable.
Then you go and approximate the (uglified) utility function! Put yourself in the virtue theorist's or Kantian's shoes. It certainly sounds to me like you're planning to discard their concerns regarding moral/ethical/correct behavior.
(Note: I don't actually understand virtue ethics at all, so I might be getting this entirely wrong.) Imagine the virtue ethicist saying "Your concerns can be encoded into the virtue of "achieves a desirable goal", and will be included in our system along with the other virtues," Would you want to know WHY the system is being built with virtues at the bottom and consequentialism as an encoding? Would your questions make sense?
It's "convenient" in the sense of giving us a general way of talking about how to make decisions. It's "convenient" in that it is set up in such a way to encode not just what you prefer more than other stuff, but how much more, etc...
Lets us then also take advantage of whatever decision theory theorems have been proven, and so on...
As far as "virtue of achieving a desirable goal", "desirable", "virtue", and "achieving" would be doing all the heavy lifting there. :)
But really, my point was simply the original comment was stated in such a way as to imply "maximizing utility" was itself a moral philosophy, ie, the sort of thing that you could say "I consider that immoral, and instead care about personal virtue". I was simply saying "huh? utility stuff is just a way of talking about whatever values you happen to have. It's not, on its own, a specific set of values. It's like, I guess, saying 'what if I don't believe in math and instead believe in electromagnetism?'"
You'll have to forgive me because I am economist by training and mentions of utility have very specific references to Jeremy Bentham.
Your definition of what the term "maximizing utility" means and the Bentham definition (who was the originator) are significantly different; If you don't know what it is then I will describe it (if you do, sorry for the redundancy).
Jeremy Bentham devised Felicific calculus which is a hedonistic philosophy and seeks as its defining purpose to maximize happiness. He was of the opinion that it was possible in theory to create a literal formula which gives optimized preferences such that it maximized happiness for the individual. This is the foundation for all utilitarian ethics as each seeks to essentially itemize all preferences.
Virtue ethics for those who do not know is the Aristotelian philosophy that posits: each sufficiently differentiated organism or object is naturally optimized for at least one specific purpose above all other purposes. Optimized decision making for a virtue theorist would be doing the things which best express or develop that specific purpose - similar to how specialty tools are best used for their specialty. Happiness is said to spring forth from this as a consequence, not as it's goal.
I just want to know, if it is the case that he came to follow the former (Bentham) philosophy, how he came to that decision (theoretically it is possible to combine the two).
So in this case, while the term may give an approximation of the optimal decision, if used in that manner is not explicitly clear in how it determines the basis for the decision is in the first place; that is unless, as some have done, it is specified that maximizing happiness is the goal (which I had just assumed people were asserting implicitly anyhow).
Okay, I was talking about utility maximization in the decision theory sense. ie, computations of expected utility, etc etc...
As far as happiness being The One True Virtue, well, that's been explicitly addressed
Anyways, "maximize happiness above all else" is explicitly not it. And utility, as discussed on this site is a reference to the decision theoretic concept. It is not a specific moral theory at all.
Now, the stuff that we consider morality would include happiness as a term, but certainly not as the only thing.
Virtue ethics, as you describe it, gives me an "eeew" reaction, to be honest. It's the right thing to do simply because it's what you were optimized for?
If I somehow bioengineer some sort of sentient living weapon thing, is it actually the proper moral thing for that being to go around committing mass slaughter? After all, that's what it's "optimized for"...
Thanks, I followed up below.
...very little, you know me, I usually just wing that epistemology stuff...
(seriously, could you expand on what this question means?)
Ha, fair enough.
I often see reference to maximizing utility and individual utility functions in your writing and it would seem to me (unless I am misinterpreting your use) that you are implying that hedonic (fellicific) calculation is the most optimal way to determine what is correct when applying counterfactual outcomes to optimizing decision making.
I am asking how you determined (if that is the case) that the best way to judge the optimality of decision making was through utilitarianism as opposed to say ethical egoism or virtue (not to equivocate). Or perhaps your reference is purely abstract and does not invoke the fellicific calculation.
See Not For The Sake of Happiness (Alone).
See The "Intuitions" Behind "Utilitarianism" for a partial answer.
Yes I remember reading both and scratching my head because both seemed to beat around the bush and not address the issues explicitly. Both lean to much on addressing the subjective aspect of non-utility based calculations, which in my mind is a red herring.
Admittedly I should have referenced it and perhaps the issue has been addressed as well as it will be. I would rather see this become a discussion as in my mind it is more important than any of the topics dealt with daily here - however that may not be appropriate for this particular thread.
"Preference satisfaction utilitarianism" is a lot closer to Eliezer's ethics than hedonic utilitarianism. In other words, there's more important things to maximize than happiness.
Which areas of science or angles of analysis currently seem relevant to the FAI problem, and which of those you've studied seem irrelevant? What about those that fall on the "AI" side of things? Fundamental math? Physics?
I think we can take a good guess on the last part of this question on what he will say: Bayes Theorem, Statistics, basic Probability Theory Mathematical Logic, and Decision Theory.
But why ask the question with this statement made by EY: "Since you don't require all those other fields, I would like SIAI's second Research Fellow to have more mathematical breadth and depth than myself." (http://singinst.org/aboutus/opportunities/research-fellow)
My point is he has answered this question before...
I add to this my own question actually it is more of a request to see EY demonstrate TDT with some worked out math on a whiteboard or some such on the video.
What five written works would you recommend to an intelligent lay-audience as a rapid introduction to rationality and its orbital disciplines?
Previously, you said that a lot of work in Artificial Intelligence is "5% intelligence and 95% rigged demo". What would you consider an example of something that has a higher "intelligence ratio", if there is one, and what efforts do you consider most likely to increase this ratio?
DARPA's Grand Challenge produced several intelligent cars and was definitely not a rigged demo.
I am 99.99% certain that he will not ignore such questions.
I am 99.995% certain that no question will receive that many votes.
There is a greater than 0.01% chance that Eliezer or another administrator will edit the site to display a score of "3^^^3" for some post. (Especially now that it's been suggested.)
I guess I need to recalibrate!
Not if you meant 3^^^3 "Vote up" clicks registered from distinct LW accounts.
Let E(t) be the set of historical information available up until some time t, where t is some date (e.g. 1934). Let p(A|E) be your estimate of the probability an optimally rational Bayesian agent would assign to the event "Self-improving artificial general intelligence is discovered before 2100" given a certain set of historical information.
Consider the function p(t)=p(A|E(t)). Presumably as t approaches 2009, p(t) approaches your own current estimate of p(A).
Describe the function p(t) since about 1900. What events - research discoveries, economic trends, technological developments, sci-fi novel publications, etc, caused the largest changes in p(t)? Is it strictly increasing, or does it fluctuate substantially? Did the publication of any impossibility proofs (e.g. No Free Lunch) cause strong decreases in p(t)? Can you point to any specific research results that increased p(t)? What about the "AI winter" and related setbacks?
I don't think this question behaves the way you want it to. Why not ask what a smart human would predict?
Can you make a living out of this rationality / SI / FAI stuff . . . or do you have to be independently wealthy?
Do you act all rational at home . . or do you switch out of work mode and stuff pizza and beer in front of the TV like any normal akrasic person? (and if you do act all rational, what do your partner/family/housemates make of it? do any of them ever give you a slap upside the head?)
:-)
*coughs*
But rationally maximizing utility will still look different from going where your whims take you. Hopefully! Otherwise there would be no value to rationality.
Even if you assume that it's about decisions rather than roles, the question is either "are you fallible?" or "what compromises do you make between sending proper signals to your partner/family/housemates and between doing what you believe makes your (and possibly their) life better?" The former is poorly conceived - what do you want him to say, "yes, but less than other people"? - and the latter poorly expressed.
How does it look like when a person "acts rationally"? Do I hear connotations with dreaded Mr. Spock?
no, not at all, I don't think rational = unemotional (and I liked EY's article explaining how it is perfectly rational to feel sad ... when something sad happens).
But rationality does seem to be stongly associated with a constant meta-analytical process: always thinking about a decision, then thinking about the way we were thinking about the decision, and then thinking about the self-imposed axioms we have used to model the way that we were thinking about the meta-thinking, and some angst about whether there are undetected biases in the way that .. yada yada yada.
which is all great stuff,
but I wondered whether rationalists are like that all the time, or whether they ever come home late, open a beer or two and pick their nose while transfixed by czechoslvakian wrestling on ESPN, without stopping to wonder why they are doing it, and wouldn't it be more rational to go to bed already.
I recently started to notice this very question popping up in my head when I find myself in a situation like the one you described :) I didn't consciously install this habit, it just started to manifest itself some time ago, probably several months.
But I do find myself in similar situations, usually after a good workday -- just substitute a cup of tea (or a small shot of Becherovka) for the beer, and mindless internet surfing for the wrestlers :)
(Related: Ego Depletion: Is the Active Self a Limited Resource?)
I am sure you're familiar with the University of Chicago "Doomsday Clock", so: if you were in charge of a Funsday Clock, showing the time until positive singularity, what time would it be on? Any recent significant changes?
(Idea of Funsday Clock blatantly stolen from some guy on Twitter.)
If a negative singularity rules out any positive singularities, wouldn't a Funsday Clock be superseded by a Singularity Clock?
Well, a negative singularity would belong on the Doomsday Clock. Actually, that might be the proper way to think of it: turn past midnight meaning negative, past noon meaning positive. It'd imply a scale, too.
How would a utopia deal with human's seemingly contradicting desires - the desire to go up in status and the desire to help lower status people go up in status. Because helping lower status people go up in status will hurt our own status positions. I remember you mentioning how in your utopia you would prefer not to reconfigure the human mind. So how would you deal with such a problem?
(If someone finds the premise of my question wrong, please point it out)
I don't think most people want to actually raise people who are lower status than themselves up to higher than themselves. I actually don't think that most people want to raise other's status very much. They seem to typically be more concerned with raising the material welfare of people who are significantly worse off, which doesn't necessarily change status. The main status effect of altruistic behavior is to raise the status of the altruist. For instance, consider the quote "It is more blessed to give than to receive." (Acts 20:35). If we think of "blessedness" as similar to status (status in the eyes of god maybe?) then a "status altruist" would read that and decide to always receive and never give in order raise the status of others. The traditional altruistic interpretation though is to give, and therefore become more blessed than the poor suckers who you are giving to.
Wouldn't you, in a perfect world, have everyone go up in status without your status being affected? Wouldn't that be the utilitarian thing to do?
That's not possible if status is zero-sum, which it appears to be. If everyone is equal in status, wouldn't it be meaningless, like everyone being equally famous?
Actually, let me qualify. Everyone being equally famous wouldn't necessarily be meaningless, but it would change the meaning of famous - instead of knowing about a few people, everyone would know about everyone. It would certainly make celebrity meaningless. I'm not really up to figuring out what equivalent status would mean.
Equivalent status is not desirable, people would just find ways of going up in status - or at least want to. Which is where the contradicting desires fall in. I guess in any utopia there will always be a status struggle. Maybe what we want is an equal opportunity at going up in status. That way we don't feel bad for going up in status ourselves.
I am not an altruist. I would like my status to be higher than it is. I would like the few people I truly care about to have higher status. Otherwise, I really don't care, except for enjoying when certain high profile d-bags lose a lot of status. But that's just me. My more general point was that even those altruistic sort who do care deeply about others don't seem to generally want to raise others up above themselves, either in terms of status or materially. Can you think of one counter-example? I can't but I'm not trying very hard.
Like I mentioned in another comment, I would feel bad for lower status people who didn't have an equal opportunity as me for reaching the level of status I have. Like I am thriving in this world for being born lucky. And AFAIK, relatively no one lower status than me had the same opportunity as me to be in the status level I am in now.
It's almost impossible for one person's morality to be significantly different from the standard. It's more likely that one who thinks themself different is simply confused.
Um, what standard of significance are you using here? Yes, humans are extremely similar compared to the vastness of that which is possible, but that doesn't mean the remaining difference isn't ridiculously important.
The standard implied by the remark I was commenting on. Literally not caring about other people seems like something you may believe about yourself, but which can't be true.
I read the original post as being about the ordinary human domain, implying an ordinary human-relative standard of significance.
This is ambiguous in two ways: which other people (few or all), and what sort of valuation (subjunctive revealed preference, some construal of reflective equilibrium)? I suppose it's plausible that for every person, some appeal to empathy would sincerely motivate that person.
The underlying genetic machinery that produces an individual's morality is a human universal. But the production of the morality is very likely dependent upon non-genetic factors. The psychological unity of humankind no more implies that people have the same morality than it implies that they have the same favorite foods.
Yes, but it's very easy for the actual large scale consequences of a human morality to be very different. We all feel compasion for freinds and fear of strangers; but when we scale our morality to the size of humanity, the difference is huge depending whether the compassion or the fear dominates.
Hitler and Ghandi may not be that different, but the consequences of their actions were.
In reference to this comment, can you give us more information about the interface between the modules. Also what leads you to believe that a human level intelligence can be decomposed nicely in such a fashion.
What was the story purpose and/or creative history behind the legalization and apparent general acceptance of non-consensual sex in the human society from Three Worlds Collide?
Excellent; I was going to ask that myself. Clearly Eliezer wanted an example to support his oft-repeated contention that the future like the past will be filled with people whose values seem abhorrent to us. But why he chose that particular example I'd like to know. Was it the most horrific(-sounding) thing he could come up with some kind of reasonable(-sounding) justification for?
It's not at all clear to me that coming up with a reasonable-sounding justification was part of the project. One isn't provided in the story, one wasn't presented as part of an answer to an earlier question of mine, etc. etc.
here
This isn't an explanation at all.
I don't see the need for more than this:
I just figured that these humans have been biologically altered to have a different attitude towards sex. Perhaps, for them, initiating sex with someone is analogous to initiating a conversation. Sure, you wish that some people wouldn't talk to you, but you wouldn't want to live in a world where everyone needed your permission before initiating a conversation. Think of all the interesting conversations you'd miss!
And if that's what's going on, that would constitute a (skeezy) answer to my question, but I'd like to hear it from the story's author. Goodness knows it would annoy me if people started drawing inaccurate conclusions about my constructed worlds when they could have just asked me and I would have explained.
Alicorn: On the topic of your constructed worlds, I would be fascinated to read how your background in world-building (which, iirc, was one focus of your education?) might contribute to our understanding of this one.
Yes, worldbuilding was my second major (three cheers for my super-cool undergrad institution!). My initial impression of Eliezer's skills in this regard from his fiction overall are not good, but that could be because he tends not to provide very much detail. It's not impossible that the gaps could be filled in with perfectly reasonable content, so the fact that these gaps are so prevalent, distracting, and difficult to fill in might be a matter of storytelling prowess or taste rather than worldbuilding abilities. (It's certainly possible to create a great world and then do a bad job of showcasing it.) I should be able to weigh in on this one in more detail if and when I get an answer to the above question, which is a particularly good example of a distracting and difficult-to-fill-in gap.
If I understand EY's philosophy of predicting the future correctly, the gaps in the world are intentional.
Suppose that you are a futurist, and you know how hard it is to predict the future, but you're convinced that the future will be large, complicated, weird, and hard to connect directly to the present. How can you provide the reader with the sensation of a large, complicated, weird, and hard-to-connect-to-the-present future?
Note that as a futurist, the conjunction fallacy (more complete predictions are less likely to be correct) is extremely salient in your thinking.
You put deliberate gaps into your stories, any resolution of which would require a large complicated explanation - that way the reader has the desired (distracting and difficult-to-fill-in) sensation, without committing the author to any particular resolution.
The purpose was to test the waters for another story he was developing; there probably wasn't an in-story purpose to it beyond the obvious one of making it clear that the younger people had a very different worldview than the one we have now. He's been unwilling to give more detail because the reaction to the concept's insertion in that story was too negative to allow him to safely (without reputational consequence, I assume) share the apparently much more questionable other story, or, seemingly, any details about it.
I did upvote your question, by the way. I want to hear more about that other story.
I don't see it doing much good to his reputation to stay silent either, given the inflammatory nature of the remark. Sure, people will be able to quote that part to trash Eliezer, but that's a lot worse than if someone could link a reasonable clarification in his defense.
Yes, I voted Alicorn's question up. I want to know too.
Actually, there's a very good clarification of his views on rape in the context of our current society later in that same comment thread that could be linked to. It didn't seem to be relevant to this conversation, though.
That's certainly an explanation. "Very good" and "clarifying" are judgment calls here...
<non-sarcastic> <non-rhetorical> How could it be better? What parts still need clarifying?
In the spirit of considering semi abyssal plans, what happens if, say, next week you discover a genuine reduction of consciousness and in turns out that... There's simply no way to construct the type of optimization process you want without it being conscious, even if very different from us?
ie, what if it turned out that The Law turned out to have the consequence of "to create a general mind is to create a conscious mind. No way around that"? Obviously that shifts the ethics a bit, but my question is basically if so, well... "now what?" what would have to be done differently, in what ways, etc?
What is your current position about the FOOM effect, when the exploding intelligence quickly acquires the surrounding matter for its own usage. Solves its computing needs by transforming everything nearby to something more computationally optimal. And that by some not so obvious physical operations, yet entirely permitted and achievable from "pure calculating" already granted to ("seed") AI?
What is a typical EY workday like? How many hours/day on average are devoted to FAI research, and how many to other things, and what are the other major activities that you devote your time to?
Your "Bookshelf" page is 10 years old (and contains a warning sign saying it is obsolete):
http://yudkowsky.net/obsolete/bookshelf.html
Could you tell us about some of the books and papers that you've been reading lately? I'm particularly interested in books that you've read since 1999 that you would consider to be of the highest quality and/or importance (fiction or not).
What is your information diet like? Do you control it deliberately (do you have a method; is it, er, intelligently designed), or do you just let it happen naturally.
By that I mean things like: Do you have a reading schedule (x number of hours daily, etc)? Do you follow the news, or try to avoid information with a short shelf-life? Do you frequently stop yourself from doing things that you enjoy (f.ex reading certain magazines, books, watching films, etc) to focus on what is more important? etc.
What do you view as your role here at Less Wrong (e.g. leader, preacher, monk, moderator, plain-old contributor, etc.)?
If you were to disappear (freak meteorite accident), what would the impact on FAI research be?
Do you know other people who could continue your research, or that are showing similar potential and working on the same problems? Or would you estimate that it would be a significant setback for the field (possibly because it is a very small field to begin with)?
In 2007, I wrote a blog post titled Stealing Artificial Intelligence: A Warning for the Singularity Institute.
Short summary: After a few more major breakthroughs, when AGI is almost ready, AI will no doubt appear on the radar of many powerful organizations, such as governments. They could spy on AGI researchers and steal the code when it is almost ready (or ready, but not yet Certified Friendly) and launch their copy first, but without all the care and understanding required.
If you think there's a real danger there, could you tell us what the SIAI is doing to minimize it? If it doesn't apply to the SIAI, do you know if other groups working on AGI have taken this into consideration? And if this scenario is not realistic, could you tell us why?
Isn't it too early to start solving this problem? There is a good chance SIAI won't even have a direct hand in programming the FAI.
That's what I've been told, but I'm not entirely convinced. Since there are so many timelines out there, and since fundamental breakthroughs are hard to predict, I think it still deserves some attention as soon as possible, if only to know what to do if things start moving rapidly (an AGI team might not have many chances to recover from security mistakes).
I'll broaden my question a bit so that it applies to all people working on AGI and not just the SIAI.
If they're going to have that exact wrong level of cluefulness, why wouldn't they already have a (much better-funded, much less careful) AGI project of their own?
As Vladimir says, it's too early to start solving this problem, and if "things start moving rapidly" anytime soon, then AFAICS we're just screwed, government involvement or no.
Maybe they do, maybe they don't. I won't try to add more details to a scenario because that's not the right way to think about this, IMO. If it happens, it probably won't be a movie plot scenario anyway ("Spies kidnap top AI research team and torture them until they make a few changes to program, granting our Great Leader dominion over all")...
What I'm interested in is security of AGI research in general. It would be extremely sad to see FAI theory go very far only to be derailed by (possibly well-intentioned) people who see AGI as a great source of power and want to have it "on their side" or whatever.
Fear of others stealing your ideas is a crank trope, which suggests it may be a common human failure mode. It's far more likely that SIAI is slower at developing (both Friendly and unFriendly) AI than the rest of the world. It's quite hard for one or a few people to be significantly more successfully innovative than usual, and the rest of the world is much, much bigger than SIAI.
I think it might be correct in the entrepreneur/startup world, but it probably isn't when it comes to technologies that are this powerful. Just think of nuclear espionage and of the kind of security that surrounds the development of military and intelligence hardware and software. If you're building something that could overthrow all the power structures in the world, it would be surprising if nobody tried to spy on you (or worse; kill you, derail the project, steal the almost finished code, etc).
I'm not saying it only applies to the SIAI (though my original post was directed only at them, my question here is about the AGI research world in general, which includes the SIAI), or that it isn't just one of many many things that can go wrong. But I still think that when you're playing with stuff this powerful, you should be concerned with security and not just expect to forever fly under the radar.
Let's be realistic here - the AGI research world is a small fringe element of AI research in general. The AGI research world generally has a high opinion of its own importance - an opinion not generally shared by the AI research world in general, or the world as a whole.
We are in a self-selected group of people who share our beliefs. This will bias our thinking, leading us to be too confident of our shared beliefs. We need to strive to counter that effect and keep a sense of perspective, particularly when we're trying to anticipate what other people are likely to do.
I'm not sure I get what you're saying.
Either the creation of smarter than human intelligence is the most powerful thing in the world, or it isn't.
If it is, it would be surprising if nobody in the powerful organizations I'm talking about realizes it, especially if a few breakthroughs are made public and as we get closer to AGI.
If that is the case, this probably means that at some point AGI researchers will be "on the radar" of these people and that they should at least think about preparing for that day.
You can't have your cake and eat it too; you can't believe that AGI is the most important thing in the world and simultaneously think that it's so unimportant that nobody's going to bother with it.
I'm not saying that right now there is much danger of that . But if we can't predict when AGI is going to happen (which means wide confidence bounds, 5 years to 100 years, as Eliezer once said.), then we don't know how soon we should start thinking about security, which probably means that as soon as possible is best.
The reason to point out the crackpot aspect (e.g. item 12 in Baez's Crackpot Index) is to adjust how people think about this question, not to argue that the question shouldn't be asked or answered.
In particular, I want people to balance (at least) two dangers - the danger of idea-stealing and the danger of insularity slowing down innovation.
Why do you have a strong interest in anime, and how has it affected your thinking?
Previously, you endorsed this position:
One counterexample has been proposed a few times: holding false beliefs about oneself in order to increase the appearance of confidence, given that it's difficult to directly manipulate all the subtle signals that indicate confidence to others.
What do you think about this kind of self-deception?
Costs outweigh the benefits.
Is your pursuit of a theory of FAI similar to, say, Hutter's AIXI, which is intractable in practice but offers an interesting intuition pump for the implementers of AGI systems? Or do you intend on arriving at the actual blueprints for constructing such systems? I'm still not 100% certain of your goals at SIAI.
How did you win any of the AI-in-the-box challenges?
Something tells me he won't answer this one. But I support the question! I'm awfully curious as well.
http://news.ycombinator.com/item?id=195959
"Oh, dear. Now I feel obliged to say something, but all the original reasons against discussing the AI-Box experiment are still in force...
All right, this much of a hint:
There's no super-clever special trick to it. I just did it the hard way.
Something of an entrepreneurial lesson there, I guess."
Why exactly do majorities of academic experts in the fields that overlap your FAI topic, who have considered your main arguments, not agree with your main claims?
Who are we talking about besides you?
I'd consider important overlapping academic fields to be AI and long term economic growth; I base my claim about academic expert opinion on my informal sampling of such folks. I would of course welcome a more formal sampling.
Who's considered my main arguments besides you?
Me - if I qualify as an academic expert is another matter entirely of course.
I'm not comfortable publicly naming names based on informal conversations. These folks vary of course in how much of the details of your arguments they understand, and of course you could always set your bar high enough to get any particular number of folks who have understood "enough."
Okay. I don't know any academic besides you who's even tried to consider the arguments. And Nick Bostrom et. al., of course, but AFAIK Bostrom doesn't particularly disagree with me. I cannot refute what I have not encountered, I do set my bar high, and I have no particular reason to believe that any other academics are in the game. I could try to explain why you disagree with me and Bostrom doesn't.
This disadvantages questions which are posted late (to a greater extent than would give people an optimal incentive to post questions early). (It also disadvantages questions which start with a low number of upvotes by historical accident and then are displayed low on the page, and are not viewed as much by users who might upvote them.)
It's not your fault; I just wish the LW software had a statistical model which explained observed votes and replies in terms of a latent "comment quality level", because of situations like this, where it could matter if a worse comment got a high rating while a better comment got a low one. (I also wish forums with comment ratings used ideas related to value of information, optimal sequential preference elicitation, and/or n-armed bandit problems to decide when to show users comments whose subjective latent quality has a low marginal mean but a high marginal variance, in case the (")true(") quality of a comment is high, because of the possibility that a user will rate the comment highly and let the forum software know that it should show the comment to other users.)
There's probably significant value in the low-hanging fruit of just tweaking the parameters in the current algorithm (which are currently set for the much larger reddit!). Don't let the perfect be the enemy of the good.
Reddit has implemented a 'best' view which tries to compensate for this kind of thing: http://blog.reddit.com/2009/10/reddits-new-comment-sorting-system.html
LW is based on reddit's source code, so it should be relatively easy to integrate.
Okay: Goedel, Escher, Bach. You like it. Big-time.
But why? Specifically, what insights should I have assimilated from reading it that are vital for AI and rationalist arts? I personally feel I learned more from Truly Part of You than all of GEB, though the latter might have offered a little (unproductive) entertainment.
Are the book(s) based on your series of posts are OB/LW still happening? Any details on their progress (title? release date? e-book or real book? approached publishers yet? only technical books, or popular book too?), or on why they've been put on hold?
http://lesswrong.com/lw/jf/why_im_blooking/
Yes, that is my current project.
What is the probability that this is the ultimate base layer of reality?
could you explain more what this means?
Are you a meat-eater?
Looks like.
What practical policies could politicians enact that would increase overall utility? When I say "practical", I'm specifically ruling out policies that would increase utility but which would be unpopular, since no democratic polity would implement them.
(The background to this question is that I stand a reasonable chance of being elected to the Scottish Parliament in 19 months time).
I'd guess that legalizing gay marriage would be pretty low-hanging fruit, but I don't know how politically possible it is.
Free trade. As a politician, you can't do more than that.
And open immigration policies
Ruling out unpopular measures is tantamount to giving up on your job as a politician; the equivalent of an individual ruling out any avenues to achieving their goals that require some effort.
Much as rationality in an individual consists of "shutting up and multiplying", i.e. computing which course of action including those we have no taste for yields the highest expected utility, politics - the useful part of it - consists of making necessary policies palatable to the public. The rest is demagoguery.
Do you think that morality or rationality recommends placing no intrinsic weight or relevance on either a) backwards-looking considerations (e.g. having made a promise) as opposed to future consequences, or b) essentially indexical considerations (e.g. that I would be doing something wrong)?
What if the friendly AI finds that our extrapolated volition is coherent and contains the value of 'self-determination' and concludes that it cannot meddle too much in our affairs? "Well, humankind, it looks like you don't want to have your destiny decided by a machine. my hands are tied. You need to save yourselves."
http://lesswrong.com/lw/xb/free_to_optimize/
Favourite album post-1960?
Of the questions you decide not to answer, which is most likely to turn out to be a vital question you should have publicly confronted?
Not the question you don't want to answer but would probably have bitten the bullet anyway. The question you would have avoided completely if it weren't for my question.
[Edit - "If I thought they were vital, I wouldn't avoid" would miss the point, as not wanting to consider something suppresses counterarguments to dismissing it. Take a step back - which question is most likely to be giving you this reaction?]
Well, Eliezer's reply to this comment prompts a follow-up question:
In "Free to optimize", you alluded to "the gift of a world that works on improved rules, where the rules are stable and understandable enough that people can manipulate them and optimize their own futures together". Can you say more about what you imagine such rules might be ?