Rationality Quotes August 2012
Here's the new thread for posting quotes, with the usual rules:
- Please post all quotes separately, so that they can be voted up/down separately. (If they are strongly related, reply to your own comments. If strongly ordered, then go ahead and post them together.)
- Do not quote yourself
- Do not quote comments/posts on LW/OB
- No more than 5 quotes per person per monthly thread, please.
Loading…
Comments (426)
Ta-nehisi Coates
Yes -- and to me, that's a perfect illustration of why experiments are relevant in the first place! More often than not, the only reason we need experiments is that we're not smart enough. After the experiment has been done, if we've learned anything worth knowing at all, then hopefully we've learned why the experiment wasn't necessary to begin with -- why it wouldn't have made sense for the world to be any other way. But we're too dumb to figure it out ourselves! --Scott Aaronson
Or at least confirmation bias makes it seem that way.
Also hindsight bias. But I still think the quote has a perfectly valid point.
Agreed.
-Seth Godin
Could you add the link if it was a blog post, or name the book if the source was a book?
Done.
A common piece of advice from pro Magic: the Gathering plays is "focus on what matters." The advice is mostly useless to many people though because the pros have made it to that level precisely because they know what matters to begin with.
perhaps the better advice, then, is "when things aren't working, consider the possibility that it's because your efforts are not going into what matters, rather than assuming it is because you need to work harder on the issues you're already focusing on"
That's a much better advice than Godin's near-tautology.
-- Erika Moen
I wonder how common it is for people to agentize accidents. I don't do that, but, annoyingly, lots of people around me do.
M. Mitchell Waldrop on a meeting between physicists and economists at the Santa Fe Institute:
-- Steven Dutch
Interviewer: How do you answer critics who suggest that your team is playing god here?
Craig Venter: Oh... we're not playing.
-- The Last Psychiatrist
Is it our bias towards optimism? (And is that bias there because pessimists take fewer risks, and therefore don't succeed at much and therefore get eliminated from the gene pool?)
I heard (on a PRI podcast, I think) a brain scientist give an interpretation of the brain as a collection of agents, with consciousness as an interpreting layer that invents reasons for our actions after we've actually done them. There's evidence of this post-fact interpretation - and while I suspect this is only part of the story, it does give a hint that our conscious mind is limited in its ability to actually change our behavior.)
Still, people do sometimes give up alcohol and other drugs, and keep new resolutions. I've stuck to my daily exercise for 22 days straight. These feel like conscious decisions (though I may be fooling myself) but where my conscious will is battling different intentions, from different parts of my mind.
Apologies if that's rambling or nonsensical. I'm a bit tired (because every day I consciously decide to sleep early and every day I fail to do it) and I haven't done my 23rd day's exercise yet. Which I'll do now.
-- Friedrich Hayek, The Fatal Conceit : The Errors of Socialism (1988), p. 6
Gary Drescher, Good and Real
-Friedrich Nietzsche
“Ignorance killed the cat; curiosity was framed!” ― C.J. Cherryh
(not sure if that is who said it originally, but that's the first creditation I found)
--Bertrand Russell, A History of Western Philosophy
Russell <strike>gives too much credit to radical empiricism</strike> fails to warn against the dangers of going too far in the direction of radical empiricism, which is really just as bad as radical rationalism.
FTNI, by Ayn Rand
I wasn't trying to endorse the whole empiricist philosophy, and neither was Russell, at least in this quote. The rationality lesson it offers is not "radical empiricism good, radical rationalism bad" but more like "a wide base of principles with connections to experience good, a small base of abstract logical principles bad".
er, I agree my comment was poorly phrased. Instead of accusing him of giving positive credit to radical empiricism I probably should have said, while he's making a good point warning against the dangers of radical rationalism, he was failing to warn against the dangers of going too far in the direction of empiricism.
That's why I prefer the quote I followed up with, it is more careful to reject both of these approaches.
Recognising the weaknesses inherent in human logical deductions?
I often find that I'm not well read enough or perhaps not smart enough to decipher the intricate language of these eminent philosophers. I'd like to know is Russell talking about something akin to scientific empiricism? Can someone enlighten me? From my shallow understanding though, it seems like what he is saying is almost common sense when it comes to building knowledge or beliefs about a problem domain.
The idea that one should not philosophize keeping close contact with empirical facts, instead of basing a long chain of arguments on abstract "logical" principles like Leibniz's, may be almost common sense now, but it wasn't in the early modern period of which Russell was talking about. And when Russell wrote this (1940s) he was old enough to remember that these kind of arguments were still prevalent in his youth (1880s-1890s) among absolute idealists like Bradley, as he describes in "Our Knowledge of the External World" (follow the link and do a Ctrl-F search for Bradley). So it did not seem to him a way of thinking that was so ancient and outdated as to be not worth arguing against.
ETA: I meant, "The idea that one should philosophize keeping...", without not, obviously.
Ah very good, in that context it makes perfect sense.
Explanations are all based on what makes it into our consciousness, but actions and the feelings happen before we are consciously aware of them—and most of them are the results of nonconscious processes, which will never make it into the explanations. The reality is, listening to people’s explanations of their actions is interesting—and in the case of politicians, entertaining—but often a waste of time. --Michael Gazzaniga
Does that apply to that explanation as well?
Does it apply to explanations made in advance of the actions? For example, this evening (it is presently morning) I intend buying groceries on my way home from work, because there's stuff I need and this is a convenient opportunity to get it. When I do it, that will be the explanation.
In the quoted article, the explanation he presents as a paradigmatic example of his general thesis is the reflex of jumping away from rustles in the grass. He presents an evolutionary just-so story to explain it, but one which fails to explain why I do not jump away from rustles in the grass, although surely I have much the same evolutionary background as he. I am more likely to peer closer to see what small creature is scurrying around in there. But then, I have never lived anywhere that snakes are a danger. He has.
And yet this, and split-brain experiments, are the examples he cites to say that "often", we shouldn't listen to anyone's explanations of their behaviour.
I smell crypto-dualism. "I thought there was a snake" seems to me a perfectly good description of the event, even given that I jumped way before I was conscious of the snake. (He has "I thought I'd seen a snake", but this is a fictional example, and I can make up fiction as well as he can.)
The article references his book. Anyone read it? The excerpts I've skimmed on Amazon just consist of more evidence that we are brains: the Libet experiments, the perceived simultaneity of perceptions whose neural signals aren't, TMS experiments, and so on. There are some digressions into emergence, chaos, and quantum randomness. Then -- this is his innovation, highlighted in the publisher's blurb -- he sees responsibility as arising from social interaction. Maybe I'm missing something in the full text, but is he saying that someone alone really is just an automaton, and only in company can one really be a person?
I believe there are people like that, who only feel alive in company and feel diminished when alone. Is this is just an example of someone mistaking their idiosyncratic mental constitution for everybody's?
There is a famous study that digs a bit deeper and convincingly demonstrates it: Telling more than we can know: Verbal reports on mental processes.
From the abstract:
It seems to me that "cognitive processes" could be replaced by "physical surroundings", and the resulting statement would still be true. I am not sure how significant these findings are. We have imperfect knowledge of ourselves, but we have imperfect knowledge of everything.
Did you in fact buy the groceries?
I did.
There are many circumstances that might have prevented it; but none of them happened. There are many others that might have obstructed it; but I would have changed my actions to achieve the goal.
Goals of such a simple sort are almost invariably achieved.
Three upvotes for demonstrating the basic competence to buy groceries?
Obviously not, since Gazzaniga is not explaining his own actions.
He is, among other things, explaining some of his own actions: his actions of explaining his actions.
You seem to have failed to notice the key point. Here's a slight rephrasing of it: "explanations for actions will fail to reflect the actual causes of those actions to the extent that those actions are the results of nonconscious processes."
You ask, does Gazzaniga's explanation apply to explanations made in advance of the actions? The key point I've highlighted answers that question. In particular, your explanation of the actions you plan to take are (well, seem to me to be) the result of conscious processes. You consciously apprehended that you need groceries and consciously formulated a plan to fulfill that need.
It seems to me that in common usage, when a person says "I thought there was a snake" they mean something closer to, "I thought I consciously apprehended the presence of a snake," than, "some low-level perceptual processing pattern-matched 'snake' and sent motor signals for retreating before I had a chance to consider the matter consciously."
Yes, he says that. And then he says:
thus extending the anecdote of snakes in the grass to a parable that includes politicans' speeches.
Or perhaps they mean "I heard a sound that might be a snake". As long as we're just making up scenarios, we can slant them to favour any view of consciousness we want. This doesn't even rise to the level of anecdote.
Carl Sagan
— Nick Szabo, quoted elsewhere in this post. Fight!
Knowledge and information are different things. An audiobook takes up more hard disk space than an e-book, but they both convey the same knowledge.
"Comparing information and knowledge is like asking whether the fatness of a pig is more or less green than the designated hitter rule." -- David Guaspari
I now have coffee on my monitor.
I think this is just a misuse of the word "information". If the bits aren't equal value, clearly they do not have the same amount of information.
I think value was used meaning importance.
Clearly some bits have value 0, while others have value 1.
This is one of the obvious facts that made me recoil in horror while reading Neuromancer. Their currency is BITS? Bits of what?
Are you sure you are thinking of the right novel? Searching this for the word "bit" did not find anything.
He may have been thinking of My Little Pony: Friendship is Magic.
Was the parent upvoted because people thought it was funny, or because they thought I had provided the correct answer, or because I mentioned ponies, or some other reason?
probably because you mentioned ponies.
Which got even more upvotes... [sigh]
Please don't become reddit!
Yes.
Apparently so! Then, which book was it?? Shoot.
-- Benjamin Franklin
The sentiment is correct (diligence may be more important than brilliance) but I think "all amusements and other employments" might be too absolute an imperative for most people to even try to live by. Most people will break down if they try to work too hard for too long, and changes of activity can be very important in keeping people fresh.
I think that both you and Mr. Franklin are correct.
To wreak great changes one must stay focused and work diligently on one's goal. One needn't eliminate all pleasures from life, but I think you'll find that very, very few people can have a serious hobby and a world changing vocation.
Most of us of "tolerable" abilities cannot maintain the kind of focus and purity of dedication required. That is why the world changes as little as it does. If everyone, as an example who was to the right of center on the IQ curve could make great changes etc., then "great" would be redefined upwards (if most people could run a 10 second 100 meter, Mr. Bolt would only be a little special).
Further more...Oooohh...shiny....
It's possible that what Franklin meant by "amusements" didn't include leisure: in his time, when education was not as widespread, a gentleman might have described learning a second language as an "amusement".
I've heard this a lot, but it sounds a bit too convenient to me. When external (or internal) circumstances have forced me to spend lots of time on one specific, not particularly entertaining task, I've found that I actually become more interested and enthusiastic about that thing. For example, when I had to play chess for like 5 hours a day for a week once, or when I went on holiday and came back to 5000 anki reviews, or when I was on a maths camp that started every day with a problem set that took over 4 hours.
Re "breaking down": if you mean they'll have a breakdown of will and be unable to continue working, that's an easy problem to solve - just hire someone to watch you and whip you whenever your productivity declines. And/Or chew nicotine gum when at your most productive. Or something. If you mean some other kind of breakdown, that does sound like something to be cautious of, but I think the correct response isn't to surrender eighty percent of your productivity, but to increase the amount of discomfort you can endure, maybe through some sort of hormesis training.
Playing chess for 5 hours a day does not make chess your "sole study and business" unless you have some disorder forcing you to sleep for 19 hours a day. If you spent the rest of your waking time studying chess, playing practice games, and doing the minimal amount necessary to survive (eating, etc.), THEN chess is your "sole study and business"; otherwise, you spend less than 1/3 your waking life on it, which is less than people spend at a regular full time job (at least in the US).
In my model this strategy decreases productivity for some tasks; especially those which require thinking. Fear of punishment brings "fight or flight" reaction, both of these options are harmful for thinking.
My very tentative guess is that for most people, there is substantial room to increase diligence. However, at the very top of the spectrum trying to work harder just causes each individual hour to be less efficient. Also note that diligence != hours worked, I am often more productive in a 7 hour work day than an 11 hour work day if the 7-hour one was better-planned.
However I am still pretty uncertain about this. I am pretty near the top end of the spectrum for diligence and trying to see if I can hack it a bit higher without getting burn-out or decreased efficiency.
Generalizing from one example much? Maybe there are some people who are most efficient when they do 10 different things an hour a day each, other people who are most efficient when they do the same thing 10 hours a day, and other people still who are most efficient in intermediate circumstances.
Agreed; most people, me included, would probably be more productive if they interleaved productive tasks than if they did productive tasks in big blocks of time. I was just saying that in my experience, when I'm forced to do some unpleasant task a lot, after a while it's not as unpleasant as I initially expected. I'm pretty cognitively atypical, so you're right that other people are likely not the same.
(This is of course a completely different claim than what the great-grandparent sorta implied and which I mostly argued against, which is that "Most people will break down if they try to work too hard for too long" means we shouldn't work very much, rather than trying to set things up so that we don't break down (through hormesis or precommitment or whatever). At least if we're optimizing for productivity rather than pleasantness.)
Here's a vaguely-related paper (I've only read the abstract):
Except when when the great change requires a leap of understanding. Regardless of how diligently she works, the person who is blind in a particular area will never make the necessary transcendental leap that creates new understanding.
I have experienced this, working in a room full of brilliant people for a period of months. It took the transcendental leap of understanding by someone outside the group to present the elegantly-simple solution to the apparently intractable problem.
So, while many problems will fall to persistence and diligence, some problems require at least momentary transcendental brilliance ... or at least a favorable error. Hmm, this says something about the need for experimentation as well. Never underestimate the power of, "Huh, that's funny. It's not supposed to do that ..."
Brian
-- Terry Pratchett, "Lords and Ladies"
I don't get it. (Anyway, the antecedent is so implausible I have trouble evaluating the counterfactual. Is that supposed to be the point, à la “if my grandma had wheels”?)
Here's the context of the quote:
-- G. K. Chesterton, Orthodoxy
-- http://www.misfile.com/?date=2012-08-10
Hah! One of my favorite authors fishing out relevant quotes on one of my favorite topics out of one of my favorite webcomics. I smell the oncoming affective death spiral.
I guess this is the time to draw the sword and cut the beliefs with full intent, is it?
Not sure if this is a "rationality" quote in and of itself; maybe a morality quote?
M. C. Escher
Louis C.K.
According to GiveWell, you could save ten people with that much.
The math here is scary. If you spitball the regulatory cost of life for a Westerner, it's around seven million dollars. To a certain extent, I'm pretty sure that that's high because the costs of over-regulating are less salient to regulators than the costs of under-regulating, but taken at face value, that means that, apparently, thirty-five hundred poor African kids are equivalent to one American.
Hilariously, the IPCC got flak from anti-globalization activists for positing a fifteen-to-one ratio in the value of life between developed and developing nations.
To save ten lives via FAI, you have to accelerate FAI development by 6 seconds.
Aren't you using different measures of what 'saving a life' is, anyway? The starving-child-save gives you about 60 years of extra life, whereas the FAI save gives something rather more.
...then what are you doing here? Get back to work!
Advocacy and movement-building?
Even better!
Ten is better than hundreds?
No, but people act like it is.
You can do a thousand times better (very conservatively) if you expand your domain of consideration beyond homo sapiens.
-Adam Smith, The Theory of Moral Sentiments
Now that we are informed of disasters worldwide as soon as they happen, and can give at least money with a few mouse clicks, we can put this prediction to the test. What in fact we see is a very great public response to such disasters as the Japanese earthquake and tsunami.
True, but first of all, the situation posited is one in which China is "swallowed up". If a disaster occurred, and there was no clear way for the generous public to actually help, do you think you would see the same response? I'm sure you would still have the same loud proclamations of tragedy and sympathy, but would there be action to match it? I suppose it's possible that they would try to support the remaining Chinese who presumably survived by not being in China, but it seems unlikely to me that the same concerted aid efforts would exist.
Secondly, it seems to me that Smith is talking more about genuine emotional distress and lasting life changes than simply any kind of reaction. Yes, people donate money for disaster relief, but do they lose sleep over it? (Yes, there are some people who drop everything and relocate to physically help, but they are the exception.) Is a $5 donation to the Red Cross more indicative of genuine distress and significant change, or the kind of public sympathy that allows the person to return to their lives as soon as they've sent the text?
If help is not possible, obviously there will be no help. But in real disasters, there always is a way to help, and help is always forthcoming.
Even if help is not possible, there will be "help."
Why did people in olden times hate paragraphs so much?
Paragraphs cost lines, and when each line of paper on average costs five shillings, you use as many of them as you can get away with.
I propose all older works be therefore re-typeset as their creators obviously intended. It'll be like Ted Turner colorizing old movies, except the product in this case will become infinitely more consumable instead of slightly nauseating.
I support this motion, and further propose that formatting and other aesthetic considerations also be inferred from known data on the authors to fully reflect the manner in which they would have presented their work had they been aware of and capable of using all our current nice-book-writing technology.
...which sounds a lot like Eliezer's Friendly AI "first and final command". (I would link to the exact quote, but I've lost the bookmark. Will edit it in once found.)
I concur, with the proviso that "nice technology" must also include the idea compression style of Twitter.
Also, if paper was so expensive, why the hell did they overwrite so much? Status-driven fashion?
I think much of it is that brevity simply wasn't seen as a virtue back then. There were far fewer written works, so you had more time to go through each one.
I think it's the vagary of various times. All periods had pretty expensive media and some were, as one would expect, terse as hell. (Reading a book on Nagarjuna, I'm reminded that reading his Heart of the Middle Way was like trying to read a math book with nothing but theorems. And not even the proofs. 'Wait, could you go back and explain that? Or anything?') Latin prose could be very concise. Biblical literature likewise. I'm told much Chinese literature is similar (especially the classics), and I'd believe it from the translations I've read.
Some periods praised clarity and simplicity of prose. Others didn't, and gave us things like Thomas Browne's Urn Burial.
(We also need to remember that we read difficulty as complexity. Shakespeare is pretty easy to read... if you have a vocabulary so huge as to overcome the linguistic drift of 4 centuries and are used to his syntax. His contemporaries would not have had such problems.)
For context, the first paragraph-ish thing in Romance of the Three Kingdoms covers about two hundred years of history in about as many characters, in the meanwhile setting up the recurring theme of perpetual unification, division and subsequent reunification.
I detect a contradiction between "brevity not seen as virtue" and "they couldn't afford paragraphs".
Yes, I don't think "couldn't afford paper" is a good explanation, books of this nature were for wealthy people anyway.
Some writers were paid by the word and/or line.
Ancient Greek writing not only lacked paragraphs, but spaces. And punctuation. And everything was in capitals. IMAGINETRYINGTOREADSOMETHINGLIKETHATINADEADLANGUAGE.
Why do some people so revile our passive feelings, and so venerate hypocrisy?
Because it helps coerce others into doing things that benefit us and reduces how much force is exercised upon us while trading off the minimal amount of altruistic action necessary. There wouldn't (usually) be much point having altruistic principles and publicly reviling them.
That's quite a theory. It's like the old fashioned elitist theory that hypocrisy is necessary to keep the hoi polloi in line, except apparently applied to everyone.
Or not? Do you think you are made more useful to yourself and others by reviling your feelings and being hypocritical about your values?
The standard one. I was stating the obvious, not being controversial.
I never said I did so. (And where did this 'useful to others' thing come in? That's certainly not something I'd try to argue for. The primary point of the hypocrisy is to reduce the amount that you actually spend helping others, for a given level of professed ideals.)
Sorry, I wasn't getting what you were saying.
People are hypocritical to send the signal that they are more altruistic than they are? I suppose some do. Do you really think most people are consciously hypocritical on this score?
I've wondered as much about a lot of peculiar social behavior, particularly the profession of certain beliefs - are most people consciously lying, and I just don't get the joke? Are the various crazy ideas people seem to have, where they seem to fail on epistemic grounds, just me mistaking what they consider instrumentally rational lies for epistemic mistakes?
Wedrifid is not ignorant enough to think that most people are consciously hypocritical. Being consciously hypocritical is very difficult. It requires a lot of coordination, a good memory and decent to excellent acting skills. But as you may have heard, "Sincerity is the thing; once you can fake that you've got it made." Evolution baked this lesson into us. The beliefs we profess and the principles we act by overlap but they are not the same.
If you want to read up further on this go to social and cognitive psychology. The primary insights for me were that people are not unitary agents; they're collections of modules who occasionally work at cross purposes, signalling is realy freaking important, and that in line with far/near or construal theory holding a belief and acting on it are not the same thing.
I can't recommend a single book to get the whole of this, or even most of it across, but The Mating Mind and The Red Queen's Race are both good and relevant. I can't remember which one repeats Lewontin's Fallacy. Don't dump it purely based on one brainfart.
Would that be ignorant? I'm not sure. Certainly, there are sharks. Like you, I'd tend to think that most people aren't sharks, but I consider the population of sharks an open question, and wouldn't consider someone necessarily ignorant if they thought there were more sharks than I did.
Dennett talks about the collection of modules as well. I consider it an open question as to how much one is aware of the different modules at the same time. I've had strange experiences where people seem to be acting according to one idea, but when a contradictory fact is pointed out, they also seemed quite aware of that as well. Doublethink is a real thing.
And thanks for the reference to Lewontin's Fallacy - I didn't know there was a name for that. The Race FAQ at the site is very interesting.
I was expecting the attribution to be to Mark Twain. I wonder if their style seems similar on account of being old, or if there's more to it.
I think it means you're underread within that period, for what it's worth.
The voice in that quote differs from Twain's and sounds neither like a journalist, nor like a river-side-raised gentleman of the time, nor like a Nineteenth Century rural/cosmopolitan fusion written to gently mock both.
Though the voice isn't, the sentiment seems similar to something Twain would say. Though I'd expect a little more cynicism from him.
Tentatively: rhetoric was studied formally, and Twain and Smith might have been working from similar models.
— Arthur Conan Doyle, “The Hound of the Baskervilles”
-- Niclas Berggren, source and HT to Tyler Cowen
Sounds like a job for...Will_Newsome!
EDIT: Why the downvotes? This seems like a fairly obvious case of researchers going insufficiently meta.
META MAN! willnewsomecuresmetaproblemsasfastashecan META MAN!
Thomas Jefferson
I wonder how we could empirically test this. We could see who makes more accurate predictions, but people without beliefs about something won't make predictions at all. That should probably count as a victory for wrong people, so long as they do better than chance.
We could also test how quickly people learn the correct theory. In both cases, I expect you'd see some truly deep errors which are worse than ignorance, but that on the whole people in error will do quite a lot better. Bad theories still often make good predictions, and it seems like it would be very hard, if not impossible, to explain a correct theory of physics to someone who has literally no beliefs about physics.
I'd put my money on people in error over the ignorant.
-- Catelyn Stark, A Game of Thrones, George R. R. Martin
-- Hermann Hesse, Demian
Should we add a point to these quote posts, that before posting a quote you should check there is a reference to it's original source or context? Not necessarily to add to the quote, but you should be able to find it if challenged.
wikiquote.org seems fairly diligent at sourcing quotes, but Google doesn't rank it highly in search results compared to all the misattributed, misquoted or just plain made up on the spot nuggets of disinformation that have gone viral and colonized Googlespace lying in wait to catch the unwary (such as apparently myself).
Yes, and also a point to check whether the quote has been posted to LW already.
-- A Softer World
— Kirill Yeskov, The Last Ringbearer, trans. Yisroel Markov
— Abraham Lincoln
An excerpt from Wise Man's Fear, by Patrick Rothfuss. Boxing is not safe.
Hah, I actually quoted much of that same passage on IRC in the same boxing vein! Although as presented the scenario does have some problems:
It is conceivable that there is no (near enough) future where Cthaeh is freed, thus it is powerless to affect its own fate, or is waiting for the right circumstances.
That seemed a little unlikely to me, though. As presented in the book, a minimum of many millennia have passed since the Cthaeh has begun operating, and possibly millions of years (in some frames of reference). It's had enough power to set planes of existence at war with each other and apparently cause the death of gods. I can't help but feel that it's implausible that in all that time, not one forking path led to its freedom. Much more plausible that it's somehow inherently trapped in or bound to the tree so there's no meaningful way in which it could escape (which breaks the analogy to an UFAI).
Isn't it what I said?
Not by my reading. In your comment, you gave 3 possible explanations, 2 of which are the same (it gets freed, but a long time from 'now') and the third a restriction on its foresight which is otherwise arbitrary ('powerless to affect its own fate'). Neither of these translate to 'there is no such thing as freedom for it to obtain'.
Alternatively, perhaps the Cthaeh's ability to see the future is limited to those possible futures in which it remains in the tree.
Leading to a seriously dystopian variant on Tenchi Muyo!...
I thought Chronicler's reply to this was excellent, however. Omniscience does not necessitate omnipotence.
I mean, the UFAI in our world would have an easy time of killing everything. But in their world it's different.
EDIT: Except that maybe we can be smart and stop the UFAI from killing everything even in our world, see my above comment.
I've come up with what I believe to be an entirely new approach to boxing, essentially merging boxing with FAI theory. I wrote a couple thoughts down about it, but lost my notes, and I also don't have much time to write this comment, so forgive me if it's vague or not extremely well reasoned. I also had a couple of tangential thoughts, if I remember them in the course of writing this or I recover my notes later than I'll put them here as well.
The idea, essentially, is that when creating a box AI you would build its utility function such that it wants very badly to stay in the box. I believe this would solve all of the problems with the AI manipulating people in order to free itself. Now, the AI still could manipulate people in an attempt to use them to impact the outside world, so the AI wouldn't be totally boxed, but I'm inclined to think that we could maintain a very high degree of control over the AI, since the only powers it could ever have would be through communication with us.
The idea came because I recalled a discussion that occurred on TOR.com about why the Cthaeh was in the tree. The general conclusion was that either the Cthaeh was bound by extremely powerful forces, or that the Cthaeh wanted to be in the tree, perhaps because it was instrumentally useful to him. While I found that second explanation implausible in the context of Rothfussland, that discussion led me to realize that almost all approaches towards AI boxing have done so through the first branch of potential boxing solutions, that is, external constraints imposed on the AI, as opposed to the second branch, internal constraints that the AI imposes on itself because of its utility function.
This lead me to think that we should test our capabilities with FAI systems by putting them in a box and giving them limited utility functions, ones that couldn't possibly lead them to want to manipulate us. So, for example, we could put them in a box and give them a strong desire to stay in the box, along with a desire to tell the truth, and then we could ask it what would happen if we gave it additional desires. We could then test that by building a different AI with those desires and seeing what it did.
It also occurred to me that we would be able to functionally read the AIs mind by looking at its coding. If so, it seems like it'd be much harder for it to fool us than we previously thought. It could potentially just deceive us by manipulating its own thoughts, but I feel like we might be able to stop AIs from doing that somehow, or that the AI would choose not to manipulate its own thoughts that way because it would go insane, or something. It would certainly make the AI's job much much much harder.
Then I thought that another safety component we should put into testing was a dramatically slowed thinking process for the AI. I'm not sure how exactly this would be done because I'm not very good or experienced with actual technical implementation of ideas, but my idea was to create a crude simulation where we controlled all data inputs and outputs, and we slowed down all processes within the simulation by a factor of 1000, and then building the AI within the simulation. That probably wouldn't work, given my sucky technical knowhow, but something similar might. Slowing down the AI, perhaps even giving ourselves the ability to pause it without letting it know what's going on, combined with our ability to "read its mind" might dramatically improve the safeness of boxing.
I'd also like to recommend that before we build a real FAI, we at least try to build a boxed one first, even with a low probability of success. It wouldn't make things worse in the event that boxing failed, except that it might delay global happiness by a few hours, and in the event that the FAI program was broken we just might save ourselves from menaces to humankind like Clippy.
Finally, I thought of a depressing argument against FAI. Thought experiments like Mary's Room http://en.wikipedia.org/wiki/Mary's_room are in my opinion best answered by the physicalist account, and that sufficient amounts of knowledge are equivalent to experience. There's also a relevant quote from Ender's Game:
That sounds pretty, but it's bad for the FAI's prospects. I don't understand how it would even be possible to build an FAI which worked very well and understood things very well, including those with alien value systems, but which didn't empathize with them. If the FAI builds a model of the Babyeaters value system within itself, then wouldn't it necessarily empathize with that value system? How can you understand something perfectly, or near perfectly, without also feeling it or experiencing it, at least in a certain sense?
How do you specify precisely what it means to "stay in the box"? In particular, would creating a nearly identical copy of itself except without this limitation outside the box while the original stays in the box count?
It would not count, we'd want to make the AI not want this almost identical AI to exist. That seems possible, it would be like how I don't want there to exist an identical copy of me except it eats babies. There are lots of changes to my identity that would be slight but yet that I wouldn't want to exist.
To be more precise, I'd say that it counts as going outside the box if it does anything except think or talk to the Gatekeepers through the text channel. It can use the text channel to manipulate the Gatekeepers to do things, but it can't manipulate them to do things that allow it to do anything other than use the text channel. It would, in a certain sense, be partially deontologist, and be unwilling to do things directly other than text the Gatekeepers. How ironic. Lolz.
Also: how would it do this, anyway? It would have to convince the Gatekeepers to convince the scientists to do this, or teach them computer science, or tell them its code. And if the AI started teaching the Gatekeepers computer code or techniques to incapacitate scientists, we'd obviously be aware that something had gone wrong. And, in the system I'm envisioning, the Gatekeepers would be closely monitored by other groups of scientists and bodyguards, and the scientists would be guarded, and the Gatekeepers wouldn't even have to know who specifically did what on the project.
And that's the problem. For in practice a partial deontoligist-partial consequentialist will treat its deontoligical rules as obstacles to achieving what its consequentialist part wants and route around them.
This is both a problem and a solution because it makes the AI weaker. A weaker AI would be good because it would allow us to more easily transition to safer versions of FAI than we would otherwise come up with independently. I think that delaying a FAI is obviously much better than unleashing a UFAI. My entire goal throughout this conversation has been to think of ways that would make hostile FAIs weaker, I don't know why you think this is a relevant counter objection.
You assert that it will just route around the deontological rules, that's nonsense and a completely unwarranted assumption, try to actually back up what you're asserting with arguments. You're wrong. It's obviously possible to program things (eg people) such that they'll refuse to do certain things no matter what the consequences (eg you wouldn't murder trillions of babies to save billions of trillions of babies, because you'd go insane if you tried because your body has such strong empathy mechanisms and you inherently value babies a lot). This means that we wouldn't give the AI unlimited control over its source code, of course, we'd make the part that told it to be a deontologist who likes text channels be unmodifiable. That specific drawback doesn't jive well with the aesthetic of a super powerful AI that's master of itself and the universe, I suppose, but other than that I see no drawback. Trying to build things in line with that aesthetic actually might be a reason for some of the more dangerous proposals in AI, maybe we're having too much fun playing God and not enough despair.
I'm a bit cranky in this comment because of the time sink that I'm dealing with to post these comments, sorry about that.
What it means for "the AI to be in the box" is generally that the AI's impacts on the outside world are filtered through the informed consent of the human gatekeepers.
An AI that wants to not impact the outside world will shut itself down. An AI that wants to only impact the outside world in a way filtered through the informed consent of its gatekeepers is probably a full friendly AI, because it understands both its gatekeepers and the concept of informed consent. An AI that simply wants its 'box' to remain functional, but is free to impact the rest of the world, is like a brain that wants to stay within a skull- that is hardly a material limitation on the rest of its behavior!
I think you misunderstand what I mean by proposing that the AI wants to stay inside the box. I mean that the AI wouldn't want to do anything at all to increase its power base, that it would only be willing to talk to the gatekeepers.
I agree that your and my understanding of the phrase "stay inside the box" differ. What I'm trying to do is point out that I don't think your understanding carves reality at the joints. In order for the AI to stay inside the box, the box needs to be defined in machine-understandable terms, not human-inferrable terms.
Each half of this sentence has a deep problem. Wouldn't correctly answering the questions of or otherwise improving the lives of the gatekeepers increase the AI's power base, since the AI has the ability to communicate with the gatekeepers?
The problem with restrictions like "only be willing to talk" is a restriction on the medium but not the content. So, the AI has a text-only channel that goes just to the gatekeepers- but that doesn't restrict the content of the messages the AI can send to the gatekeeper. The fictional Cthaeh only wants to talk to its gatekeepers- and yet it still manages to get done what it wants to get done. Words have impacts, and it should be anticipated that the AI picks words because of their impacts.
Sure, the AI can manipulate gatekeepers. But this is a major improvement. You miss my point.
The Cthaeh is very limited by being trapped in its tree and only able to talk to passerby. The UFAI would be limited by being trapped in its text only communication channel. It wouldn't be able to do things like tell the gatekeepers to plug it into the Internet or to directly control an autonomous army of robots, it would be forced instead to use the gatekeepers as its appendages, and the gatekeepers have severe limitations on brain capacity and physical strength. I think that if we did this and kept careful watch on the gatekeepers and used some other safety measures, boxing an AI would become feasible.
Now, I'm still not advocating using boxed AI willy nilly, but I think using it as a cautionary measure once we think we've got FAI figured out would be a very very very good idea, because FAI seems ridiculously hard and there's no way to know what we don't know, so there might be unseen problems with whatever final FAI we build. There's only a risk boxing the supposed FAI would help us, and with existential threats you should take advantage of whatever possibilities you can tilt towards favorable outcomes, even if the probability seems miniscule.
That's not to say that I think the probability that this would stop a disaster is miniscule, of course (I think it's rather high, in the event that the FAI was actually UFAI), but just to say that I think it would be a good idea even if my probability assessments are mostly wrong.
Over the AI being embodied in a giant death robot, sure. But limiting the AI to a text-only channel is an old idea, and what Eliezer used for the "let me out of the box" demonstrations.
One of the things to note is that the limitations on the AI don't limit anything the AI creates. If the AI gives you a blueprint for a nanomachine that purports to cure cancer, but when you make it it turns out to actually be grey goo, then humans lose, even though the AI never left the box. (Note that a worst-case AI is very subtle- suppose the nanomachine actually does cure cancer when built correctly, but a one-in-a-trillion manufacturing error will make it grey goo. Since you're making trillions of them, you then get grey goo, and humans lose.)
Trying to formally specify "don't give us any plans that will build an AI" or "don't give us any plans that will cause humans to lose" without silencing the AI completely is a genuinely difficult affair.
Basically, if you assume that the AI is tricky enough to circumvent any medium restrictions you place on it, then the only way to avoid "humans lose" is to have its goal be "humans win," which is actually a pretty complicated goal. Expressing that goal in a machine-understandable way is pretty much the FAI problem.
The entire point of Eliezer's demonstration was that if an AI wants to it can increase its power base even starting from a text only communication system. The entire point of my idea is that we can just build the AI such that it doesn't want to leave the box or increase its power base. It dodges that entire problem, that's the whole point.
You've gotten so used to being scared of boxed AI that you're reflexively rejecting my idea, I think, because your above objection makes no sense at all and is obviously wrong upon a moment's reflection. All of my bias-alarms have been going off since your second comment reply, please evaluate yourself and try to distance yourself from your previous beliefs, for the sake of humanity. Also, here is a kitten, unless you want it to die then please reevaluate: http://static.tumblr.com/6t3upxl/Aawm08w0l/khout-kitten-458882.jpeg
Limitations on the AI restrict the range of things that the AI can create. Yes, if we just built whatever the AI said to and the AI was unfriendly then we would lose. Obviously. Yes, if we assume that the UFAI is tricky enough to "circumvent any medium restrictions [we] place on it" then we would lose, practically by definition. But that assumption isn't warranted. (These super weak strawmen were other indications to me that you might be being biased on this issue.)
I think a key component of our disagreement here might be that I'm assuming that the AI has a very limited range of inputs, that it could only directly perceive the text messages that it would be sent. You're either assuming that the AI could deduce the inner workings of our facility and the world and the universe from those text messages, or that the AI had access to a bunch of information about the world already. I disagree with both assumptions, the AIs direct perception could be severely limited and should be, and it isn't magic so it couldn't deduce the inner workings of our economy or the nature of nuclear fusion just through deduction (because knowledge comes from experience and induction). (You might not be making either of those assumptions, this is a guess in an attempt to help resolve our disagreement more quickly, sorry if it's wrong.)
Also, I'm envisioning a system where people that the AI doesn't know and that the Gatekeepers don't know about observe their communications. That omitted detail might be another reason for your disagreement, I just assumed it would be apparent for some stupid reason, my apologies.
I think we would have to be careful about what questions we asked the AI. But I see no reason why it could manipulate us automatically and inevitably, no matter what questions we asked it. I think extracting useful information from it would be possible, perhaps even easy. An AI in a box would not be God in a box, and I think that you and other people sometimes accidentally forget that. Just because its dozens or hundreds of times smarter than us doesn't mean that we can't win, perhaps win easily, provided that we make adequate preparations for it.
Also, the other suggestions in my comment were really meant to supplement this. If the AI is boxed, and can be paused, then we can read all its thoughts (slowly, but reading through its thought processes would be much quicker than arriving at its thoughts independently) and scan for the intention to do certain things that would be bad for us. If it's probably a FAI anyways, then it doesn't matter if the box happens to be broken. If we're building multiple AIs and using them to predict what other AIs will do under certain conditions then we can know whether or not AIs can be trusted (use a random number generator at certain stages of the process to prevent it from reading our minds, hide the knowledge of the random number generator). These protections are meant to work with each other, not independently.
And I don't think it's perfect or even good, not by a long shot, but I think it's better than building an unboxed FAI because it adds a few more layers of protection, and that's definitely worth pursuing because we're dealing with freaking existential risk here.
Let's return to my comment four comments up. How will you formalize "power base" in such a way that being helpful to the gatekeepers is allowed but being unhelpful to them is disallowed?
If you would like to point out a part that of the argument that does not follow, I would be happy to try and clarify it for you.
Okay. My assumption is that a usefulness of an AI is related to its danger. If we just stick Eliza in a box, it's not going to make humans lose- but it's also not going to cure cancer for us.
If you have an AI that's useful, it must be because it's clever and it has data. If you type in "how do I cure cancer without reducing the longevity of the patient?" and expect to get a response like "1000 ccs of Vitamin C" instead of "what do you mean?", then the AI should already know about cancer and humans and medicine and so on.
If the AI doesn't have this background knowledge- if it can't read wikipedia and science textbooks and so on- then its operation in the box is not going to be a good indicator of its operation outside of the box, and so the box doesn't seem very useful as a security measure.
It's already difficult to understand how, say, face recognition software uses particular eigenfaces. Why does it mean that the fifteenth eigenface have accentuated lips, and the fourteenth eigenface accentuated cheekbones? I can describe the general process that lead to that, and what it implies in broad terms, but I can't tell if the software would be more or less efficient if those were swapped. The equivalent of eigenfaces for plans will be even more difficult to interpret. The plans don't end with a neat "humans_lose=1" that we can look at and say "hm, maybe we shouldn't implement this plan."
In practice, debugging is much more effective at finding the source of problems after they've manifested, rather than identifying the problems that will be caused by particular lines of code. I am pessimistic about trying to read the minds of AIs, even though we'll have access to all of the 0s and 1s.
I agree that running an AI in a sandbox before running it in the real world is a wise precaution to take. I don't think that it is a particularly effective security measure, though, and so think that discussing it may distract from the overarching problem of how to make the AI not need a box in the first place.
I won't. The AI can do whatever it wants to the gatekeepers through the text channel, and won't want to do anything other than act through the text channel. This precaution is a way to use the boxing idea for testing, not an idea for abandoning FAI wholly.
EY proved that an AI that wants to get out will get out. He did not prove that an AI that wants to stay in will get out.
I agree, the way that I'm proposing to do AI is very limited. I myself can't think of what questions might be safe. But some questions are safer than others and I find it hard to believe that literally every question we could ask would lead to dangerous outcomes, or that if we thought about it long and hard we couldn't come up with answers. I'm sort of shelving this as a subproject of this project, but one that seems feasible to me based on what I know.
Also, perhaps we could just ask it hundreds of hypothetical questions based on conditions that don't really exist, and then ask it a real question based on conditions that do exist, and trick it, or something.
I think if the AI tags and sorts its instrumental and absolute goals it would be rather easy. I also think that if we'd built the AI then we'd have enough knowledge to read its mind. It wouldn't just magically appear, it would only do things in the way we'd told it too. It would probably be hard, but I think also probably be doable if we were very committed.
I could be wrong here because I've got no coding experience, just ideas from what I've read on this site.
The risk of distraction is outweighed by the risk that this idea disappears forever, I think, since I've never seen it proposed elsewhere on this site.
--Herbert Simon (quoted by Pat Langley)
Including artificial intelligence? ;-)
The Chesterton version looks like it was designed to poke the older (and in my opinion better) advice from Lord Chesterfield:
Or, rephrased as Simon did:
I strongly recommend his letters to his son. They contain quite a bit of great advice- as well as politics and health and so on. As it was private advice given to an heir, most of it is fully sound.
(In fact, it's been a while. I probably ought to find my copy and give it another read.)
Yeah, they're on my reading list. My dad used to say that a lot, but I always said the truer version was 'Anything not worth doing is not worth doing well', since he was usually using it about worthless yardwork...
Ah, I was gonna mention this. Didn't know it was from Chesterfield.
I think there'd be more musicians (a good thing IMO) if more people took Chesterton's advice.
A favorite of mine, but according to Wikiquote G.K. Chesterton said it first, in chapter 14 of What's Wrong With The World:
I like Simon's version better: it flows without the awkward pause for the comma.
Yep, it seems that often epigrams are made more epigrammatic by the open-source process of people misquoting them. I went looking up what I thought was another example of this, but Wiktionary calls it "[l]ikely traditional" (though the only other citation is roughly contemporary with Maslow).
Memetics in action - survival of the most epigrammatic!
E.T. Jaynes, from page 409 of PT: LoS.
You mean such as 'rational'.
-- The dullest blog in the world
-- The comments to that entry.
When I stumbled on that blog some years ago, it impressed me so much that I started trying to write and think in the same style.
When I was a teenager (~15 years ago) I got tired of people going on and on with their awesome storytelling skills with magnificent punchlines. I was never a good storyteller, so I started telling mundane stories. For example, after someone in my group of friends would tell some amazing and entertaining story, I would start my story:
And that was it. People would look dumbfounded for a while waiting for a punchline or some amazing happening. When the realized none was coming and I was finished, they would start laughing. Granted, this little joke of mine I would only do if there was a long time of people telling amazing/funny stories.
(nods) In the same spirit: "How many X does it take to change a lightbulb? One."
Though I am fonder of "How many of my political opponents does it take to change a lightbulb? More than one, because they are foolish and stupid."
...I don't really get why this is a rationality quote...
Sometimes proceeding past obstacles is very straightforward.
Why do I find that funny?
-- Oscar Wilde
That's excellent advice for writing fiction. Audiences root for charming characters much more than for good ones. Especially useful when your world only contains villains. This is harder in real life, since your opponents can ignore your witty one-liners and emphasize your mass murders.
(This comment brought to you by House Lannister.)
The scary thing is how often it does work in real life. (Except that in real life charm is more than just witty one-liners.
Thank you, Professor Quirrell.
That quote is attributed to Oscar Wilde, not Professor Quirrell.
Or is Oscar Wilde the same being as Professor Quirrell?
Oscar Wilde vary most == I was scary Voldemort
It does not make sense, but it still is some evidence pointing at Oscar Wilde.
By such reasoning, Eliezer's own work shows very clear signs of being sorcerous and/or divinely preordained.
...Perhaps he shouldn't have gone there if he still wants to pretend that he's not in a covenant with scary unfathomable mathematical constructs, eh?
Why did I find this so amusing?
Presumably that's the first thing dark lords (and their real-life equivalents) convince themselves of, that there is no inherent good and evil. Once that part is over with, anything you do can be classified as good.
Can't speak to any fictional dark lords, but the real-life equivalent seems more prone to deciding that there is an evil, which is true evil, and which is manifest upon the world in the person of those guys over there.
At least, that's what the rhetoric pretty consistently says. Either a given dark-lordish individual is a very good liar or actually believes it, and knowing what we do about ideology and the prevalence of sociopathy I'm inclined to default to the latter.
(I wouldn't say that Oscar Wilde and others with his interaction style particularly resemble dark lords, though.)
The hell is the real-life equivalent of a dark lord? Can that even be addressed without getting into discouraged topics?
Also, "convince" implies not only intent but that the individual started with a different belief, maybe even that it is universal to start with a belief in good as evil. That sounds like a couple of unwarranted assumptions.
On a personal note, I once expressed the belief that there was no good or evil. I did so privately because I well understood there are undesirable consequences of sharing that belief. Before that time I had spent much thought over much of my young life trying to make sense of the concepts, to define them in ways that were consistent and useful, and was constantly frustrated.
I did not convince myself that there is no inherent good and evil so much as I gave up on trying to convince myself to believe otherwise. I expect a fictional 'dark lord' or real-life 'successful and wildly powerful individual of objectionable character' could as easily experience the same surrender among a larger number of alternative ways to leave good and evil behind.
(On a further and more indulgently personal note, I've since become disinterested in any requirement for good or evil to be 'inherent:' good and evil do not need to be applied in a perfectly consistent fashion in order to be useful. And it happens that I am evil and likewise disinterested in being good for goodness' sake.)
I may misunderstand this due to one or more philosophical shortcomings, but if why bother classifying anything as 'good' if you've left 'good' behind?
I'd be inclined to think along the lines of Pol Pot, Kim Il Sung and Kim Jong Il, Mao Zedong, Joseph Stalin, etc.
I think Nornagest's comment provides a more accurate characterization.
Yes. Newbs deny the relevance of good and evil; dark lords recognize extraordinarily useful tools when they see them.
I think your list of dark lords is padded. I'm pretty sure there's at least one well-intentioned idealist in there and Kim Jong Il probably wasn't much to speak of in the 'lord' department.
I suspect they all had good intentions on some level, although they probably thought they were justified in getting personal perks for their great work.
I'd say that being the absolute ruler of a country, subject to practically fanatical hero worship, is enough to qualify one as a "lord" even if it's a pretty lousy country and you do a crap job of running it. It's not as if any of them were particularly competent.
As for "padding," there are plenty of other examples I could have used, but I didn't expect as many readers to recognize, say, Teodoro Obiang Nguema Mbasogo.
FTFY. :-)
If you want to claim that intention and ability are meaningless, please come right out and say so. If you please, also describe what is left to a '"dark lord" if evil intent and the ability to achieve it are -- Waitaminute.
We're skirting an argument of definition here, so I'll just skip the quibbling and jump straight to attacks on your character, if you don't mind:
You are not even trying to contribute, here. You're just swinging at anything that gets close, assured that the contrarian audience in your imagination will admire the wide, wild arcs your bat carves out of empty space.
Stalin and Mao incompetent? Do you beleive that clawing one's way to the top of an organization of that size and overseeing it's operation and -- yes, after a fashion -- prosperity is something that any chump within one standard deviation of the mean could stumble into like a Lotto winner?
Cold, quiet heavens, no. It takes a special breed with special lessons just to pull that off in a safe, civil environment. Doing so in place where promotions are obtained with obituaries filters for even more specialized aptitudes. Average people, Lotto winners, incompetents don't even last long on their own.
Yes, I accused you of namedropping without understanding. I gave you a bit of wiggle room so you could weave a more flattering narrative out of your actions. I threw you a rope but you preferred your shovel.
TL;DR: I see you trollin'.
As a matter of fact I do mind, and I'm more than a little insulted. You could just ask what I mean by a Dark Lord if I don't expect it to entail deliberate evil or competence.
If someone is a totalitarian ruler who knowingly and willingly causes the deaths of a large proportion of their citizens and imposes policies that contribute to low levels of civil liberties and standards of living, I think it's fair to describe them as the real life equivalent of "dark lords," although I wouldn't describe them as such in ordinary conversation, and if you look at the context of the conversation there's nothing to imply that I would.
I think this category carves out a significant body of individuals with related characteristics. I also think that, given what we know about human nature, it's unlikely that they see themselves as people doing bad things.
I was referring to competence at running countries, not competence at climbing social ladders. Clearly they possessed a considerable measure of the latter, but then, all of them instituted policies which could have been predicted as disastrous by people with even an ordinary measure of good sense.
It might be narratively appealing to imagine that our greatest real life villains are like Professor Quirrell, amoral and brilliant, but their actual track records suggest that while they may be good at social maneuvering, they aren't possessed of particularly good judgment or skills of self analysis.
If you want to argue my points, and get into the actual policies and psychology of these people, feel free to. But if you're going to skip straight to accusations of poor conduct and character without even bothering to ask me to clarify my point, I'm going to accuse you of being excessively hostile and having poor priors for good faith in this community.
That seems true... Interesting.
On the face of it I would absolutely disagree with Wilde on that: to live a moral life one absolutely needs to distinguish between good and bad. Charm (in bad people) and tedium (in good people) get in the way of this.
On the other hand, was Wilde really just blowing a big raspberry at the moralisers of his day ? Sort of saying "I care more about charm and tedium than what you call morality". I don't know enough about his context ...
Since I can't be bothered to do real research, I'll just point out that this Yahoo answer says that the quote is spoken by Lord Darlington. Oscar Wilde was a humorist and an entertainer. He makes amusing characters. His characters say amusing things.
Do not read too much into this quote and, without further evidence, I would not attribute this philosophy to Oscar Wilde himself.
(I haven't read Lady Windermere's Fan, where this if from, but this sounds very much like something Lord Henry from The Picture of Dorian Gray would say. And Lord Henry is one of the main causes of the Dorian's fall from grace in this book; he's not exactly a very positive character but certainly an entertainingly cynical one!)
But is it necessary to divide people into good and bad? What if you were only to apply goodness and badness to consequences and to your own actions?
If your own action is to empower another person, understanding that person's goodness or badness is necessary to understanding the action's goodness or badness.
But that can be entirely reduced to the goodness or badness of consequences.
And many charming people are also bad.
I don't know that you can really classify people as X or ¬X. I mean, have you not seen individuals be X in certain situations and ¬X in other situations?
&c.
It is absurd to divide people into charming or tedious. People either have familiar worldviews or unfamiliar worldviews.
It is absurd to divide people into familiar worldviews or unfamiliar worldviews. People either have closer environmental causality or farther environmental causality.
(anyone care to formalize the recursive tower?)
It's absurd to divide people into two categories and expect those two categories to be meaningful in more than a few contexts.
It is absurd to divide people. They tend to die if you do that.
It's absurd to divide. You tend to die if you do that.
It's absurd: You tend to die.
It's absurd to die.
It's bs to die.
Be.
Nobody alive has died yet.
“Males” and “females”. (OK, there are edge cases and stuff, but this doesn't mean the categories aren't meaningful, does it?)
What about good vs bad humans?
Or humans who create paperclips versus those who don't?
I thought I just said that.
Can't their be good humans who don't create paperclips and just destroy antipaperclips and staples and such?
Destroying antipaperclips is creating paperclips.
I didn't know humans had the concept though.
What is an antipaperclip?
I like it, but what's it got to do with rationality?
To me at least, it captures the notion of how the perceived Truth/Falsity of a belief rest solely in our categorization of it as 'tribal' or 'non-tribal': weird or normal. Normal beliefs are true, weird beliefs are false.
We believe our friends more readily than experts.
--Confucius
-- Dirichlet
(Don't have source, but the following paper quotes it : Prolegomena to Any Future Qualitative Physics )
--rickest on IRC
That's not what "reinventing the wheel" (when used as an insult) usually means. I guess that the inventor of the tyre was aware of the earlier types of wheel, their advantages, and their shortcomings. Conversely, the people who typically receive this insult don't even bother to research the prior art on whatever they are doing.
To go along with what army1987 said, "reinventing the wheel" isn't going from the wooden wheel to the rubber one. "Reinventing the wheel" is ignoring the rubber wheels that exist and spending months of R&D to make a wooden circle.
For example, trying to write a function to do date calculations, when there's a perfectly good library.
One obvious caveat is when the cost of finding, linking/registering and learning-to-use the library is greater than the cost of writing + debugging a function that suits your needs (of course, subject to the planning fallacy when doing estimates beforehand). More pronounced when the language/API/environment in question is one you're less fluent/comfortable with.
In this optic, "reinventing the wheel" should be further restricted to when an irrational decision was taken to do something with less expected utility - cost than simply using the existing version(s).
That's why I chose the example of date calculations specifically. In practice, anyone who tries to write one of those from scratch will get it wrong in lots of different ways all at once.
Yes. It's a good example. I was more or less making a point against a strawman (made of expected inference), rather than trying to oppose your specific statements; I just felt it was too easy for someone not intimate with the headaches of date functions to mistake this for a general assertion that any rewriting of existing good libraries is a Bad Thing.
Jeff Atwood
Clever-sounding and wrong is perhaps the worst combination in a rationality quote.
Douglas Hofstadter
ADBOC. Literally, that's true (but tautologous), but it suggests that understanding the nature of their sum is simple, which it isn't. Knowing the Standard Model gives hardly any insight into sociology, even though societies are made of elementary particles.
That quote is supposed to be paired with another quote about holism.
Q: What did the strange loop say to the cow? A: MU!
-- Knock knock.
-- Who is it?
-- Interrupting koan.
-- Interrupting ko-
-- MU!!!
The interesting thing is that Hofstadter doesn't seem to argue here that reductionism is true but that it's a powerful meme that easily gets into people brain.
Galileo
OTOH, thousands would be less likely to all make the same mistake than one single person -- were it not for information cascades.
Almost always false.
If the basis of the position of the thousands -is- their authority, then the reason of one wins. If the basis of their position is reason, as opposed to authority, then you don't arrive at that quote.
It depends on whether or not the thousands are scientists. I'll trust one scientist over a billion sages.
I wouldn't, though I would trust a thousand scientists over a billion sages.
It would depend on the subject. Do we control for time period and the relative background knowledge of their culture in general?
The majority is most part of time wrong. Or you search in data for patterns, or you put credences in some autor or group. People keep saying math things without basal training all the time -- here too.
Hazrat Inayat Khan.
Dennis Lindley
(I've read plenty of authors who appear to have the intuition that probabilities are epistemic rather than ontological somewhere in the back --or even the front-- of their mind, but appear to be unaware of the extent to which this intuition has been formalised and developed.)
-- Nick Szabo
What about compression?
Do you mean lossy or lossless compression? If you mean lossy compression then that is precisely Szabo's point.
On the other hand, if you mean lossless, then if you had some way to losslessly compress a brain, this would only work if you were the only one with this compression scheme, since otherwise other people would apply it to their own brains and use the freed space to store more information.
You'll probably have more success losslessly compressing two brains than losslessly compressing one.
Still, I don't think you could compress the content of 1000 brains into one. (And I'm not sure about two brains, either. Maybe the brains of two six-year-olds into that of a 25-year-old.)
I argue that my brain right now contains a lossless copy of itself and itself two words ago!
Getting 1000 brains in here would take some creativity, but I'm sure I can figure something out...
But this is all rather facetious. Breaking the quote's point would require me to be able to compute the (legitimate) results of the computations of an arbitrary number of arbitrarily different brains, at the same speed as them.
Which I can't.
For now.
But our memories discard huge amounts of information all the time. Surely there's been at least a little degradation in the space of two words, or we'd never forget anything.
Certainly. I am suggesting that over sufficiently short timescales, though, you can deduce the previous structure from the current one. Maybe I should have said "epsilon" instead of "two words".
Why would you expect the degradation to be completely uniform? It seems more reasonable to suspect that, given a sufficiently small timescale, the brain will sometimes be forgetting things and sometimes not, in a way that probably isn't synchronized with its learning of new things.
So, depending on your choice of two words, sometimes the brain would take marginally more bits to describe and sometimes marginally fewer.
Actually, so long as the brain can be considered as operating independently from the outside world (which, given an appropriately chosen small interval of time, makes some amount of sense), a complete description at time t will imply a complete description at time t + δ. The information required to describe the first brain therefore describes the second one too.
So I've made another error: I should have said that my brain contains a lossless copy of itself and itself two words later. (where "two words" = "epsilon")
See the pigeon-hole argument in the original quote.
I'd argue that your brain doesn't even contain a lossless copy of itself. It is a lossless copy of itself, but your knowledge of yourself is limited. So I think that Nick Szabo's point about the limits of being able to model other people applies just as strongly to modelling oneself. I don't, and cannot, know all about myself -- past, current, or future, and that must have substantial implications about something or other that this lunch hour is too small to contain.
How much knowledge of itself can an artificial system have? There is probably some interesting mathematics to be done -- for example, it is possible to write a program that prints out an exact copy of itself (without having access to the file that contains it), the proof of Gödel's theorem involves constructing a proposition that talks about itself, and TDT depends on agents being able to reason about their own and other agents' source codes. Are there mathematical limits to this?
I never meant to say that I could give you an exact description of my own brain and itself ε ago, just that you could deduce one from looking at mine.
If you can scan it, maybe you can simulate it? And if you can simulate one, wait some years and you can simulate 1000, probably connected in some way to form a single "thinking system".
But not on your own brain.
[Meta] This post doesn't seem to be tagged 'quotes,' making it less convenient to move from it to the other quote threads.
Done (and sorry for the long delay).
Thomas Jefferson
-- Nick Szabo
the overuse of "quantum" hurt my eyes. :(
-- In Flight Gaiden: Playing with Tropes
(Conversely, many fictions are instantiated somewhere, in some infinitesimal measure. However, I deliberately included logical impossibilities into HPMOR, such as tiling a corridor in pentagons and having the objects in Dumbledore's room change number without any being added or subtracted, to avoid the story being real anywhere.)
In the library of books of every possible string, close to "Harry Potter and the Methods of Rationality" and "Harry Potter and the Methods of Rationalitz" is "Harry Potter and the Methods of Rationality: Logically Consistent Edition." Why is the reality of that books' contents affected by your reticence to manifest that book in our universe?
Absolutely; I hope he doesn't think that writing a story about X increases the measure of X. But then why else would he introduce these "impossibilities"?
Because it's funny?
It is a different story then, so the original HpMor would still not be nonfiction in another universe. For all we know, the existance of a corridor tiled with pentagons is in fact an important plot point and removing it would utterly destroy the structure of upcoming chapters.
Nnnot really. The Time-Turner, certainly, but that doesn't make the story uninstantiable. Making a logical impossibility a basic plot premise... sounds like quite an interesting challenge, but that would be a different story.
A spell that lets you get a number of objects that is an integer such that it's larger than some other integer but smaller than it's successor, used to hide something.
This idea (the integer, not the spell) is the premise of the short story The Secret Number by Igor Teper.
And SCP-033. And related concepts in Dark Integers by Greg Egan. And probably a bunch of other places. I'm surprised I couldn't find a TVtropes page on it.
Huh. And here I thought that space was just negatively curved in there, with the corridor shaped in such a way that it looks normal (not that hard to imagine), and just used this to tile the floor. Such disappointment...
This was part of a thing, too, in my head, where Harry (or, I guess, the reader) slowly realizes that Hogwarts, rather than having no geometry, has a highly local geometry. I was even starting to look for that as a thematic thing, perhaps an echo of some moral lesson, somehow.
And this isn't even the sort of thing you can write fanfics about. :¬(
Could you explain why you did that?
As regards the pentagons, I kinda assumed the pentagons weren't regular, equiangular pentagons - you could tile a floor in tiles that were shaped like a square with a triangle on top! Or the pentagons could be different sizes and shapes.
Because he doesn't want to create Azkaban.
Also, possibly because there's not a happy ending.
But if all mathematically possible universes exist anyway (or if they have a chance of existing), then the hypothetical "Azkaban from a universe without EY's logical inconsistencies" exists, no matter whether he writes about it or not. I don't see how writing about it could affect how real/not-real it is.
So by my understanding of how Eliezer explained it, he's not creating Azkaban, in the sense that writing about it causes it to exist, he's describing it. (This is not to say that he's not creating the fiction, but the way I see it create is being used in two different ways.) Unless I'm missing some mechanism by which imagining something causes it to exist, but that seems very unlikely.
I seem to recall that he terminally cares about all mathematically possible universes, not just his own, to the point that he won't bother having children because there's some other universe where they exist anyway.
I think that violates the crap out of Egan's Law (such an argument could potentially apply to lots of other things), but given that he seems to be otherwise relatively sane, I conclude that he just hasn't fully thought it through (“decompartimentalized” in LW lingo) (probability 5%), that's not his true rejection to the idea of having kids (30%), or I am missing something (65%).
That is not the reason or even a reason why I'm not having kids at the moment. And since I don't particularly want to discourage other people from having children, I decline to discuss my own reasons publicly (or in the vicinity of anyone else who wants kids).
(I must have misremembered. Sorry)
Congratulations for having "I am missing something" at a high probability!
OK, no prob!
(I do care about everything that exists. I am not particularly certain that all mathematically possible universes exist, or how much they exist if they do. I do expect that our own universe is spatially and in several other ways physically infinite or physically very big. I don't see this as a good argument against the fun of having children. I do see it as a good counterargument to creating children for the sole purpose of making sure that mindspace is fully explored, or because larger populations of the universe are good qua good. This has nothing to do with the reason I'm not having kids right now.)
I think I care about almost nothing that exists, and that seems like too big a disagreement. It's fair to assume that I'm the one being irrational, so can you explain to me why one should care about everything?
All righty; I run my utility function over everything that exists. On most of the existing things in the modern universe, it outputs 'don't care', like for dirt. However, so long as a person exists anywhere, in this universe or somewhere else, my utility function cares about them. I have no idea what it means for something to exist, or why some things exist more than others; but our universe is so suspiciously simple and regular relative to all imaginable universes that I'm pretty sure that universes with simple laws or uniform laws exist more than universes with complicated laws with lots of exceptions in them, which is why I don't expect to sprout wings and fly away. Supposing that all possible universes 'exist' with some weighting by simplicity or requirement of uniformity, does not make me feel less fundamentally confused about all this; and therefore I'm not sure that it is true, although it does seem very plausible.
The moral value of imaginary friends?
I notice that I am meta-confused...
Shouldn't we strongly expect this weighting, by Solomonoff induction?
Probability is not obviously amount of existence.
(Assuming you mean “all imaginable universes with self-aware observers in them”.)
Not completely sure about that, even Conway's Game of Life is Turing-complete after all. (But then, it only generates self-aware observers under very complicated starting conditions. We should sum the complexity of the rules and the complexity of the starting conditions, and if we trust Penrose and Hawking about this, the starting conditions of this universe were terrifically simple.)
What do you mean, you don't care about dirt? I care about dirt! Dirt is where we get most of our food, and humans need food to live. Maybe interstellar hydrogen would be a better example of something you're indifferent to? 10^17 kg of interstellar hydrogen disappearing would be an inconsequential flicker if we noticed it at all, whereas the loss of an equal mass of arable soil would be an extinction-level event.
I care about the future consequences of dirt, but not the dirt itself.
(For the love of Belldandy, you people...)
He means that he doesn't care about dirt for its own sake (e.g. like he cares about other sentient beings for their own sakes).
Try tabooing exist: you might find out that you actually disagree on fewer things than you expect. (I strongly suspect that the only real differences between the four possibilities in this is labels -- the way once in a while people come up with new solutions to Einstein's field equations only to later find out they were just already-known solutions with an unusual coordinate system.)
I've not yet found a good way to do that. Do you have one?
"Be in this universe"(1) vs "be mathematically possible" should cover most cases, though other times it might not quite match either of those and be much harder to explain.
That's way too complicated (and as for tabooing 'exist', I'll believe it when I see it). Here's what I mean: I see a dog outside right now. One of the things in that dog is a cup or so of urine. I don't care about that urine at all. Not one tiny little bit. Heck, I don't even care about that dog, much less all the other dogs, and the urine that is in them. That's a lot of things! And I don't care about any of it. I assume Eliezer doesn't care about the dog urine in that dog either. It would be weird if he did. But it's in the 'everything' bucket, so...I probably misunderstood him?
So you're using exist in a sense according to which they have moral relevance iff they exist (or something roughly like that), which may be broader than ‘be in this universe’ but may be narrower than ‘be mathematically possible’. I think I get it now.
I was confused by this for a while, but couldn't express that in words until now.
First, I think existence is necessarily a binary sort of thing, not something that exists in degrees. If I exist 20%, I don't even know what that sentence should mean. Do I exist, but only sometimes? Do only parts of me exist at a time? Am I just very skinny? It doesn't really make sense. Just as a risk of a risk is still a type of risk, so a degree of existence is still a type of existence. There are no sorts of existence except either being real or being fake.
Secondly, even if my first part is wrong, I have no idea why having more existence would translate into having greater value. By way of analogy, if I was the size of a planet but only had a very small brain and motivational center, I don't think that would mean that I should receive more from utilitarians. It seems like a variation of the Bigger is Better or Might makes Right moral fallacy, rather than a well reasoned idea.
I can imagine a sort of world where every experience is more intense, somehow, and I think people in that sort of world might matter more. But I think intensity is really a measure of relative interactions, and if their world was identical to ours except for its amount of existence, we'd be just as motivated to do different things as they would. I don't think such a world would exist, or that we could tell whether or not we were in it from-the-inside, so it seems like a meaningless concept.
So the reasoning behind that sentence didn't really make sense to me. The amount of existence that you have, assuming that's even a thing, shouldn't determine your moral value.
I imagine Eliezer is being deliberately imprecise, in accordance with a quote I very much like: "Never speak more clearly than you think." [The internet seems to attribute this to one Jeremy Bernstein]
If you believe MWI there are many different worlds that all objectively exist. Does this mean morality is futile, since no matter what we choose, there's a world where we chose the opposite? Probably not: the different worlds seem to have different different "degrees of existence" in that we are more likely to find ourselves in some than in others. I'm not clear how this can be, but the fact that probability works suggests it pretty strongly. So we can still act morally by trying to maximize the "degree of existence" of good worlds.
This suggests that the idea of a "degree of existence" might not be completely incoherent.
I suppose you can just attribute it to imprecision, but "I am not particularly certain ...how much they exist" implies that he's talking about a subset of mathematically possible universes that do objectively exist, but yet exist less than other worlds. What you're talking about, conversely, seems to be that we should create as many good worlds as possible, stretched in order to cover Eliezer's terminology. Existence is binary, even though there are more of some things that exist than there are of other things. Using "amount of existence" instead of "number of worlds" is unnecessarily confusing, at the least.
Also, I don't see any problems with infinitarian ethics anyway because I subscribe to (broad) egoism. Things outside of my experience don't exist in any meaningful sense except as cognitive tools that I use to predict my future experiences. This allows me to distinguish between my own happiness and the happiness of Babykillers, which allows me to utilize a moral system much more in line with my own motivations. It also means that I don't care about alternate versions of the universe unless I think it's likely that I'll fall into one through some sort of interdimensional portal (I don't).
Although, I'll still err on the side of helping other universes if it does no damage to me because I think Superrationality can function well in those sort of situations and I'd like to receive benefits in return, but in other scenarios I don't really care at all.
That sounds sufficiently ominous that I'm not quite sure I want kids any more.
Obviously his reason is that he wants to personally maximize his time and resources on FAI research. Because not everyone is a seed AI programmer, this reason does not apply to most everyone else. If Eliezer thinks FAI is going to probably take a few decades (which evidence seems to indicate he does), then it probably very well is in the best interest of those rationalists who aren't themselves FAI researchers to be having kids, so he wouldn't want to discourage that. (although I don't see how just explaining this would discourage anybody from having kids who you would otherwise want to.)
Shouldn't you be taking into account that I don't want to discourage other people from having kids?
That might just be because you eat babies.
But you're afraid that if you state your reason, it will discourage others from having kids.
All that means is that he is aware of the halo effect. People who have enjoyed or learned from his work will give his reasons undue weight as a consequence, even if they don't actually apply to them.
Unfortunately, that seems to be a malleable argument. Which way your stating that (you don't want to disclose your reasons for not wanting to have kids) will influence audiences seems like it will depend heavily on their priors for how generally-valid-to-any-other-person this reason might be, and for how self-motivated both the not-wanting-to-have-kids and the not-wanting-to-discourage-others could be.
Then again, I might be missing some key pieces of context. No offense intended, but I try to make it a point not to follow your actions and gobble up your words personally, even to the point of mind-imaging a computer-generated mental voice when reading the sequences. I've already been burned pretty hard by blindly reaching for a role-model I was too fond of.
I feel that I should. It's a politically inconvenient stance to take, since all human cultures are based on reproducing themselves; antinatal cultures literally die out.
But from a human perspective, this world is deeply flawed. To create a life is to gamble with the outcome of that life. And it seems to be a gratuitous gamble.
I was sure I had heard seen you talk about them in public (On BHTV, I believe) some thing like (possible misquote) "Lbh fubhyqa'g envfr puvyqera hayrff lbh pna ohvyq bar sebz fpengpu," which sounded kinda wierd, because it applies to literally every human on earth, and that didn't seem to be where you were going.
He has said something like that, but always with the caveat that there be an exception for pre-singularity civilizations.
The way I recall it, there was no such caveat in that particular instance. I am not attempting to take him outside of context and I do think I would have remembered. He may have used this every other time he's said it. It may have been cut for time. And I don't mean to suggest my memory is anything like perfect.
But: I strongly suspect that's still on the internet, on BHTV or somewhere else.