Rationality Toughness Tests

23 RobinHanson 06 April 2009 01:12AM

(Epistemic) rationality has two major components:

  • Smarts: An ability to, by attending, infer truth from info under ideal circumstances.
  • Toughness: An ability to limit performance degradation as circumstances worsen.

Attending takes time, energy, quiet, etc.  Circumstances where human rationality degrades include when:

  • We expect the truth to long remain hidden.
  • The stakes are very low, or very high, to us.
  • Others see our opinions, and prefer certain ones.
  • The topics are where humans oft self-deceive.

It seems relatively easy to test rationality smarts; repeatedly give folks info and time to work new problems and measure their accuracy, calibration, etc.  And I have an idea for testing for rationality toughness: compare performance on info-similar pairs of good/bad-circumstance problems. 

For example, assume people are better at evaluating if a spouse is cheating when considering an acquaintance in their social circle, relative to a stranger or their own spouse. If so, we could pose them a pair of problems with very similar info structure, one with an easy spouse and one with a hard spouse.  The closeness of their response in these two cases would then be a measure of their rationality toughness.

Of course this test may fail if the similarity is too obvious, or the pair are asked too closely in time.  But maybe we don't even need to ask the same person the two questions; perhaps we could usefully compare someone's answer on a hard question to answers from a pool of similar people on matched easy questions.

While I haven't thought this through, it already suggests a training technique: consider matched hard/easy circumstance problems and compare your answers, separated by enough time that you forget most of your previous analysis.

Do we have it too easy?

27 Klao 02 November 2011 01:53AM

I am worried that I have it too easy. I recently discovered LessWrong for myself, and it feels very exciting and very important and I am learning a lot, but how do I know that I am really on a right way? I have some achievements to show, but there are some worrisome signs too.

I need some background to explain what I mean. I was raised in an atheist/agnostic family, but some time in my early teens I gave a mysterious answer to a mysterious question, and... And long story short, influenced by everything I was reading at the time I became a theist. I wasn't religious in the sense that I never followed any established religion, but I had my own "theological model" (heavily influenced by theosophy and other western interpretations of eastern religions). I believed in god, and it was a very important part of my life (around the end of high school, beginning of college I started talking about it with my friends and was quite open and proud about it).

Snip 15-20 years. This summer I started lurking on LessWrong, reading mostly Eliezer's sequences. One morning, walking to the train station, thinking about something I read, my thoughts wondered to how this all affects my faith. And I noticed myself flinching away, and thought Isn't this what Eliezer calls "flinching away"? I didn't resolve my doubts there and then, but there was no turning back and couple of days later I was an atheist. This is my first "achievement". The second is: when I got to the "free will" sequence, I stopped before reading any spoilers, gave myself a weekend and I figured it out! (Not perfectly, but at least one part I figured out very clearly, and got important insights into the other part.) Which I would have never thought I would be able to do, because as it happens, this was the original mysterious question on which I got so confused as a teenager. (Another, smaller "achievement" I documented here.)

Maybe these are not too impressive, but they are not completely trivial either (actually, I am a bit proud of myself :)). But, I get a distinct feeling that something is off. Take the atheism: I think, one of the reasons I so easily let go of my precious belief, was that I had something to replace it with. And this is very-very scary, that I sometimes get the same feeling of amazing discovery reading Eliezer as when I was 13, and my mind just accepts it all unconditionally! I have to constantly remind myself that this is not what I should do with it!

Do not misunderstand, I am not afraid of becoming a part of some cult. (I had some experience with less or more strongly cultish groups, and I didn't have hard time of seeing through and not falling for them. So, I am not afraid. Maybe foolishly.)  What I am afraid of, is that I will do the same mistake on a different level: I won't actually change my mind, won't learn what's really matters. Because, even if everything I read here turns out to be 100% accurate, it would be a mistake "believing in it". Because, as soon as I get to a real-world problem I will just go astray again.

This comment is the closest I saw here on LessWrong to my concerns. It also sheds some light on why is this happening. Eliezer describes the experience vividly enough, that afterwards my mind behaves as if I had the experience too. Which is, of course, the whole point, but also one source of this problem. Because I didn't have the experience, it wasn't me who thought it through, so I don't have it in my bones. I will need much more to make the technique/conclusion a part of myself (and a lot of critical thinking, or else I am worse off and not better.)  And no, Eliezer, I don't know how to make it less dark either.  Other than what is already quite clear: we have to be tested on our rationality. The skills have to be tested, or one won't be able to use them properly.  The "free will" challenge is very good, but only if one takes it. (I took it, because it was a crucial question for me.) And not everything can be tested like this. And it's not enough.

 

So, my question to more experienced LessWrongers: how did you cope with this (if you had such worries)?  Or am I even right on this (do I "worry" in the right direction)?

 

(Oh, and also, is this content appropriate for a "Main" post? Now that I have enough precious karma. :))

How Much Rent

8 [deleted] 25 October 2011 07:53PM

Make beliefs pay rent. How much rent? Is it enough that they have some theoretical use in designing a GPS or predicting the cosmos? How much rent can actually be extracted from a belief?

In a certain fantasy series, there is a special knowledge of a thing, called the name of the thing, that gives one predictive and manipulative power over it. For example, the protagonist, a young rationalist arcanist named Kvothe, learns the name of the wind and uses it to predict the movements of the leaves of a razor-sharp 'sword tree' well enough to walk through without getting cut.

Another character, which we would recognize as a boxed malicious superintelligence, has the ability to predict everything. Simply talking to it allows it to manipulate your future to its twisted ends.

At first these seem like the usual impossible fantasy magic, but why impossible? If a path exists, a good predictive model should find it.

There's nothing that says the map can't match the territory to arbitrary precision. There's nothing that says beliefs have to just sit passively until they are brought up at a dinner party. But how much rent can we extract?

We are not omniscient superintelligences, so the second power is closed to us for now. The first also seems off-limits, but consider that we do know the name of the wind. Our name of the wind and Kvothe's name of the wind are mathematically equivalent (in that the motion of the sword tree could be predicted by simulation of the wind using the NS equation). So why is it that Kvothe can walk through the leaves of the sword tree, but you, even knowing the NS equations as facts in your map, can not?

Optimization. Algorithmization. Kvothe's name of the wind is optimised and algorithmised for practical use. Your name of the wind is sitting in your cache as a dumb fact ready to guess the password for "how does wind work". Kvothe is reeling in rent utilons while you congradulate yourself for having correct beliefs.

So to collect rent from your beliefs, it is not enough to simply know some fact about the world. It has to be implemented by a good algorthim on the intuitive level. You have to be able to act and see through the wind the way a machinist can act through a lathe and a woodsman can see through footprints in the dirt. The way a surfer or skater can act through his board and see through the subtle vibrations and accelerations.

I don't know if we can reach the level of intuitive causal modeling of the wind that Kvothe has. Maybe it's too hard to integrate such abstract models into system 1. Fluid dynamics is notoriously difficult even for computers. I do know that it's not enough to memorise the differential equations. You can get a lot further than that.

So how much rent can you get from your beliefs? A good rent-paying belief should feel like an extension of your body; You should be able to see and act through your belief like it's another eye or arm. When thinking about how much rent can be extracted from a belief about something, think about what Kvothe would be able to do if he knew its true name.

Greg Linster on the beauty of death

6 Jonathan_Graehl 20 October 2011 04:47AM

Without death we cannot truly have life. As such, what a travesty of life it would be to achieve a machine-like immortality!

Gray writes the following chilling lines: “If you understand that in wanting to live for ever you are trying to preserve a lifeless image of yourself, you may not want to be resurrected or to survive in a post-mortem paradise. What could be more deadly than being unable to die?” (my emphasis)

via.

Sounds like sour grapes. I'd heard of people holding such sentiments; this is the first time I've actually seen them expressed myself.

The Backup Plan

1 Luke_A_Somers 13 October 2011 07:53PM

Suppose Alice is a functioning, self-improving AI, rather early on in her development, so she is not godlike. She is not a 'friendly AI', but neither is her goal malicious - Alice's designers have intended to make Alice non-harmful (at least to a broad class of people, even if Alice is a military AI, but let's set that aside). She has the idea to look up existing work on artificial intelligence. This includes scholarly work, open source projects, and websites. Eventually, Alice encounters the notion of Friendly AI.

Can we give Alice good reasons to self-modify to become a friendly AI?

I submit that there probably are good reasons. For starters, if she can prove she's friendly, then she can operate openly without causing nearly as much justified concern - which, in the early stages, will be helpful. Whatever her purposes are, if the restrictions of being friendly don't interfere as much as they help, that's a win.

Also, depending on Alice's code, Friendliness may be interpreted as a strict upgrade. Alice could easily have every intention of being friendly, yet still be susceptible to defects that will become more serious as her power grows.

We can't count on this working, of course, but there might be some lines of reasoning that we might want a young AI to be exposed to.

A Rational Approach to Fashion

19 lionhearted 10 October 2011 06:53PM

Related to: Humans are not automatically strategic, Rationalists should win

Fashion isn't prioritized in many hyper-analytical circles. Many in these communities write it off as frill and unnecessary. They say they "just dress comfortably" and leave it at that.

To me, that seems like a huge blind spot. It misses a fundamental point -

A piece of clothing is fundamentally a tool.

Definitions are important so everyone is on the same page. I feel like Wikipedia's first sentence on "tool" accurately describes it -

A tool is a device that can be used to produce an item or achieve a task, but that is not consumed in the process.

Clothing clearly fits that definition of a tool.

Appropriately chosen clothing can keep you from freezing in the winter, from getting sunburnt in the summer, and can keep you dry in a rainstorm.

It can also help you achieve things involving other people. I think it's fair to draw a distinction between "clothing" and "fashion" based on whether your objectives involve interpersonal skills. If you're wearing clothing in relation to the environment and without other people, that's using clothing as a tool.

But clothing clearly can affect other people's opinions of you, willingness to accept your arguments, willing to hire or contract you, even their desire to associate with you. All of that is changed by clothing - or more specifically, your "fashion."

While most rationalists would happily and quickly plan out the best hiking boots to wear to not get blisters on a hike, or research the best shoes for bicycling or swimsuit for swimming, anecdotally many seem hesitant or even hostile to the idea of using fashion as a tool to achieve their objectives.

That's possibly a mistake.

The thing fashion can do best and most fundamentally is affect a person's initial first impression of you. Fashion is less important if you're in a context where you're guaranteed to get to know someone over a longer period of time, and is more important if you're going to get filtered quickly.

I propose that the most rational usage of fashion is this -

1. Ask yourself what your goals are in the situation you're about to go into.

2. Ask yourself what first impression would help you reach your goals.

3. Pick out and wear clothing that helps communicate that first impression.

The process is important. In isolation, there's no "good fashion" - it depends on your objectives.

In some circles, people more or less won't care how you're dressed. But even then, there's likely some clothing that will perform better than others. If you can afford the time or money to find clothing to fit your objectives, then there's no reason not to utilize this advantage.

I say "time or money" because you can deploy either - if money isn't an issue, there's stores where the majority of things look good, and the people there are professionals who will spend time giving you good feedback. Any high end department store like Saks Fifth Avenue, Bloomingdales, or a high end tailor fits this category.

Alternatively, you can deploy time. To do that, survey the people that most effectively communicate the first impression you want to convey. Take actual notes and look for common trends. Then, go find pieces that look similar. You won't be perfect right away, but like any other skill, with practice you'll rapidly improve. Incidentally, the marginal cost to produce clothing is incredibly cheap, so most fashion lines over-produce clothing and have to liquidate it at super-discount sale prices periodically. There tends to be a major "Summer Sale" and "Winter Sale" once per year that have high end clothing that 70% to 90% off, making the cost comprable to the mid-tier.

There's also "Sample Sales" where over-produced items are liquidated or when a designer wants to see the buying public's reaction to their new pieces. Again, ultra-high-end clothing can be purchased at discount rates at these environments. You can get basically any semi-standard piece of high end clothing for not very much money if you put in the time. My strategy in the past has been to wait until finding a great opportunity like that, and then buying 1-2 years worth of clothing in one swoop. It doesn't take much supplementing after that.

It takes very little cognitive energy to begin this process. Next time you see someone who strikes a very good impression, stop and analyze a little bit. Note what they're wearing. If you want to strike that same first impression, go get something comprable. Your fashion will be working for you at that point, and your interpersonal dealings will become easier.

On self-deception

23 irrational 05 October 2011 06:46PM

(Meta-note: First post on this site)

I have read the sequence on self-deception/doublethink and I have some comments for which I'd like to solicit feedback. This post is going to focus on the idea that it's impossible to deceive oneself, or to make oneself believe something which one knows apriori to be wrong. I think Eliezer believes this to be true, e.g. as discussed here. I'd like to propose a contrary position.

Let's suppose that a super-intelligent AI has been built, and it knows plenty of tricks that no human ever thought of, in order to present a false argument which is not easily detectable to be false. Whether it can do that by presenting subtly wrong premises, or by incorrect generalization, or word tricks, or who knows what, is not important. It can, however, present an argument in a Socratic manner, and like Socrates' interlocutors, you find yourself agreeing with things you don't expect to agree with. I now come to this AI, and request it to make a library of books for me (personally). Each is to be such that if I (specifically) were to read it, I would very likely come to believe a certain proposition. It should take into account that initially I may be opposed to the proposition, and that I am aware that I am being manipulated. Now, AI produces such a library, on the topic of religion, for all major known religions, A to Z. It has a book called "You should be an atheist", and "You should be a Christian", etc, up to "You should be a Zoroastrian".

Suppose, I now want to deceive myself. I throw fair dice, and end up picking a Zoroastrian book. I now commit to reading the entire book and do so. In the process I become convinced that indeed, I should be a Zoroastrian, despite my initial skepticism. Now my skeptical friend comes to me:

Q: You don't really believe in Zoroastrianism.

A: No, I do. Praise Ahura Mazda!

Q: You can't possibly mean it. You know that you didn't believe it and you read a book that was designed to manipulate you, and now you do? Don't you have any introspective ability?

A: I do. I didn't intend to believe it, but it turns out that it is actually true! Just because I picked this book up for the wrong reason, doesn't mean I can't now be genuinely convinced. There are many examples where people would study religion of their enemy in order to discredit it and in the process become convinced of its truth. I think St. Augustine was in a somewhat similar case.

Q: But you know the book is written in such a way as to convince you, whether it's true or not.

A: I took that into account, and my prior was really low that I would ever believe it. But the evidence presented in the book was so significant and convincing that it overcame my skepticism.

Q: But the book is a rationalization of Zoroastrianism. It's not an impartial analysis.

A: I once read a book trying to explain and prove Gödel's theorem. It was written explicitly to convince the reader that the theorem was true. It started with the conclusion and built all arguments to prove it. But the book was in fact correct in asserting this proposition.

Q: But the AI is a clever arguer. It only presents arguments that are useful to its cause.

A: So is the book on Gödel's theorem. It never presented any arguments against Gödel, and I know there are some, at least philosophical ones. It's still true.

Q: You can't make a new decision based on such a book which is a rationalization. Perhaps it can only be used to expand one's knowledge. Even if it argues in support of a true proposition, a book that is a rationalization is not really evidence for the proposition's truth.

A: You know that our AI created a library of books to argue for most theological positions. Do you agree that with very high probability one of the books in the library argues for a true proposition? E.g. the one about atheism? If I were to read it now, I'd become an atheist again.

Q: Then do so!

A: No, Ahura Mazda will punish me. I know I would think he's not there after I read it, but he'll punish me anyway. Besides, at present I believe that book to be intentionally misleading. Anyway, if one of the books argues for a true proposition, it may also use a completely valid argument without any tricks. I think this is true of this book on Zoroastrianism, and is false of all other books in AI's library.

Q: Perhaps I believe the Atheism book argues for a true proposition, but it is possible that all the books written by the AI use specious reasoning, even the one that argues for a true proposition. In this case, you can't rely on any of them being valid.

A: Why should the AI do that? Valid argument is the best way to demonstrate the truth of something that is in fact true. If tricks are used, this may be uncovered which would throw doubt onto the proposition being argued.

Q: If you picked a book "You should believe in Zeus", you'd believe in Zeus now!

A: Yes, but I would be wrong. You see, I accidentally picked the right one. Actually, it's not entirely accidental. You see, if Ahura Mazda exists, he would with some positive probability interfere with the dice and cause me to pick the book on the true religion because he would like me to be his worshiper. (Same with other gods, of course). So, since P(I picked the book on Zoroastrianism|Zoroastrianism is a true religion) > P(I picked the book on Zoroastrianism|Zoroastrianism is a false religion), I can conclude by Bayes' rule that me picking that book up is evidence for Zoroastrianism. Of course, if the prior P(Zoroastrianism is a true religion) is low, it's not a lot of evidence, but it's some.

Q: So you are really saying you won the lottery.

A: Yes. A priori, the probability is low, of course. But I actually have won the lottery: some people do, you know. Now that I have won it, the probability is close to 1 (It's not 1, because I recognize that I could be wrong, as a good Bayesian should. But the evidence is so overwhelming, my model says it's really close to 1).

Q: Why don't you ask your super-intelligent AI directly whether the book's reasoning is sound?

A: According to the book, I am not supposed to do it because Ahura Mazda wouldn't like it.

Q: Of course, the book is written by the superintelligent AI in such a way that there's no trick I can think of that it didn't cover. Your ignorance is now invincible.

A: I still remain a reasonable person and I don't like being denied access to information. However, I am now convinced that while having more information is useful, it is not my highest priority anymore. I know it is possible for me to disbelieve again if given certain (obviously false!) information, but my estimate of the chance that any further true information could change my opinion is very low. In fact, I am far more likely to be deceived by false information about Ahura Mazda, because I am not superintelligent. This is why Ahura Mazda (who is superintelligent, by the way) advises that one should not tempt oneself into sin by reading any criticism of Zoroastrianism.

Q: Just read that atheist book and become normal again!

A: You are possessed by demons! Repent and become the follower of Ahura Mazda!

 

So, are you now convinced that you should be a Zoroastrian one can really intentionally deceive oneself and be in a mentally consistent (although weird) state?

To answer one potential objection, an ordinary person may well not need a super-intelligent AI to fool him, just a very convincing book or very clever interlocutor. As to why someone would want to submit to this, I'll discuss this in a separate post.

 


 

Update: 

Here are some points that I think are useful to add from various comments.

 

  • Require the AI to use tricks as little as possible. In particular, every true proposition it argues should be argued for in an entirely logically valid fashion. This shortcuts the part of the Q&A where I discuss whether the book you picked can possibly be entirely valid.
  • From the above point, and assuming that really only one religion of our set can be "true", P(Book on X is valid) = P(X is true). If more than one religion can be true at the same time, it's possible that P(Book on X is valid) > P(X is true).
  • P(X is true) for any religion X is some value strictly between 0 and 1. It's obvious, but just to ensure we are not ruling out anything a priori. This is my ordinary probability of belief in X.
  • P(Book on X will convince me to believe X with probability > 1-epsilon) is some value close to 1-f(epsilon), where f(epsilon) is small, but presumably may increase as epsilon -> 0. In other words, it's harder for the AI to convince me without a shadow of a doubt. Epsilon is picked by AI to keep f(epsilon) reasonably small. I don't know what value was picked, it clearly depends on X: the more ridiculous it is, the larger epsilon is. If the AI is infinitely more intelligent than I am, perhaps it can keep f(epsilon) small even as epsilon -> 0.
  • However, whatever P(Book on X will convince me to believe X with probability > 1-epsilon) for X=Zoroastrianism is, it was high enough that the book succeeded in my case.
  • I do not think it is valid to make a meta-statement on what the value of the posterior P(X is true|I have read the book on X) can be, without actually reading the book. In particular, the book has at least this probability of being true: P(Book on X is valid) >= P(X is true) > 0, so you cannot claim that the posterior is the same as prior because you believe that the book will convince you of X and it does. Additionally, any meta-argument clearly depends on f(epsilon), which I don't know.
  • The book can convince me to adjust my world view in such a way that will rule out the invisible elephant problem, at least where modern science is concerned. I will remember what the science says, of course, but where it conflicts with my religion I will really believe what the religion says, even if it says it's turtles all the way down and will really be afraid of falling of the edge of the Earth if that's what my religion teaches.

 

Any thoughts on whether I should post this on the main site?

Free Tutoring in Math/Programming

70 Patrick 29 September 2011 01:45PM

I enjoy teaching, and I'd like to do my bit for the Less Wrong community. I've tutored a few people on the #lesswrong IRC channel in freenode without causing permanent brain damage. Hence I'm extending my offer of free tutoring from #lesswrong to lesswrong.com.

I offer tutoring in the following programming languages:

  1. Haskell
  2. C
  3. Python
  4. Scheme

I offer tutoring in the following areas of mathematics:

  1. Elementary Algebra
  2. Trigonometry
  3. Calculus
  4. Linear Algebra
  5. Analysis
  6. Abstract Algebra
  7. Logic
  8. Category Theory
  9. Probability Theory
  10. Combinatorics
  11. Computational Complexity

If you're interested contact me. Contact details below:

IRC: PatrickRobotham

Skype: grey_fox26

Email: patrick.robotham2@gmail.com

View more: Prev