Beautiful.
And it almost makes me want to be Brennan - with all the training of the Bayesian Conspiracy could give me, a fiercely desired goal in my mind, and the whole world open in front of me. (And - presumably - no existential risks to worry about.)
You're assuming that his goal is not, in itself, an existential risk. Something that is almost, but not quite, unattainable.. we know of such a technology already, and Eliezer is aiming for it.
Does the "If you don't know what you need, take power" quote have any origin before this? I searched for it but only found it in a post on heuristics that linked back here. I would be interested in more discussion around it before I adopt it as a method.
Reminds me of The Left Hand of Darkness - one bit where the main character says something along the lines of, 'If action ceases to be profitable, gather information.' but there were two layers to it, with something else to do either between the two or after gathering information.
As they say in the Ekumenical School, when action grows unprofitable, gather information; when information grows unprofitable, sleep.
Of course there was something he wanted. He knew exactly what he wanted. Wanted so desperately he could taste it like an sharp tinge on his tongue.
This alone would've been the most useful rationalist technique ever if it worked. But it doesn't work. Wanting to want doesn't make me want.
That said, best of wishes on your quest!
I think cousin_it was going for "having one's desires feel strongly, proportional to the amount they really want it", rather than partly saying "I really want X, but my more immediate desires that I don't 'really want' are desires I feel/experience much more strongly and end up being something I want to but find difficulty struggling against."
ie, I'm guessing the point was ultimately an akrasia issue.
I find a helpful counterweight to the near mode is to keep an intuition pump close at hand. A small collection of readily available facts and anecdotes about one's Cause, and what one plans to do about it, needs to be curated frequently but is most definitely worth it.
This story originally had a few more italicized words, and they make a big difference:
"Don't," Jeffreyssai said. There was real pain in it.
"I do not know," said Jeffreyssai, from which the overall state of the evidence was obvious enough.
Some of the formatting must have been lost when it was imported to LessWrong 2.0. You can see the original formatting at readthesequences.com and in Rationality: AI to Zombies.
There are also minor rewordings throughout. The LessWrong version differs from the other two, which (from a limited sample) agree with each other. I would guess that the latter is the authorised version.
Brennan asked: "Is this the only way in which Bayesian masters come to be, sensei?"
And thought: "How could Jeffreyssai possibly have known before Brennan knew himself?"
He wants to find a better way to train Bayesian masters.
I'd wager 10 karma points against 1 that this is not the desire Eliezer has imagined for Brennan.
I'd only wager 5 against 1 that he has a specific desire imagined for Brennan.
(Thus begins the prediction market for the Bardic Conspiracy...)
I'd wager 10 karma points against 1
low priority, I'm sure, but I'd be entirely in favor of a means by which we could enact bets like these.
Might be too easily gamed. Imagine that A and B are both keen to get more karma -- of course A and B might really be the same person, though not the same LW-user -- then they both make 1000-to-1 bets against one another, etc. (Of course the karma system can already be subverted a little by simpler means -- A and B just upvote one another -- but that's much slower and milder.)
[EDIT, later: This comment is simply wrong; I wasn't thinking straight. Sorry.]
You can't do that for uneven bets, like orthonormal's. (I suppose you could have a negative-sum system where you say "I'm willing to gain 1 point at the risk of N", and then you need to find N people who will all bet with you on those terms; they all make 1-for-1 bets, but if you win you only get 1. But that doesn't seem terribly appealing.)
I don't understand the problem.
If Eliezer imagined Brennan as wanting to create more Bayesian masters, ortho would lose 10 points, which MrShaggy would gain. Under the reverse case, ortho gains a point, MrShaggy loses one.
I voted this down and the parent up because, while it's a fine apology, you should not actually get more karma for admitting a mistake than the person who corrected you gets.
I voted this down, and the immediate parent up, because recognizing one's errors and acknowledging them is worthy of Karma, even if the error was pointed out to you by another.
I voted this down, and the immediate parent up, because I hope that someone will find my comment ridiculous and vote it down and its parent up and say so, causing someone else to disagree with that response, voting the response down and this comment up.
I voted Eliezer up because I think his observation was perfectly reasonable and didn't deserve downvoting, and because his action seems eminently reasonable (I'm always glad of extra karma, but I can hardly claim to be entitled to +5 rather than +4 for being an idiot and then admitting it).
I voted Guy's comment down, then up, then down again, then back to neither-up-nor-down. I hope that's sufficiently ridiculous to match his comment.
I voted this down and the immediate parent up because I think this conversation was funny and I want the chain to be as long as possible for maximum funniness. And I'm willing to pay a karma point to do it.
And then I changed my downvote of gjm to an upvote, because his comment was actually good.
I voted this up, and the immediate parent down.
Bertha Jorkins voted this up and its immediate parent down, and she now has an IQ of 180 and an army of artificially intelligent robot slaves. Charlie Gordon voted this up and its immediate parent down, and gained 120 IQ points, but he lost them all again because he broke the chain.
DON'T BREAK THE CHAIN!
I rolled a 1d3 dice using the lambdabot in #haskell to determine what I would do, assigning 1 to vote parent and its parent up, 2 to do nothing, and 3 to downvote parent and its parent. I got a 2.
I rolled 1d12 with an actual d12 (Hey, you kids! Get offa my lawn!). 1-4 to upvote gwern, 5-8 to do nothing, and 9-12 to downvote. I got a 10. Then I upvoted all the comments between gwern and Eliezer, inclusive, as a celebration of starting this thread up again. (I also found another comment of gjm's and voted it up, because the Bertha Jorkins and Charlie Gordon references are brilliant.
Hopefully a*|I voted this down and immediate parent down>*|other-stuff-1> + b*|I voted this up and immediate parent down>*|other-stuff-2> + c*|I voted this down and immediate parent up>*|other-stuff-3> + d*|I voted this up and immediate parent up>*|other-stuff-4> + e*|other possible outcomes>
with a, b, c, d having hopefully approximately the same modulus and e having small modulus, by requesting from hotbits (which claims to use a quantum source of uncertainty) one byte, taking the two lowest order bits, with the 1 bit being for the immediate comment I'm replying to and the 2 bit being for its immediate parent, going by the rule of 0 = downvote and 1 = upvote. (Well, okay, really a mixed state given how it'll all work, but there will be in the mixed state a sum of states of the form described above, so...)
Requesting byte... now:
And this blob of quantum amplitude is a blob that received FC. So let's see, that means... both low order bits zero. So downvotes for both Normal_anomaly and gwern. Awww. Well, at least there'll hopefully be other branches of equal weight in which you both got upvotes from this procedure.
(Hotbits apparently stores up random bits and generally serves requests by peeling off the stack of stored random bits the number of bytes requested. So if the pre stored stuff entangled itself with the rest of the world sufficiently that my decision to do this ended up nontrivially entangled with the particular byte I got, then no promises about other branches. But again, probably end up with basically just a whole lot of states similar to the desired one except that the "other stuff" parts are a tad different.)
I voted on this and the immediate parent, but I won't reveal why, or which direction, or how many times, or which account I used.
Blame me, because I restarted the chain. I voted this down, because it was not very amusing, and the parent up, because I assume it was an HPMOR reference and that is awesome.
Voted up all comments in this chain except this one, because I can't vote on my own comments anymore.
I just want to say that I find this chain ridiculously funny beyond all expected measure. None of it (past a point) has any reason to exist, but it still went on quite a while. Good job everyone on writing something so amazingly ridiculous :-)
Voted down this comment, because 2 other people voted it up and didn't even have the guts to admit to it.
I'm voting up some pseudo-random selection of comments in this chain, because...well because I found that there was a bottom, and it seemed natural to extend it one further.
I voted up on every comment in this chain on which someone stated that they voted it up, and down on every comment on this chain on which someone stated that they voted it down, removing votes when they cancelled out and using strong-votes instead when they added together. I regret to say that the comment by Dorikka seems to have had three more people say that they voted it up than that they voted it down, so although I gave it a strong upvote, I have only been able to replicate two-thirds of the original vote. I upvoted Dorikka's last comment on another post to bring the universe back into balance.
Fine with me. (I'm going to take the self-flattering route and assume that my comment got voted up because being prepared to admit one's errors is a good thing, rather than because the observation "gjm is an idiot" is particularly worthy of upvotes...)
I guess this raises a different question: I've been attempting to use my up and down votes as a straight expression of how I regard the post or comment. While I can't guarantee that I am never drawn to inadvertently engage in corrective voting (where I attempt to bring a post or comment's karma in line with where I think it should be in an absolute sense or relative to another post), it seems as though this is your conscious approach.
What are the advantages/disadvantages or the two approaches?
The correct voting system looks like this: everyone assigns to each post the score they think it should have. The voting system adds a number of "fake votes" at each threshhold to ensure that posts with few votes don't get too high a rating, and then takes the median vote as the score. That way there's no need for "corrective voting" - voting for the score you want to see will always do the right thing.
You win, naturally.
I can think of a desire that Brennan might have, and indeed had it in mind at the time; but since it doesn't appear in the story, it would seem that my particular belief is not particularly privileged...
...but I agree with you that Brennan's desire is almost certainly not "a better way to train Bayesian masters", that would be way the hell out of his revealed character.
Also note: "Power, he'd sought at first. Strength to prevent a repetition of the past. "If you don't know what you need, take power" - so went the proverb. He had gone first to the Competitive Conspiracy, then to the beisutsukai."
But the teacher has promised failure as a seemingly necessary step to mastery on this path, so it has not fulfilled what he went there for yet.
You may take advice you should not take.
I understand that this means to just ask for advice, not necessarily follow it. Why can this be a bad thing?
For a true Bayesian, information would never have negative expected utility. But humans aren’t perfect Bayes-wielders; if we’re not careful, we can cut ourselves. How can we cut ourselves in this case? I suppose you could have made up your mind to follow a course of action that happens to be correct and then ask someone for advice and the someone will change your mind.\
Lets say you already have lots of evidence for one hypothesis so asking someone is unlikely to change your mind. Yet if you are underconfident you might still be tempted to ask and if someone gives you contradictory advice you as a human will still feel the uncertainty and doubt inside you. This will just be a wasted motion.
Is there more to it?
Sunlight enriched air already alive with curiosity, as dawn rose on Brennan and his fellow students in the place to which Jeffreyssai had summoned them.
They sat there and waited, the five, at the top of the great glassy crag that was sometimes called Mount Mirror, and more often simply left unnamed. The high top and peak of the mountain, from which you could see all the lands below and seas beyond.
(Well, not all the lands below, nor seas beyond. So far as anyone knew, there was no place in the world from which all the world was visible; nor, equivalently, any kind of vision that would see through all obstacle-horizons. In the end it was the top only of one particular mountain: there were other peaks, and from their tops you would see other lands below; even though, in the end, it was all a single world.)
"What do you think comes next?" said Hiriwa. Her eyes were bright, and she gazed to the far horizons like a lord.
Taji shrugged, though his own eyes were alive with anticipation. "Jeffreyssai's last lesson doesn't have any obvious sequel that I can think of. In fact, I think we've learned just about everything that I knew the beisutsukai masters know. What's left, then -"
"Are the real secrets," Yin completed the thought.
Hiriwa and Taji and Yin shared a grin, among themselves.
Styrlyn wasn't smiling. Brennan suspected rather strongly that Styrlyn was older than he had admitted.
Brennan wasn't smiling either. He might be young, but he kept high company, and had witnesssed some of what went on behind the curtains of the world. Secrets had their price, always, that was the barrier that made them secrets; and Brennan thought he had a good idea of what this price might be.
There was a cough from behind them, at a moment when they had all happened to be looking in any other direction but that one.
As one, their heads turned.
Jeffreyssai stood there, in a casual robe that looked more like glass than any proper sort of mirrorweave.
Jeffreyssai stood there and looked at them, a strange abiding sorrow in those inscrutable eyes.
"Sen...sei," Taji started, faltering as that bright anticipation stumbled over Jeffreyssai's return look. "What's next?"
"Nothing," Jeffreyssai said abruptly. "You're finished. It's done."
Hiriwa, Taji, and Yin all blinked, a perfect synchronized gesture of shock. Then, before their expressions could turn to outrage and objections -
"Don't," Jeffreyssai said. There was real pain in it. "Believe me, it hurts me more than it hurts you." He might have been looking at them; or at something far away, or long ago. "I don't know exactly what roads may lie before you - but yes, I know that you're not ready. That I'm sending you out unprepared. That everything I taught you is incomplete. I know that what I said is not what you heard. That I left out the one most important thing. That the rhythm at the center of everything is missing and astray. I know that you will harm yourself in the course of trying to use what I taught. So that I, personally, will have shaped, in some fashion unknown to me, the very knife that will cut you..."
"...that's the hell of being a teacher, you see," Jeffreyssai said. Something grim flickered in his expression. "Nonetheless, you're done. Finished, for now. What lies between you and mastery is not another classroom. We are fortunate, or perhaps not fortunate, that the road to power does not wend only through lecture halls. Else the quest would be boring to the bitter end. Still, I cannot teach you; and so it is a moot point whether I would if I could. There is no master here whose art is entirely inherited. Even the beisutsukai have never discovered how to teach certain things; it is possible that such an event has been prohibited. And so you can only arrive at mastery by using to the fullest the techniques you have already learned, facing challenges and apprehending them, mastering the tools you have been taught until they shatter in your hands -"
Jeffreyssai's eyes were hard, as though steeled in acceptance of unwelcome news.
"- and you are left in the midst of wreckage absolute. That is where I, your teacher, am sending you. You are not beisutsukai masters. I cannot create masters. I have never known how to create masters. Go forth, then, and fail."
"But -" said Yin, and stopped herself.
"Speak," said Jeffreyssai.
"But then why," she said, "why teach us anything in the first place?"
Brennan's eyelids flickered some tiny amount.
It was enough for Jeffreyssai. "Answer her, Brennan, if you think you know."
"Because," Brennan said, "if we were not taught, there would be no chance at all of our becoming masters."
"Even so," said Jeffreyssai. "If you were not taught - then when you failed, you might simply think you had reached the limits of Reason itself. You would be discouraged and bitter within your disaster. You might not even realize when you had failed. No; you have been shaped into something that may emerge from the wreckage, determined to remake your Art. And then you may remember much that will help you. I cannot create masters, but if you had not been taught, your chances would be - less." His gaze passed over the group. "It should be obvious, but understand that you cannot provoke the moment of your crisis artificially. To teach you something, the catastrophe must come to you as a surprise. You must go as far as you can, as best you can, and fail honestly. The higher road begins after the Art seems to fail you; though the reality will be that it was you who failed your Art."
Brennan made the gesture with his hand that indicated a question; and Jeffreyssai nodded in reply.
"Is this the only way in which Bayesian masters come to be, sensei?"
"I do not know," said Jeffreyssai, from which the overall state of the evidence was obvious enough. "But I doubt there would ever be a road to mastery that goes only through the monastery. We are the heirs in this world of mystics as well as scientists, just as the Competitive Conspiracy inherits from chessplayers alongside cagefighters. We have turned our impulses to more constructive uses - but we must still stay on our guard against old failure modes."
Jeffreyssai took a breath. "Three flaws above all are common among the beisutsukai. The first flaw is to look just the slightest bit harder for flaws in arguments whose conclusions you would rather not accept. If you cannot contain this aspect of yourself then every flaw you know how to detect will make you that much stupider. This is the challenge which determines whether you possess the art or its opposite: Intelligence, to be useful, must be used for something other than defeating itself."
"The second flaw is cleverness. To invent great complicated plans and great complicated theories and great complicated arguments - or even, perhaps, plans and theories and arguments which are commended too much by their elegance and too little by their realism. There is a widespread saying which runs: 'The vulnerability of the beisutsukai is well-known; they are prone to be too clever.' Your enemies will know this saying, if they know you for a beisutsukai, so you had best remember it also. And you may think to yourself: 'But if I could never try anything clever or elegant, would my life even be worth living?' This is why cleverness is still our chief vulnerability even after its being well-known, like offering a Competitor a challenge that seems fair, or tempting a Bard with drama."
"The third flaw is underconfidence, though it will seem to you like modesty or humility. You have learned so many flaws in your own nature, some of them impossible to fix, that you may think that the rule of wisdom is to confess your own inability. You may question yourself, without resolution or testing to determine the self-answers. You may refuse to decide, pending further evidence, when a quick decision is necessary. You may take advice you should not take. Jaded cynicism and sage despair are less fashionable than once they were, but you may still be tempted by them. Or you may simply - lose momentum."
Jeffreyssai fell silent then.
He looked from each of them, one to the other, with quiet intensity.
And said at last, "Those are my final words to you. If and when we meet next, you and I - if and when you return to this place, Brennan, or Hiriwa, or Taji, or Yin, or Styrlyn - I will no longer be your teacher."
And Jeffreyssai turned and walked swiftly away, heading back toward the glassy tunnel that had emitted him.
Even Brennan was shocked. For a moment they were all speechless.
Then -
"Wait!" cried Hiriwa. "What about our final words to you? I never said -"
"I will tell you what my sensei told me," Jeffreyssai's voice came back as he disappeared. "You can thank me after you return, if you return. One of you at least seems likely to come back."
"No, wait, I -" Hiriwa fell silent. In the mirrored tunnel, the fractured reflections of Jeffreyssai were already fading. She shook her head. "Never... mind, then."
There was a brief, uncomfortable silence, as the five of them looked at each other.
"Good heavens," Taji said finally. "Even the Bardic Conspiracy wouldn't try for that much drama."
Yin suddenly laughed. "Oh, this was nothing. You should have seen my send-off when I left Diamond Sea University." She smiled. "I'll tell you about it sometime - if you're interested."
Taji coughed. "I suppose I should go back and... pack my things..."
"I'm already packed," Brennan said. He smiled, ever so slightly, when the other three turned to look at him.
"Really?" Taji asked. "What was the clue?"
Brennan shrugged with artful carelessness. "Beyond a certain point, it is futile to inquire how a beisutsukai master knows a thing -"
"Come off it!" Yin said. "You're not a beisutsukai master yet."
"Neither is Styrlyn," Brennan said. "But he has already packed as well." He made it a statement rather than a question, betting double or nothing on his image of inscrutable foreknowledge.
Styrlyn cleared his throat. "As you say. Other commitments call me, and I have already tarried longer than I planned. Though, Brennan, I do feel that you and I have certain mutual interests, which I would be happy to discuss with you -"
"Styrlyn, my most excellent friend, I shall be happy to speak with you on any topic you desire," Brennan said politely and noncommitally, "if we should meet again." As in, not now. He certainly wasn't selling out his Mistress this early in their relationship.
There was an exchange of goodbyes, and of hints and offers.
And then Brennan was walking down the road that led toward or away from Mount Mirror (for every road is a two-edged sword), the glassy pebbles clicking under his feet.
He strode out along the path with purpose, vigor, and determination, just in case someone was watching.
Some time later he stopped, stepped off the path, and moved just far enough away to prevent anyone from finding him unless they were deliberately following.
Then Brennan sagged back against a tree-trunk. It was a sparse clearing, with only a few trees poking out of the ground; not much present in the way of distracting scenery, unless you counted the red-tinted stream flowing out of a dark cave-mouth. And Brennan deliberately faced away from that, leaving only the far grey of the horizons, and the blue sky and bright sun.
Now what?
He had thought that the Bayesian Conspiracy, of all the possible trainings that existed in this world, would have cleared up his uncertainty about what to do with the rest of his life.
Power, he'd sought at first. Strength to prevent a repetition of the past. "If you don't know what you need, take power" - so went the proverb. He had gone first to the Competitive Conspiracy, then to the beisutsukai.
And now...
Now he felt more lost than ever.
He could think of things that made him happy, but nothing that he really wanted.
The passionate intensity that he'd come to associate with his Mistress, or with Jeffreyssai, or the other figures of power that he'd met... a life of pursuing small pleasures seemed to pale in comparison, next to that.
In a city not far from the center of the world, his Mistress waited for him (in all probability, assuming she hadn't gotten bored with her life and run away). But to merely return, and then drift aimlessly, waiting to fall into someone else's web of intrigue... no. That didn't seem like enough.
Brennan plucked a blade of grass from the ground and stared at it, half-unconsciously looking for anything interesting about it; an old, old game that his very first teacher had taught him, what now seemed like ages ago.
Why did I believe that going to Mount Mirror would tell me what I wanted?
Well, decision theory did require that your utility function be consistent, but...
If the beisutsukai knew what I wanted, would they even tell me?
At Mount Mirror they taught doubt. So now he was falling prey to the third besetting sin of which Jeffreyssai had spoken, lost momentum, for he had learned to question the image that he held of himself in his mind.
Are you seeking power because that is your true desire, Brennan?
Or because you have a picture in your mind, of the role that you play as an ambitious young man, and you think it is what someone playing your role would do?
Almost everything he'd done up until now, even going to Mount Mirror, had probably been the latter.
And when he blanked out the old thoughts and tried to see the problem as though for the first time...
...nothing much came to mind.
What do I want?
Maybe it wasn't reasonable to expect the beisutsukai to tell him outright. But was there anything they had taught him by which he might answer?
Brennan closed his eyes and thought.
First, suppose there is something I would passionately desire. Why would I not know what it is?
Because I have not yet encountered it, or ever imagined it?
Or because there is some reason I would not admit it to myself?
Brennan laughed out loud, then, and opened his eyes.
So simple, once you thought of it that way. So obvious in retrospect. That was what they called a silver-shoes moment, and yet, if he hadn't gone to Mount Mirror, it wouldn't ever have occurred to him.
Of course there was something he wanted. He knew exactly what he wanted. Wanted so desperately he could taste it like an sharp tinge on his tongue.
It just hadn't come to mind earlier, because... if he acknowledged his desire explicitly... then he also had to see that it was difficult. High, high, above him. Far out of his reach. "Impossible" was the word that came to mind, though it was not, of course, physically impossible.
But once he asked himself if he preferred to wander aimlessly through his life - once it was put that way, the answer became obvious. Pursuing the unattainable would make for a hard life, but not a sad one. He could think of things that made him happy, either way. And in the end - it was what he wanted.
Brennan stood up, and took his first steps, in the exact direction of Shir L'or, the city that lies in the center of the world. He had a plot to hatch, and he did not know who would be part of it.
And then Brennan almost stumbled, when he realized that Jeffreyssai had already known.
One of you at least seems likely to come back...
Brennan had thought he was talking about Taji. Taji had probably thought he was talking about Taji. It was what Taji said he wanted. But how reliable of an indicator was that, really?
There was a proverb about that very road he had just left: Whoever sets out from Mount Mirror seeking the impossible, will surely return.
When you considered Jeffreyssai's last warning - and that the proverb said nothing of succeeding at the impossible task itself - it was a less optimistic saying than it sounded.
Brennan shook his head wonderingly. How could Jeffreyssai possibly have known before Brennan knew himself?
Well, beyond a certain point, it is futile to inquire how a beisutsukai master knows a thing -
Brennan halted in mid-thought.
No.
No, if he was going to become a beisutsukai master himself someday, then he ought to figure it out.
It was, Brennan realized, a stupid proverb.
So he walked, and this time, he thought about it carefully.
As the sun was setting, red-golden, shading his footsteps in light.