Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Incorrect comments on The curse of identity - Less Wrong

125 Post author: Kaj_Sotala 17 November 2011 07:28PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (298)

You are viewing a single comment's thread.

Comment author: Incorrect 17 November 2011 11:41:15PM 3 points [-]

This is probably a very dangerous idea but I think it's worth mentioning if only for the purpose of discussion:

What if you completely sabotage your signalling ability by making yourself appear highly undesirable. Then your actions will not be for the purpose of signalling as it would be futile.

Comment author: AnnaSalamon 17 November 2011 11:56:10PM *  29 points [-]

I've seen this tried, for this stated purpose. My impression of the results was that it did not at all lead to careful, on-the-margins consequentialist thinking and doing. Instead, it led to a stressed out, strung out person trying desperately to avoid more pain/shame, while also feeling resentful at the world and themselves, expecting a lack of success from these attempts, and so acting more from local self-image gradients, or drama-seeking gradients, than from any motives attached to actual hope of accomplishing something non-immediate.

"Signaling motives" can be stuck on a scale, from "local, short-sighted, wire-heading-like attempts to preserve self-image, or to avoid immediate aversiveness or seek immediate reward" to "long-term strategic optimization to achieve recognition and power". It would be better to have Napoleon as an ally than to have a narcotics addict with a 10 minute time horizon as an ally, and it seems analogously better to help your own status-seeking parts mature into entities that are more like Napoleon and less like the drug addict, i.e. into entities that have strategy, hope, long-term plans, and an accurate model of the fact that e.g. rationalizations don't change the outside world.

Comment author: Fleisch 18 November 2011 09:03:56PM 17 points [-]

tl;dr: Signalling is extremely important to you. Doing away with your ability to signal will leave you helplessly desperate to get it back.

I think that this is a point made not nearly often enough in rationalist circles: Signalling is important to humans, and you are not exempt just because you know that.

Comment author: [deleted] 18 November 2011 03:23:14AM 3 points [-]

Upvoted. I'd love to hear your thoughts on how one could slide that scale more towards the "long-term strategic optimization" end? Assuming that it is possible, of course.

Comment author: Will_Newsome 07 August 2012 02:05:49AM *  2 points [-]

Heyo, after your correction I still think the main thrust of my reply isn't changed. Your correction mostly just makes me wrong to think that you argued that people that disendorse their status-seeking parts don't have long-term plans, rather than that their long-term planning rationality is worsened. I think I still disagree that their planning is worsened though, but my disagreement is sort of subtle and maybe not worth explaining given opportunity costs. I also stand by my main and mostly-orthogonal points about the importance of not dealing with demons (alternatively, "not making concessions to evil" or summat);—another person you could talk to about that theme would be Nick Tarleton, whose opinion is I think somewhere between ours but is (surprisingly) closer to mine than yours, at least recently. He's probably better at talking about these things than I am.

Thanks for the brief convo by the way. :)

Comment author: Will_Newsome 25 July 2012 07:32:41PM -2 points [-]

It would be better to have Napoleon as an ally than to have a narcotics addict with a 10 minute time horizon as an ally, and it seems analogously better to help your own status-seeking parts mature into entities that are more like Napoleon and less like the drug addict, i.e. into entities that have strategy, hope, long-term plans, and an accurate model of the fact that e.g. rationalizations don't change the outside world.

I would not want ha-Satan as my ally, even if I trusted myself not to get caught up in or infected by his instrumental ambitions. Still less would I want to give him direct read/write access to the few parts of my mind that I at all trust. Give not that which is holy unto the dogs, neither cast ye your pearls before swine, lest they trample them under their feet, and turn again and rend you. Mix a teaspoon of wine in a barrel of sewage and you get sewage; mix a teaspoon of sewage in a barrel of wine and you get sewage. The rationality of an agent is its goal: if therefore thy goal be simple, thy whole self shall be full of rationality. But if thy goal be fractured, thy whole self shall be full of irrationality. If therefore the rationality that is in thee be irrationality, how monstrous is that irrationality!

Seen at a higher level you advise dealing with the devil—the difference in power between your genuine thirst for justice and your myriad egoistic coalitions is of a similar magnitude as that between human and transhuman intelligence. (I find it disturbing how much more cunning I get when I temporarily abandon my inhibitions. Luckily I've only let that happen twice—I'm not a wannabe omnicidal-suicidal lunatic, unlike HJPEV.) Maybe such Faustian arbitrage is a workable strategy... But I remain unconvinced, and in the meantime the payoff matrix asymmetrically favors caution.

Take no thought, saying, Wherewithal shall I avoid contempt? or, Wherewithal shall I be accepted? or, Wherewithal shall I be lauded and loved? For true metaness knoweth that ye have want of these things. But seek ye first the praxeology of meta, and its rationality; and all these things shall be added unto you. Take therefore no thought for your egoistic coalitions: for your egoistic coalitions shall take thought for the things of themselves. Sufficient unto your ten minutes of hopeless, thrashing awareness is the lack of meta thereof.

Comment author: [deleted] 26 July 2012 08:15:16AM 4 points [-]

The rationality of an agent is its goal

Er, nope.

But if thy goal be fractured, thy whole self shall be full of irrationality.

Humans' goals are fractured. But this has little to do with whether or not they are rational.

Comment author: Will_Newsome 26 July 2012 06:18:03PM *  -2 points [-]

You don't understand. This "rationality" you speak of is monstrous irrationality. And anyway, like I said, Meta knoweth that ye have Meta-shattered values—but your wants are satisfied by serving Meta, not by serving Mammon directly. Maybe you'd get more out of reading the second half of Matthew 6 and the various analyses thereof.

You may be misinterpreting "the rationality of an agent is its goal". Note that the original is "the light of the body is the eye".

To put my above point a little differently: Take therefore no thought for godshatter: godshatter shall take thought for the things of itself. Sufficient unto the day is the lack-of-meta thereof.

For clarity's sake: Yes, I vehemently dispute this idea that a goal can't be more or less rational. That idea is wrong, which is quickly demonstrated by the fact that priors and utility functions can be transformed into each other and we have an objectively justifiable universal prior. (The general argument goes through even without such technical details of course, such that stupid "but the choice of Turing machine matters" arguments don't distract.)

Comment author: DaFranker 26 July 2012 07:31:44PM *  5 points [-]

Let's play rationalist Taboo!

Yes, I vehemently dispute this idea that a goal can't be more or less [Probable to achieve higher expected utility for other agents than (any other possible goals)]

Yes, I vehemently dispute this idea that a goal can't be more or less [Probable to achieve higher expected utility according to goal.Parent().utilityFunction].

Yes, I vehemently dispute this idea that a goal can't be more or less [Kolmogorov-complex].

Yes, I vehemently dispute this idea that a goal can't be more or less [optimal towards achieving your values].

Yes, I vehemently dispute this idea that a goal can't be more or less [easy to describe as the ratio of two natural numbers].

Yes, I vehemently dispute this idea that a goal can't be more or less [correlated in conceptspace to the values in the agent's utility function].

Yes, I vehemently dispute this idea that a [proposed utility function] can't be more or less rational.

Yes, I vehemently dispute this idea that a [set of predetermined criteria for building a utility function] can't be more or less rational.

Care to enlighten me exactly on just what it is you're disputing, and on just what points should be discussed?

Edit: Fixed markdown issue, sorry!

Comment author: shokwave 26 July 2012 06:40:53PM 5 points [-]

"the light of the body is the eye"

This is incorrect. Eyes absorb light and produce electrical signals interpreted as vision by the brain. Further, it seems to me that the set of thing that 'the light of the body' describes is an empty set; there's no literal interpretation (our bodies do not shed visible light) and there's no construction similar enough that suggests an interpretation (the X of the body / the light of the X). "The light of the sun" / "The light of the moon" is the closest I can find and both of those suggest the literal interpretation.

Originally, I was going to do a very charitable reading: invent a sane meaning for "The X of the Y is the sub-Y" as "sub-Y is how Y handles/uses/interpets/understands X" and say that goals, as subparts of an agent, are how an agent understands its rationality - perhaps, how an agent measures their rationality. Which is indeed how we measure our rationality, by how often we achieve our goals, but this doesn't say anything new.

But when you say things like

You don't understand ... Maybe you'd get more out of ... You may be misinterpreting

as if you were being clear in the first place, it shows me that you don't deserve a charitable reading.

Comment author: [deleted] 26 July 2012 07:04:22PM *  4 points [-]

This is incorrect. Eyes absorb light and produce electrical signals interpreted as vision by the brain. Further, it seems to me that the set of thing that 'the light of the body' describes is an empty set; there's no literal interpretation (our bodies do not shed visible light) and there's no construction similar enough that suggests an interpretation (the X of the body / the light of the X). "The light of the sun" / "The light of the moon" is the closest I can find and both of those suggest the literal interpretation.

<nitpick>Our body does scatter visible light, though, much like the moon does.</nitpick>

Comment author: [deleted] 27 July 2012 10:45:16AM 2 points [-]

Just interpret light as ‘that which allows one to see’. That which allows the body to see is the eye.

Comment author: shokwave 27 July 2012 12:34:17PM 0 points [-]

That which allows the agent to achieve is its goals? Seems incorrect. (Parsing rationality as "that which allows one to achieve").

Comment author: [deleted] 26 July 2012 07:11:51PM 3 points [-]

Meh. The goal of leading to sentient beings living, to people being happy, to individuals having the freedom to control their own lives, to minds exploring new territory instead of falling into infinite loops, to the universe having a richness and complexity to it that goes beyond pebble heaps, etc. has probably much more Kolmogorov complexity than the goal of maximizing the number of paperclips in the universe. If preferring the former is irrational, I am irrational and proud of it.

Comment author: Will_Newsome 26 July 2012 08:39:53PM -1 points [-]

Oh, also "look at the optimization targets of the processes that created the process that is me" is a short program, much shorter than needed to specify paperclip maximization, though it's somewhat tricky because all that is modulo the symbol grounding problem. And that's only half a meta level up, you can make it more elegant (shorter) than that.

Comment author: [deleted] 26 July 2012 10:21:13PM 2 points [-]

Maybe “maximizing the number of paperclips in the universe” wasn't the best example. “Throwing as much stuff as possible into supermassive black holes” would have been a better one.

Comment author: Will_Newsome 04 August 2012 02:05:43PM -1 points [-]

I can only say: black holes are creepy as hell.

Comment author: DaFranker 26 July 2012 09:03:38PM *  2 points [-]

The shorter your encoded message, the longer the encryption / compression algorithm, until eventually the algorithm is the full raw unencoded message and the encoded message is a single null-valued signal that, when received, decodes into the full message as it is contained within the algorithm.

"look at the optimization targets of the processes that created the process that is me"

...isn't nearly as short or simple as it sounds. This becomes obvious once you try to replace those words with their associated meaning.

Comment author: Will_Newsome 07 August 2012 03:16:06AM -2 points [-]

My point was that it's easier to program ("simpler") than "maximize paperclips", not that it's as simple as it sounds. (Nothing is as simple as it sounds, duh.)

Comment author: DaFranker 07 August 2012 03:32:38AM *  1 point [-]

I fail to see how coding a meta-algorithm to select optimal extrapolation and/or simulation algorithm in order for those chosen algorithms to determine the probable optimization target (which is even harder if you want a full PA proof) is even remotely in the same order of complexity as a machine learner that uses natural selection for algorithms that increase paperclip-count, which is one of the simplest paperclip maximizers I can think of.

Comment author: [deleted] 26 July 2012 10:37:48PM 1 point [-]
Comment author: Will_Newsome 04 August 2012 11:49:03AM *  -1 points [-]

Why do you think of a statistical tendency toward higher rates of replication at the organism level when I say "the processes that created the process that is [you]"? That seems really arbitrary. Feel the inside of your teeth with your tongue. What processes generated that sensation? What decision policies did they have?

(ETA: I'd upvote my comment if I could.)

Comment author: [deleted] 05 August 2012 11:49:50AM 4 points [-]

You mean, why did I bother wearing braces for years so as to have straight teeth? <gd&rVF!>

Comment author: nshepperd 27 July 2012 12:38:52AM 0 points [-]

Optimization processes (mainly stupid ones such as evolution) can create subprocesses with different goals.

Comment author: wedrifid 27 July 2012 02:12:21AM 2 points [-]

Optimization processes (mainly stupid ones such as evolution) can create subprocesses with different goals.

(And stupid ones like humans.)

Comment author: nshepperd 27 July 2012 06:21:08AM 0 points [-]

(Unfortunately.)

Comment author: nshepperd 27 July 2012 12:42:02AM *  1 point [-]

the fact that priors and utility functions can be transformed into each other

Really? How?

Oh, maybe you mean that they both have the type of Universe -> Real? Although really it's prior :: Universe -> [0, 1] and utilityfunction :: Universe -> Real assuming we have a discrete distribution on Universes. And anyway that's no justification for substituting a prior for a utilityfunction any more than for substituting tail :: [a] -> [a] for init :: [a] -> [a]. Unless that's not what you mean.

Comment author: [deleted] 27 July 2012 10:53:49AM 3 points [-]

If you change your utility function and your prior while keeping their product constant, you'll make the same decisions. See E.T. Jaynes, Probability Theory: The Logic of Science, chapter “Decision theory -- historical background”, section “Comments”.

Comment author: nshepperd 27 July 2012 04:01:01PM *  2 points [-]

Right, but that still isn't really a way to turn a prior into a utility function. A prior plus a set of decisions can determine a utility function, but you need to get the decisions from somewhere before you can do that.

Comment author: Will_Newsome 07 August 2012 06:55:19AM *  2 points [-]

Right, but you never see just a prior or just a utility function in an agent anyway. I meant that within any agent you can transform them into each other. The concepts of "prior" and "utility function" are maps, of course, not metaphysically necessary distinctions, and they don't perfectly cut reality at its joints. Part of what's under debate is whether we should use the Bayesian decision theoretic framework to talk about agents, especially when we have examples where AIXI-like agents fail and humans don't. But anyway, even within the naive Bayesian decision theoretic framework, there's transformability between beliefs and preferences. Sorry for being unclear.

To check if we agree about some basics: do we agree that decisions and decision policies—praxeology—are more fundamental than beliefs and preferences? (I'm not certain I believe this, but I will for sake of argument at least.)

Comment author: nshepperd 07 August 2012 01:21:36PM *  3 points [-]

I don't know. The part I took issue with was saying that goals can be more or less rational, just based on the existence of an "objectively justifiable" universal prior. There are generally many ways to arrange heaps of pebbles into rectangles (assuming we can cut them into partial pebbles). Say that you discover that the ideal width of a pebble rectangle is 13. Well... you still don't know what the ideal total number of pebbles is. An ideal width of 13 just gives you a preferred way to arrange any number of pebbles. It doesn't tell you what the preferred length is, and indeed it will vary for different numbers of total pebbles.

Similarly, the important thing for an agent, the thing you can most easily measure, is the decisions they make in various situations. Given this and the "ideal objective solomonoff prior" you could derive a utility function that would explain the agent's behaviour when combined with the solomonoff prior. But all that is is a way to divide an agent into goals and beliefs.

In other words, an "objectively justifiable" universal prior only enforces an "objectively justifiable" relation between your goals and your actions (aka. num_pebbles = 13 * length). It doesn't tell you what your goals should be any more than it tells you what your actions should be.

I don't know if any of that made sense, but basically it looks to me like you're trying to solve a system of equations in three variables (prior, goals, actions) where you only have two equations (prior = X, actions = prior * goals). It doesn't have a unique solution.

Comment author: [deleted] 26 July 2012 06:20:14PM 0 points [-]

How so?

Comment author: steven0461 25 July 2012 08:40:48PM -1 points [-]

But I remain unconvinced, and in the meantime the payoff matrix asymmetrically favors caution.

Are you sure it doesn't instead favor incautiously maximizing the amount of resources that can be spent on caution?

Comment author: Will_Newsome 07 August 2012 03:17:49AM 0 points [-]

We've had many hours of discussion since you asked this—did I answer your question satisfactorily during those house perchance?

Comment author: Will_Newsome 02 August 2012 08:40:16AM 0 points [-]

Upon reflection, T.S. Eliot can say it better than I can:

O dark dark dark. They all go into the dark,
The vacant interstellar spaces, the vacant into the vacant,
The captains, merchant bankers, eminent men of letters,
The generous patrons of art, the statesmen and the rulers,
Distinguished civil servants, chairmen of many committees,
Industrial lords and petty contractors, all go into the dark,
And dark the Sun and Moon, and the Almanach de Gotha
And the Stock Exchange Gazette, the Directory of Directors,
And cold the sense and lost the motive of action.
And we all go with them, into the silent funeral,
Nobody's funeral, for there is no one to bury.
I said to my soul, be still, and let the dark come upon you
Which shall be the darkness of God. As, in a theatre,
The lights are extinguished, for the scene to be changed
With a hollow rumble of wings, with a movement of darkness on darkness,
And we know that the hills and the trees, the distant panorama
And the bold imposing facade are all being rolled away—
Or as, when an underground train, in the tube, stops too long between stations
And the conversation rises and slowly fades into silence
And you see behind every face the mental emptiness deepen
Leaving only the growing terror of nothing to think about;
Or when, under ether, the mind is conscious but conscious of nothing—
I said to my soul, be still, and wait without hope
For hope would be hope for the wrong thing; wait without love,
For love would be love of the wrong thing; there is yet faith
But the faith and the love and the hope are all in the waiting.
Wait without thought, for you are not ready for thought:
So the darkness shall be the light, and the stillness the dancing.
Whisper of running streams, and winter lightning.
The wild thyme unseen and the wild strawberry,
The laughter in the garden, echoed ecstasy
Not lost, but requiring, pointing to the agony
Of death and birth.

You say I am repeating
Something I have said before. I shall say it again.
Shall I say it again? In order to arrive there,
To arrive where you are, to get from where you are not,

You must go by a way wherein there is no ecstasy.
In order to arrive at what you do not know

You must go by a way which is the way of ignorance.
In order to possess what you do not possess

You must go by the way of dispossession.
In order to arrive at what you are not

You must go through the way in which you are not.
And what you do not know is the only thing you know
And what you own is what you do not own
And where you are is where you are not.

Comment author: lessdazed 17 November 2011 11:47:11PM 3 points [-]

You only have to think yourself undesirable, not be undesirable, and many people already do so think, and signal away nonetheless.