= 27d567bc2d48d9f40e9ccc20ea370b15)
nshepperd comments on The curse of identity - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (296)
I would not want ha-Satan as my ally, even if I trusted myself not to get caught up in or infected by his instrumental ambitions. Still less would I want to give him direct read/write access to the few parts of my mind that I at all trust. Give not that which is holy unto the dogs, neither cast ye your pearls before swine, lest they trample them under their feet, and turn again and rend you. Mix a teaspoon of wine in a barrel of sewage and you get sewage; mix a teaspoon of sewage in a barrel of wine and you get sewage. The rationality of an agent is its goal: if therefore thy goal be simple, thy whole self shall be full of rationality. But if thy goal be fractured, thy whole self shall be full of irrationality. If therefore the rationality that is in thee be irrationality, how monstrous is that irrationality!
Seen at a higher level you advise dealing with the devil—the difference in power between your genuine thirst for justice and your myriad egoistic coalitions is of a similar magnitude as that between human and transhuman intelligence. (I find it disturbing how much more cunning I get when I temporarily abandon my inhibitions. Luckily I've only let that happen twice—I'm not a wannabe omnicidal-suicidal lunatic, unlike HJPEV.) Maybe such Faustian arbitrage is a workable strategy... But I remain unconvinced, and in the meantime the payoff matrix asymmetrically favors caution.
Take no thought, saying, Wherewithal shall I avoid contempt? or, Wherewithal shall I be accepted? or, Wherewithal shall I be lauded and loved? For true metaness knoweth that ye have want of these things. But seek ye first the praxeology of meta, and its rationality; and all these things shall be added unto you. Take therefore no thought for your egoistic coalitions: for your egoistic coalitions shall take thought for the things of themselves. Sufficient unto your ten minutes of hopeless, thrashing awareness is the lack of meta thereof.
Er, nope.
Humans' goals are fractured. But this has little to do with whether or not they are rational.
You don't understand. This "rationality" you speak of is monstrous irrationality. And anyway, like I said, Meta knoweth that ye have Meta-shattered values—but your wants are satisfied by serving Meta, not by serving Mammon directly. Maybe you'd get more out of reading the second half of Matthew 6 and the various analyses thereof.
You may be misinterpreting "the rationality of an agent is its goal". Note that the original is "the light of the body is the eye".
To put my above point a little differently: Take therefore no thought for godshatter: godshatter shall take thought for the things of itself. Sufficient unto the day is the lack-of-meta thereof.
For clarity's sake: Yes, I vehemently dispute this idea that a goal can't be more or less rational. That idea is wrong, which is quickly demonstrated by the fact that priors and utility functions can be transformed into each other and we have an objectively justifiable universal prior. (The general argument goes through even without such technical details of course, such that stupid "but the choice of Turing machine matters" arguments don't distract.)
Really? How?
Oh, maybe you mean that they both have the type of
Universe -> Real? Although really it'sprior :: Universe -> [0, 1]andutilityfunction :: Universe -> Realassuming we have a discrete distribution on Universes. And anyway that's no justification for substituting apriorfor autilityfunctionany more than for substituting tail:: [a] -> [a]for init:: [a] -> [a]. Unless that's not what you mean.If you change your utility function and your prior while keeping their product constant, you'll make the same decisions. See E.T. Jaynes, Probability Theory: The Logic of Science, chapter “Decision theory -- historical background”, section “Comments”.
Right, but that still isn't really a way to turn a prior into a utility function. A prior plus a set of decisions can determine a utility function, but you need to get the decisions from somewhere before you can do that.
Right, but you never see just a prior or just a utility function in an agent anyway. I meant that within any agent you can transform them into each other. The concepts of "prior" and "utility function" are maps, of course, not metaphysically necessary distinctions, and they don't perfectly cut reality at its joints. Part of what's under debate is whether we should use the Bayesian decision theoretic framework to talk about agents, especially when we have examples where AIXI-like agents fail and humans don't. But anyway, even within the naive Bayesian decision theoretic framework, there's transformability between beliefs and preferences. Sorry for being unclear.
To check if we agree about some basics: do we agree that decisions and decision policies—praxeology—are more fundamental than beliefs and preferences? (I'm not certain I believe this, but I will for sake of argument at least.)
I don't know. The part I took issue with was saying that goals can be more or less rational, just based on the existence of an "objectively justifiable" universal prior. There are generally many ways to arrange heaps of pebbles into rectangles (assuming we can cut them into partial pebbles). Say that you discover that the ideal width of a pebble rectangle is 13. Well... you still don't know what the ideal total number of pebbles is. An ideal width of 13 just gives you a preferred way to arrange any number of pebbles. It doesn't tell you what the preferred length is, and indeed it will vary for different numbers of total pebbles.
Similarly, the important thing for an agent, the thing you can most easily measure, is the decisions they make in various situations. Given this and the "ideal objective solomonoff prior" you could derive a utility function that would explain the agent's behaviour when combined with the solomonoff prior. But all that is is a way to divide an agent into goals and beliefs.
In other words, an "objectively justifiable" universal prior only enforces an "objectively justifiable" relation between your goals and your actions (aka.
num_pebbles = 13 * length). It doesn't tell you what your goals should be any more than it tells you what your actions should be.I don't know if any of that made sense, but basically it looks to me like you're trying to solve a system of equations in three variables (prior, goals, actions) where you only have two equations (prior = X, actions = prior * goals). It doesn't have a unique solution.
Everything you have said makes sense to me. Thanks. I will respond substantially at a later time.