Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: So8res 16 April 2014 04:57:13PM *  3 points [-]

Incorrect -- your implementation itself also affects the environment via more than your chosen output channels. (Your brain can be scanned, etc.) If you define waste heat, neural patterns, and so on as "output channels" then sure, we can say you only interact via I/O (although the line between I and O is fuzzy enough and your control over the O is small enough that I'd personally object to the distinction).

However, AIXI is not an agent that communicates with the environment only via I/O in this way: if you insist on using the I/O model then I point out that AIXI neglects crucial I/O channels (such as its source code).

until I see the actual math

In fact, Botworld is a tool that directly lets us see where AIXI falls short. (To see the 'actual math', simply construct the game described below with an AIXItl running in the left robot.)

Consider a two-cell Botworld game containing two robots, each in a different cell. The left robot is running an AIXI, and the left square is your home square. There are three timesteps. The right square contains a robot which acts as follows:

1. If there are no other robots in the square, Pass.
2. If an other robot just entered the square, Pass.
3. If an other robot has been in the square for a single turn, Pass.
4. If an other robot has been in the square for two turns, inspect its code.
.. If it is exactly the smallest Turing machine which never takes any action,
.. move Left.
5. In all other cases, Pass.

Imagine, further, that your robot (on the left) holds no items, and that the robot on the right holds a very valuable item. (Therefore, you want the right robot to be in your home square at the end of the game.) The only way to get that large reward is to move right and then rewrite yourself into the smallest Turing machine which never takes any action.

Now, consider the AIXI running on the left robot. It quickly discovers that the Turing machine which receives the highest reward acts as follows:

1. Move right
2. Rewrite self into smallest Turing machine which does nothing ever.

The AIXI then, according to the AIXI specification, does the output of the Turing machine it's found. But the AIXI's code is as follows:

1. Look for good Turing machines.
2. When you've found one, do it's output.

Thus, what the AIXI will do is this: it will move right, then it will do nothing for the rest of time. But while the AIXI is simulating the Turing machine that rewrites itself into a stupid machine, the AIXI itself has not eliminated the AIXI code. The AIXI's code is simulating the Turing machine and doing what it would have done, but the code itself is not the "do nothing ever" code that the second robot was looking for -- so the AIXI fails to get the reward.

The AIXI's problem is that it assumes that if it acts like the best Turing machine it found then it will do as well as that Turing machine. This assumption is true when the AIXI only interacts with the environment over I/O channels, but is not true in the real world (where eg. we can inspect the AIXI's code).

Comment author: Cyan 17 April 2014 01:40:31AM *  -2 points [-]

Actually, an AI that believes it only communicates with the environment via input/output channels cannot represent the hypothesis that it will stop receiving input bits.

But I am an intelligence that can only communicate with the environment via input/output channels!

Incorrect -- your implementation itself also affects the environment via more than your chosen output channels.

Okay, fair enough. But until you pointed that out, I was an intelligence that believed it only communicated with the environment via input/output channels (that was your original phrasing, which I should have copied in the first place), and yet I did (and do) believe that it is possible for me to die.

Thus, what the AIXI will do is this: it will move right, then it will do nothing for the rest of time.

Incorrect. I'll assume for the sake of argument that you're right about what AIXI will do at first. But AIXI learns by Solomonoff induction, which is infallible at "noticing that it is confused" -- all Turing machines that fail to predict what actually happens get dropped from the hypothesis space. AIXI does nothing just until that fails to cause the right-room robot to move, whereupon any program that predicted that merely outputting "Pass" forever would do the trick gets zeroed out.

The AIXI's problem is that it assumes that if it acts like the best Turing machine it found then it will do as well as that Turing machine.

If there are programs in the hypothesis space that do not make this assumption (and as far as I know, you and I agree that naturalized induction would be such a program), then these are the only programs that will survive the failure of AIXI's first plan.

Has Paul Christiano looked at this stuff?

ETA: I don't usually mind downvotes, but I find these ones (currently -2) are niggling at me. I don't think I'm being conspicuously stupid, and I do think that discussing AIXI in a relatively concrete scenario could be valuable, so I'm a bit at a loss for an explanation. ...Perhaps it's because I appealed to Paul Christiano's authority?

Comment author: So8res 15 April 2014 07:51:47PM 2 points [-]

Actually, an AI that believes it only communicates with the environment via input/output channels cannot represent the hypothesis that it will stop receiving input bits. See this post for a discussion of the finer details.

Comment author: Cyan 16 April 2014 03:37:59PM *  1 point [-]

But I am an intelligence that can only communicate with the environment via input/output channels! And so are you!

How is it that we are able to represent the hypothesis that one can die? I refuse to accept that humans do something that AIXI can't until I see the actual math. (I don't affirm the opposite claim, mind you.)

Comment author: Cyan 15 April 2014 12:53:20PM *  2 points [-]

Such formalisms are perhaps ill-suited for real self-modifying agents, which are embedded within their environments. Indeed, the agent/environment separation is somewhat reminiscent of Cartesian dualism: any agent using this framework to reason about the world does not model itself as part of its environment. For example, such an agent would be unable to understand the concept of the environment interfering with its internal computations, e.g. by inducing errors in the agent’s RAM through heat.

I'm not sure that "unable" is the best way of describing this state of affairs. It's fairer to say that such agents start with no model of itself as part of the environment and would have to build such a model from scratch -- rather like how humans tend to believe that we have souls that will persist beyond death, and have to get educated about how the world works before we understand the implausibility of that belief.

Comment author: EHeller 08 April 2014 03:51:17AM 4 points [-]

So in my limited experience with physics and statistics phds I've sent here, its north of 80%.

Comment author: Cyan 08 April 2014 04:12:36AM *  2 points [-]

Ouch. (The statistics PhD thing is saddening to me but not surprising, alas.)

Comment author: Eliezer_Yudkowsky 07 April 2014 11:59:42PM 4 points [-]

The primary failure mode of writing is that nobody reads it. I don't know how to write like Bostrom in a way that people will read. I'm already worried about things like the tiling agents paper dropping off the radar.

Comment author: Cyan 08 April 2014 03:36:36AM 8 points [-]

The steps you could take to avoid the nobody-reads-it failure mode seem to me to be orthogonal to the steps you could take to avoid the author-is-a-colossal-prick failure mode. Given that you started this whole damn web site and community as insurance against the possibility that there might be someone out there with more innate talent for FAI, lukeprog's suggestion that you take steps to mitigate the author-is-a-colossal-prick failure mode in furtherance of that mission seems like a pretty small ask to me. And I say this as one who has always enjoyed your writing.

Comment author: eli_sennesh 07 April 2014 11:10:26AM 0 points [-]

Basically, I accept that critique, but only at an engineering level. Ditto on the "how much" issue: it's engineering. Neither of these issues actually makes me believe that a welfare state strapped awkwardly on top of a fundamentally industrial-capitalist, resource-capitalist, or financial-capitalist system - and constantly under attack by anyone perceiving themselves as a put-upon well-heeled taxpayer to boot - is actually a better solution to poverty and inequality than a more thoroughly socialist system in which such inequalities and such poverty just don't happen in the first place (because they're not part of the system's utility function).

I certainly believe that we have not yet designed or located a perfect socialist system to implement. What I do note, as addendum to that, is that nobody who supports capitalism believes the status quo is a perfect capitalism, and most people who aren't fanatical ideologues don't even believe we've found a perfect capitalism yet. The lack of a preexisting design X and a proof that X Is Perfect do not preclude the existence of a better system, whether redesigned from scratch or found by hill-climbing on piecemeal reforms.

All that lack means is that we have to actually think and actually try -- which we should have been doing anyway, if we wish to act according to our profession to be rational.

Comment author: Cyan 07 April 2014 01:15:08PM 1 point [-]

Good answer. (Before this comment thread I was, and I continue to be, fairly sympathetic to these efforts.)

Comment author: eli_sennesh 07 April 2014 08:10:09AM 2 points [-]

For what it's worth, it seems to me that you've used the term "socialism" to refer to two different, conflated, specific policies. In the OP you seem to be talking about direct redistribution of money, which isn't necessarily equivalent to the notion of worker control of the means of production that you introduce in the parent; and the term "socialism" doesn't pick out either specific policy in my mind.

Ah, here's the confusion. No, in the OP I was talking about worker control of the means of production, and criticizing Effective Altruism for attempting to fix poverty and sickness through what I consider an insufficiently effective intervention, that being direct redistribution of money.

Comment author: Cyan 07 April 2014 10:53:18AM *  1 point [-]

Oh, I see. Excellent clarification.

How would you respond to (what I claim to be) Krugman's account, i.e., in current conditions poor households are budget-constrained and would, if free to do so, liquidate their ownership of the means of production for money to buy the things they need immediately? Just how much redistribution of ownership are you imagining here?

Comment author: eli_sennesh 06 April 2014 04:10:12PM *  0 points [-]

The whole point of the second quotation and the paragraph after that was to... Oh never mind, should I just assume henceforth that contrary to its usage in socialist discourse, to outsiders "socialism" always means state-owned monopoly? In that case, what sort of terminology should I use for actual worker control of the means of production, and such things?

Comment author: Cyan 07 April 2014 01:48:06AM *  1 point [-]

"Anarcho-syndicalism" maybe? All's I know is that my socialized health insurance is a state-run oligopsony/monopoly (and so is my province's liquor control board). In any event, if direct redistribution of wealth is the key identifier of socialism, then Milton Friedman was a socialist, given his support for negative income taxes.

Prolly the best thing would be to avoid jargon as much as possible when talking to outsiders and just state what concrete policy you're talking about. For what it's worth, it seems to me that you've used the term "socialism" to refer to two different, conflated, specific policies. In the OP you seem to be talking about direct redistribution of money, which isn't necessarily equivalent to the notion of worker control of the means of production that you introduce in the parent; and the term "socialism" doesn't pick out either specific policy in my mind. (An example of how redistribution and worker ownership are not equivalent: on Paul Krugman's account, if you did direct redistribution right now, you'd increase aggregate demand but not even out ownership of capital. This is because current household consumption seems to be budget-constrained in the face of the ongoing "secular stagnation" -- if you gave poor people a whack of cash or assets right now, they'd (liquidate and) spend it on things they need rather than investing/holding it. )

Comment author: eli_sennesh 06 April 2014 02:35:26PM *  -2 points [-]

The principle of indifference. — The idea that from an altruistic point of view, we should care about people who are unrelated to us as much as we do about people who are related to us. For example, in The Life You Can Save: How to Do Your Part to End World Poverty, Peter Singer makes the case that we should show a similar degree of moral concern for people in the developing world who are suffering from poverty as we do to people in our neighborhoods. I'd venture the guess its popularity among rationalists is an artifact of culture or a selection effect rather than a consequence of rationality. Note that concern about global poverty is far more prevalent than interest in rationality (while still being low enough so that global poverty is far from alleviated).

Without deliberately bringing up mind-killy things, I would have to ask, if we tie together Effective Altruism and rationality, why Effective Altruists are not socialists of some sort.

Rawls does not deny the reality of political power, nor does he claim that it has its roots elsewhere than in the economic arrangements of a society. But by employing the models of analysis of the classical liberal tradition and of neo-classical [welfare] economics, he excludes that reality from the pages of his book.

-- Understanding Rawls: A Reconstruction and Critique of a Theory of Justice, by Robert Paul Wolff

I actually picked that up on the recommendation from an LW thread to read about Rawls, but I hope the highlight gets Wolff's point across. Elsewhere, he phrases it roughly as: by default, the patterns of distribution arise directly from the patterns of production, and therefore we can say or do very little about perceived distributional problems if we are willing to change nothing at all about the underlying patterns of production producing (ahaha) the problematic effect.

Or in much simpler words: why do we engage in lengthy examinations of sending charity to people who could look after themselves just fine if we would stop robbing them of resources? The success of GiveDirectly should be causing us to reexamine the common assumption that poor people are poor for some reason other than that they lack property to capitalize for themselves.

Anyway, I'm going to don my flame-proof suit now. (And in my defense, my little giving this year so far has already included $720 to CareerVillage for advising underprivileged youth in the First World and $720 to GiveDirectly for direct transfer to the poor in the Third World. I support interventions that work!)

Comment author: Cyan 06 April 2014 04:04:43PM *  2 points [-]

I would have to ask, if we tie together Effective Altruism and rationality, why Effective Altruists are not socialists of some sort.

I'm a loyal tovarisch of Soviet Canuckistan, and I have to say that doesn't seem like a conundrum to me: there's no direct contradiction between basing one's charitable giving on evidence about charitable organizations' effectivenesses and thinking that markets in which individual are free to act will lead to more preferable outcomes than markets with state-run monopolies/monopsonies.

Comment author: B_For_Bandana 03 April 2014 12:27:32AM *  2 points [-]

(Edited to add context)

Context: The speakers work for a railroad. An important customer has just fired them in favor of a competitor, the Phoenix-Durango Railroad.

Jim Taggart [Company president, antagonist]: "What does he expect? That we drop all our other shippers, sacrifice the interests of the whole country and give him all our trains?"

Eddie Willers [Junior exec, sympathetic character]: "Why, no. He doesn't expect anything. He just deals with the Phoenix-Durango."

  • Atlas Shrugged

It gets at the idea talked about here sometimes that reality has no obligation to give you tests you can pass; sometimes you just fail and that's it.

ETA: On reflection, what I think the quote really gets at is that Taggart cannot understand that his terminal goals may be only someone else's instrumental goals, that other people are not extensions of himself. Taggart's terminal goal is to run as many trains as possible. If he can help a customer, then the customer is happy to have Taggart carry his freight, and Taggart's terminal goal aligns with the customer's instrumental goal. But the customer's terminal goal is not to give Taggart Inc. business, but just to get his freight shipped. If the customer can find a better alternative, like competing railroad, he'll switch. For Taggart, of course, that is not a better alternative at all, hence his anger and confusion.

(Apologies for lack of context initially).

Comment author: Cyan 03 April 2014 06:11:28PM 8 points [-]

Without context, it's a bit difficult to see how this is a rationality quote. Not everyone here has read Atlas Shrugged...

View more: Next