Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: Decius 17 February 2017 04:23:42PM 2 points [-]

You assume that balrogs can only be stopped by unmined bedrock. Since the chance of a given balrog being stopped by bedrock but not by the combined efforts of the dwarves is muniscule compared to the chance of a weak one that can be stopped by mithril-clad soldiers or a strong one that can dig through mere stone, the best defense against balrogs is to mine and guard the mines well.

Comment author: casebash 05 January 2016 05:46:36AM *  0 points [-]

"You are looking for an agent who is capable of choosing The Highest Number" - the agent wants to maximise utility, not to pick the highest number for its own sake, so that is misrepresenting my position. If you want to taboo utility, let's use the word "lives saved" instead. Anyway, you say "Therefore this agent (the perfect life maximising agent) can not exist", which is exactly what I was concluding. Concluding the exact same thing as I concluded, supports my argument, it doesn't contradict it like you seem to think it does.

"Alternately, letting "utility" back in, in a universe of finite time, matter, and energy, there does exist a maximum finite utility" - my argument is that there does not exist perfect rationality within the imagined infinite universe. I said nothing about the actual, existing universe.

Comment author: Decius 06 January 2016 04:39:10AM 0 points [-]

"Lives saved" is finite within a given light cone.

Comment author: casebash 05 January 2016 11:40:57PM 0 points [-]

"Add in a time limit and calculation rate, and you're back to normal rationality" - I am intentionally modelling a theoretical construct, not reality. Claims that my situation isn't realistic aren't valid, as I have never claimed that this theoretical situation does correspond to reality. I have purposefully left this question open.

Comment author: Decius 06 January 2016 04:37:20AM -2 points [-]

The perfectly rational agent considers all possible different world-states, determines the utility of each of them, and states "X", where X is the utility of the perfect world.

For the number "X+epsilon" to have been a legal response, the agent would have had to been mistaken about their utility function or what the possible worlds were.

Therefore X is the largest real number.

Note that this is a constructive proof, and any attempt at counterexample should attempt to prove that the specific X discovered by a perfectly rational omniscient abstract agent with a genie. If the general solution is true, it will be trivially true for one number.

Comment author: Decius 06 January 2016 04:30:03AM 0 points [-]

For the Unlimited Swap game, are you implicitly assuming that the time spent swapping back and forth has some small negative utility?

Comment author: AlexanderEarlheart 05 January 2016 06:01:16PM 0 points [-]

You are right, theory is overrated. Just because you don't have a theoretical justification for commencing an action doesn't mean that the action isn't the right action to take if you want to try to "win." Of course, it is very possible to be in a situation where "winning" is inherently impossible, in which case you could still (rationally) attempt various strategies that seem likely to make you better off than you would otherwise be...

As a practicing attorney, I've frequently encountered real-life problems similar to the above. For example, in a negotiation on behalf of a client, there is often what's called a "bargaining zone" that represents a range of options for possible "deals" that both parties are theoretically willing to accept. Any given "deal" would be Pareto Efficient, and any "deal" within the "bargaining zone," if it takes place, would make both parties to the negotiation better off than they were before. However, it is possible to strike a superior deal for your client if you are more aggressive and push the terms into the "upper" range of the bargaining zone. On the other hand, you don't typically know the extent of the "bargaining zone" before you begin negotiations. If you are TOO aggressive and push outside of the range of the other party's acceptable options, the other party/counsel might get frustrated with you and call off the negotiations entirely, in which case you will lose the deal for everyone and make your client angry with you.

To the extent "winning" is possible here, the strategy for attorneys on both sides is to push the terms of the "deal" as close as possible to the "edge" of what the other will accept without pushing too far and getting the talks called off. Although there are reasonable strategies to the process, very often there isn't a theoretical "optimally rational strategy" for "winning" a negotiation -- you just have to play the game and make your strategic decisions based on new information as it becomes available.

Comment author: Decius 06 January 2016 12:48:53AM 0 points [-]

There is an optimal strategy for negotiation. It requires estimating the negotiation zone of the other party and the utility of various outcomes (including failure of negotiation).

Then it's just a strategy that maximizes the sum of the probability of each outcome times the utility thereto.

The hard parts aren't the P(X1)U(X1) sums, it's getting the P(X1) and U(X1) in the first place.

Comment author: casebash 05 January 2016 12:53:38PM 1 point [-]

Spoilers, haha.

I was actually reading this post and I was trying to find a solution to the coalition problem where Eliezer wonders how rational agents can solve a problem with the potential for an infinite loop, which lead me to what I'll call the Waiting Game, where you can wait n units of time and gain n utility for any finite n, which then led me to this post.

Comment author: Decius 06 January 2016 12:41:38AM *  1 point [-]

Suppose instead that the game is "gain n utility". No need to speak the number, wait n turns, or even to wait for a meat brain to make a decision or comprehend the number.

I posit that a perfectly rational, disembodied agent would decide to select an n such that there exists no n higher. If there is a possible outcome that such an agent prefers over all other possible outcomes, then by the definition of utility such an n exists.

Comment author: Decius 31 October 2015 08:45:54PM 0 points [-]

All A1 needs to do is say "A0 will not take actions which it does not find a proof in T that they are 'safe', and will not create A-1 that does not contain these restrictions."

That would restrict the A-series from improving their fundamental logic systems; is that drawback too severe?

Comment author: Manfred 13 June 2015 08:39:27AM 1 point [-]

I think the typical example is if you do a search for a proof of inconsistency in Peano arithmetic. You don't expect to find any inconsistencies, but you can't prove that you won't.

Comment author: Decius 19 June 2015 01:48:25AM -1 points [-]

More like trying to find the Godel statement of the universe; it provably exists, and provably cannot be positively identified.

Comment author: Manfred 10 June 2015 05:26:36PM *  2 points [-]

See the Church-Turing thesis for more on this topic.

Comment author: Decius 12 June 2015 11:58:10PM 0 points [-]

I think I was on a slightly different topic:

Some Turing machines have been proven to not halt; some have not. There must exist at least one Turing machine which no Turing machine can ever prove does not halt. (It is trivial to prove that a Turing machine halts if it does)

Since there are a countably infinite number of Turing machines, there must be at most a countably infinite number of characteristics such that only every non-halting Turing machine has one or more of those characteristics. If we suppose that each of these characteristics can be checked by a single Turing machine that halts when it proves that the target does not halt, then we have a contradiction (since we can build a Turing metamachine oracle that diagonalizes a countably infinite number of machines each testing one property).

Therefore there exists some characteristic of a Turing machine which is sufficient for that machine to be non-halting, such that it cannot be proven that said characteristic is sufficient.

I wonder what a program that doesn't halt but cannot be proven not to halt looks like? What does the output look like after 2BB(n) steps? It must have a HALT function somewhere accessible, or else it would be trivial to prove that it never halted, but likewise said function must never happen, which means that the condition must never happen; but the condition must be accessible, or it would be trivial to prove that it never halted...

Comment author: Manfred 09 June 2015 07:00:50AM 4 points [-]

One common method of resolving this is to cash out "representable" numbers in terms of outputs of halting turing machines, so that paradoxes of Berry's sort require solving the halting problem and are therefore not themselves representations.

Comment author: Decius 10 June 2015 03:37:30AM 0 points [-]

... Unless there exists something other than a Turing machine that can solve the halting problem.

Which of course leads us to things like "The set of Turing machines that do not halt and cannot be proven to not halt by any Turing machine".

View more: Next