A puzzling question is why your brain doesn't get this right automatically. In particular, deciding whether to gather some food before sleeping is an issue mammals have faced in the EEA for millions of years.
Temporal difference learning seems so basic that brains ought to implement it reasonably accurately. Any idea why we might do the wrong thing in this case?
Can you imagine any mechanism whereby a box would drive you mad if you put a coin in it? (I can't.)
Perhaps sticking a coin in it triggers the release of some psychoactive gas or aerosol?
Are there any psychoactive gases or aerosols that drive you mad?
I suppose a psychedelic might push someone over the edge if they were sufficiently psychologically fragile. I don't know of any substances that specifically make people mad, though.
A universal ontology is intractable, no argument there. As is a tree of (meta)*-probabilities. My point was about how to regard the problem.
As for an actual solution, we start with propositions like "this box has a nontrivial potential to kill, injure or madden me.". I can find a probability for that based on my knowledge of you and on what you've said. If the probability is small enough, I can subdivide that by considering another proposition.
One aspect of what I consider the correct solution is that the only question that needs to be answered is "do I think putting a coin in the box has positive or negative utility", and one can answer that without any guess about what it is actually going to do.
What is your base rate for boxes being able to drive you mad if you put a coin in them?
Can you imagine any mechanism whereby a box would drive you mad if you put a coin in it? (I can't.)
I don't have a full strategy, but I have an idea for a data-gathering experiment:
I hand you a coin and try to get you to put it in the box for me. If you refuse, I update in the direction of the box harming people who put coins in it. If you comply, I watch and see what happens.
Excellent! This is very much pointing in the direction of what I consider the correct general approach. I hadn't thought of what you suggest specifically, but it's an instance of the general category I had in mind.
Can you please give us a top level post at some point, be it in Discussion or Main, arguing that "the universe is not a bit string"? I find that very interesting, relevant, and plausible.
Thanks for the encouragement! I have way too many half-completed writing projects, but this does seem an important point.
"Interesting" tends to mean "whatever it would be, it does that more" in the context of possibly psuedo-Faustian bargains and signals of probable deceit. From what I know, I do not start with reason to trust you, and the evidence found in the OP suggests that I should update the probability that you are concealing information updating on which would lead me not to use the black box to "much higher".
Oh, goodness, interesting, you do think I'm evil!
I'm not sure whether to be flattered or upset or what. It's kinda cool, anyway!
Solomonoff induction is extraordinarily unhelpful, I think... that it is uncomputable is only one reason.
Because it's output is not human-readable being the other?
I mean, even if I've got a TARDIS to use as a halting oracle, an Inductive Turing Machine isn't going to output something I can actually use to make predictions about specific events such as "The black box gives you money under X, Y, and Z circumstances."
Well, the problem I was thinking of is "the universe is not a bit string." And any unbiased representation we can make of the universe as a bit string is going to be extremely large—much too large to do even sane sorts of computation with, never mind Solomonoff.
Maybe that's saying the same thing you did? I'm not sure...
Part of the motivation for the black box experiment is to show that the metaprobability approach breaks down in some cases.
Ah! I didn't quite pick up on that. I'll note that infinite regress problems aren't necessarily defeaters of an approach. Good minds that could fall into that trap implement a "Screw it, I'm going to bed" trigger to keep from wasting cycles even when using an otherwise helpful heuristic.
Maybe the thought experiment ought to have specified a time limit. Personally, I don't think enumerating things the boxes could possibly do would be helpful at all. Isn't there an easier approach?
Maybe, but I can't guarantee you won't get blown up by a black box with a bomb inside! As a friend, I would be furiously lending you my reasoning to help you make the best decision, worrying very little what minds better and faster than both of ours would be able to do.
It is, at the end of the day, just the General AI problem: Don't think too hard on brute-force but perfect methods or else you might skip a heuristic that could have gotten you an answer within the time limit! But when do you know whether the time limit is at that threshold? You could spend cycles on that too, but time is wasting! Time limit games presume that the participant has already underwent a lot of unintentional design (by evolution, history, past reflections, etc.). This is the "already in-motion" part which, frustratingly, cannot ever be optimal unless somebody on the outside designed you for it. It's a formal problem what source code performs best under what game. Being a source code involves taking the discussion we're having now and applying it the best you can, because that's what your source code does.
I can't guarantee you won't get blown up
Yes—this is part of what I'm driving at in this post! The kinds of problems that probability and decision theory work well for have a well-defined set of hypotheses, actions, and outcomes. Often the real world isn't like that. One point of the black box is that the hypothesis and outcome spaces are effectively unbounded. Trying to enumerate everything it could do isn't really feasible. That's one reason the uncertainty here is "Knightian" or "radical."
In fact, in the real world, "and then you get eaten by a black hole incoming near the speed of light" is always a possibility. Life comes with no guarantees at all.
Often in Knightian problems you are just screwed and there's nothing rational you can do. But in this case, again, I think there's a straightforward, simple, sensible approach (which so far no one has suggested...)
Going back to the basic question about the black box:
What is the probability of its giving you $2?
Too small to be worth considering. I might as well ask, what's the probability that I'll find $2 hidden half way up the nearest tree? Nothing has been claimed about the black box to specifically draw "it will pay you $2 for $1" out of hypothesis space.
Hmm... given that the previous several boxes have either paid $2 or done nothing, it seems like that primes the hypothesis that the next in the series also pays $2 or does nothing. (I'm not actually disagreeing, but doesn't that argument seem reasonable?)
View more: Next
= f037147d6e6c911a85753b9abdedda8d){kind=logo}
= 27d567bc2d48d9f40e9ccc20ea370b15)
= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Your link to Ap is broken:( overall, this was really interesting and understandable. Thank you.
Glad you liked the post! Thanks for pointing out the link problem. I've fixed it, for now. It links to a PDF of a file that's found in many places on the internet, but any one of them might be taken down at any time.