LESSWRONG
LW

All of David_Chapman's Comments + Replies

Building up to an Internal Family Systems model

Have you read Minsky's _Society of Mind_? It is an AI-flavored psychological model of subagents that draws heavily on psychotherapeutic ideas. It seems quite similar in flavor to what you propose here. It inspired generations of students at the MIT AI Lab (although attempts to code it never worked out).

1eggsyntax1y

Do you happen to recall where you got that information? I've wondered occasionally what later became of Minsky's approach; it's intuitively pretty compelling. I'd love to find a source of info on follow-up work.

1Kenny6y

I've read that book. One thing I think it's missing, if I'm remembering it correctly, is any interplay between 'bottom-up' and 'top-down' sub-agents. That seems to be a key dynamic à la perceptual control theory.

3Kaj_Sotala6y

I looked at the beginning of it a bit before writing this post, but at least the beginning of it gave the impression that its subagents were very low-level (IIRC, it started with an example of building a tower of blocks, or taking some similar physical action, using many different subagents) and overall it had a strong vibe of 80's AI, so then it didn't feel like the most useful thing to be reading.

My attempt to explain Looking, insight meditation, and enlightenment in non-mysterious terms

David_Chapman7y100

Quote from Richard Feynman explaining why there are no objects here.

I've begun a STEM-compatible attempt to explain a "no objectively-given objects" ontology in "Boundaries, objects, and connections." That's supposed to be the introduction to a book chapter that is extensively drafted but not yet polished enough to publish.

Really glad you are working on this also!

Looking and the no-self

David_Chapman7y80

I would not say that selves don't exist (although it's possible that I have done so somewhere, sloppily).

Rather, that selves are both nebulous and patterned ("empty forms," in Tantric terminology).

Probably the clearest summary of that I've written so far is "Selfness," which is supposed to be the introduction to a chapter of the Meaningness book that does not yet otherwise exist.

Renouncing the self is characteristic of Sutrayana.

Looking and the no-self

David_Chapman7y110

[FWOMP Summoned spirit appears in Kaj's chalk octogram. Gouts of eldritch flame, etc. Spirit squints around at unfamiliar environment bemusedly. Takes off glasses, holds them up to the candlelight, grimaces, wipes glasses on clothing, replaces on nose. Grunts. Speaks:]

Buddhism is a diverse family of religions, with distinct conceptions of enlightenment. These seem to be quite different and contradictory.

According to one classification of disparate doctrines, Buddhism can be divided into Vajrayana (Tantra plus Dzogchen) and Sutrayana (everything else, excep... (read more)

Probability, knowledge, and meta-probability

David_Chapman10y00

Glad you liked the post! Thanks for pointing out the link problem. I've fixed it, for now. It links to a PDF of a file that's found in many places on the internet, but any one of them might be taken down at any time.

Applying reinforcement learning theory to reduce felt temporal distance

David_Chapman11y90

A puzzling question is why your brain doesn't get this right automatically. In particular, deciding whether to gather some food before sleeping is an issue mammals have faced in the EEA for millions of years.

Temporal difference learning seems so basic that brains ought to implement it reasonably accurately. Any idea why we might do the wrong thing in this case?

0ThisSpaceAvailable11y

How is temporal difference learning basic? Do you think that if I give my dog a treat every morning if he obeyed my command to sit the previous day, that would teach him to sit? How would he connect those two events, out of all the events over the day?

7Kaj_Sotala11y

I'm guessing that it has to do with the kinds of "things" that are linked to a later consequence. For example, we seem to be pretty good at avoiding or frequenting the kinds of places where we tend to have negative or positive experiences. And we're also good at linking physical items or concrete actions to their consequences - like in Roko's example about the bills: But "not going to the store results in hunger the next morning" seems like a more abstract thing. The fact that it's the lack of an action, rather than the presence of one, seems particularly relevant. Neither the store nor the act of going there is something that's directly associated with getting hungry. If anything it's my earlier thought of possibly needing to go to the store... and I guess it's possible that to the extent that anything gets negatively reinforced, it's the act of me even considering it, since it's the only concrete action that my brain can directly link to the consequence! Also, if I do go to the store, there isn't any clear reward that would reinforce my behavior. The reward is simply that I won't be hungry the next morning... but that's not something that would be very out of the ordinary, for not-being-hungry is just the normal state of being. And being in a neutral state doesn't produce a reward. I guess that if I enjoyed food more, getting to eat could be more of a reward in itself. (I'm very sure that there exist mountains of literature on this very topic that could answer the question rather conclusively, but I don't have the energy to go do a lit search right now.)

0Kawoomba11y

A bird in the hand is worth two in the bush. Until you've become comparatively good at predicting the future (entails good models, which entails cognitive effort, which necessitates a reasonably developed cognitive architecture), an immediate benefit will often outweigh some nebulous possible future reward (in OP's parlance, value).

Probability and radical uncertainty

David_Chapman11y00

Are there any psychoactive gases or aerosols that drive you mad?

I suppose a psychedelic might push someone over the edge if they were sufficiently psychologically fragile. I don't know of any substances that specifically make people mad, though.

0Vaniver11y

I'm not a psychiatrist. Maybe? It looks like airborne transmission of prions might be possible, and along an unrelated path the box could go the Phineas Gage route.

Probability and radical uncertainty

David_Chapman11y00

One aspect of what I consider the correct solution is that the only question that needs to be answered is "do I think putting a coin in the box has positive or negative utility", and one can answer that without any guess about what it is actually going to do.

What is your base rate for boxes being able to drive you mad if you put a coin in them?

Can you imagine any mechanism whereby a box would drive you mad if you put a coin in it? (I can't.)

1dspeyer11y

Given that I'm inside a hypothetical situation proposed on lesswrong, the likelihood of being inside a Lovecraft crossover or something similar is about .001. Assuming a Lovecraft crossover, the likelihood of a box marked in eldritch runes containing some form of Far Realm portal is around .05. So say .0005 from that method, which is what was on my mind when I wrote that.

0Vaniver11y

Perhaps sticking a coin in it triggers the release of some psychoactive gas or aerosol?

Probability and radical uncertainty

David_Chapman11y00

Excellent! This is very much pointing in the direction of what I consider the correct general approach. I hadn't thought of what you suggest specifically, but it's an instance of the general category I had in mind.

Probability and radical uncertainty

David_Chapman11y20

Thanks for the encouragement! I have way too many half-completed writing projects, but this does seem an important point.

Probability and radical uncertainty

David_Chapman11y40

Oh, goodness, interesting, you do think I'm evil!

I'm not sure whether to be flattered or upset or what. It's kinda cool, anyway!

2Bayeslisk11y

I think that avatar-of-you-in-this-presented-scenario does not remotely have avatar-of-me-in-this-scenario's best interests at heart, yes.

Probability and radical uncertainty

David_Chapman11y30

Well, the problem I was thinking of is "the universe is not a bit string." And any unbiased representation we can make of the universe as a bit string is going to be extremely large—much too large to do even sane sorts of computation with, never mind Solomonoff.

Maybe that's saying the same thing you did? I'm not sure...

5torekp11y

Can you please give us a top level post at some point, be it in Discussion or Main, arguing that "the universe is not a bit string"? I find that very interesting, relevant, and plausible.

Probability and radical uncertainty

David_Chapman11y20

I can't guarantee you won't get blown up

Yes—this is part of what I'm driving at in this post! The kinds of problems that probability and decision theory work well for have a well-defined set of hypotheses, actions, and outcomes. Often the real world isn't like that. One point of the black box is that the hypothesis and outcome spaces are effectively unbounded. Trying to enumerate everything it could do isn't really feasible. That's one reason the uncertainty here is "Knightian" or "radical."

In fact, in the real world, "and then ... (read more)

3CoffeeStain11y

As you know, this attitude isn't particularly common 'round these parts, and while I fall mostly in the "Decision theory can account for everything" camp, there may still be a point there. "Rational" isn't really a category so much as a degree. Formally, it's a function on actions that somehow measures how much that action corresponds to the perfect decision-theoretic action. My impression is that somewhere there's Godelian consideration lurking, which is where the "Omega fines you exorbitantly for using TDT" thought experiment comes into play. That thought experiment never bothered me much, as it just is what it is: a problem where you are just screwed, and there's nothing rational you can do to improve your situation. You've already rightly programmed yourself to use TDT, and even your decision to stop using TDT would be made using TDT, and unless Omega is making exceptions for that particular choice (in which case you should self-modify to non-TDT), Omega is just a jerk that goes around fining rational people. In such situations, the words "rational" and "irrational" are less useful descriptors than just observing source code being executed. If you're formal about it using metric R, then you would be more R, but its correlation to "rational" wouldn't really be at point. So, I don't think the black box is really one of the situations I've described. It seems to me a decision theorist training herself to be more generally rational is in fact improving her odds at winning the black box game. All the approaches outlined so far do seem to also improve her odds. I don't think a better solution exists, and she will often lose if she lacks time to reflect. But the more rational she is, the more often she will win.

Probability and radical uncertainty

David_Chapman11y50

Hmm... given that the previous several boxes have either paid $2 or done nothing, it seems like that primes the hypothesis that the next in the series also pays $2 or does nothing. (I'm not actually disagreeing, but doesn't that argument seem reasonable?)

0Richard_Kennaway11y

Priming a hypothesis merely draws it to attention; it does not make it more likely. Every piece of spam, every con game, "primes the hypothesis" that it is genuine. It also "primes the hypothesis" that it is not. "Priming the hypothesis" is no more evidence than a purple giraffe is evidence of the blackness of crows. Explicltly avoiding saying that it does pay $2, and saying instead that it is "interesting", well, that pretty much stomps the "priming" into a stain on the sidewalk.

Probability and radical uncertainty

David_Chapman11y20

To answer this we engage our big amount of human knowledge about boxes and people who hand them to you.

Of comments so far, this comes closest to the answer I have in mind... for whatever that's worth!

Probability and radical uncertainty

David_Chapman11y00

Part of the motivation for the black box experiment is to show that the metaprobability approach breaks down in some cases. Maybe I ought to have made that clearer! The approach I would take to the black box does not rely on metaprobability, so let's set that aside.

So, your mind is already in motion, and you do have priors about black boxes. What do you think you ought to in this case? I don't want to waste your time with that... Maybe the thought experiment ought to have specified a time limit. Personally, I don't think enumerating things the boxes could possibly do would be helpful at all. Isn't there an easier approach?

2CoffeeStain11y

Ah! I didn't quite pick up on that. I'll note that infinite regress problems aren't necessarily defeaters of an approach. Good minds that could fall into that trap implement a "Screw it, I'm going to bed" trigger to keep from wasting cycles even when using an otherwise helpful heuristic. Maybe, but I can't guarantee you won't get blown up by a black box with a bomb inside! As a friend, I would be furiously lending you my reasoning to help you make the best decision, worrying very little what minds better and faster than both of ours would be able to do. It is, at the end of the day, just the General AI problem: Don't think too hard on brute-force but perfect methods or else you might skip a heuristic that could have gotten you an answer within the time limit! But when do you know whether the time limit is at that threshold? You could spend cycles on that too, but time is wasting! Time limit games presume that the participant has already underwent a lot of unintentional design (by evolution, history, past reflections, etc.). This is the "already in-motion" part which, frustratingly, cannot ever be optimal unless somebody on the outside designed you for it. It's a formal problem what source code performs best under what game. Being a source code involves taking the discussion we're having now and applying it the best you can, because that's what your source code does.

Probability and radical uncertainty

David_Chapman11y10

The evidence that I didn't select it at random was my saying “I find this one particularly interesting.”

I also claimed that "I'm probably not that evil." Of course, I might be lying about that! Still, that's a fact that ought to go into your Bayesian evaluation, no?

3Bayeslisk11y

"Interesting" tends to mean "whatever it would be, it does that more" in the context of possibly psuedo-Faustian bargains and signals of probable deceit. From what I know, I do not start with reason to trust you, and the evidence found in the OP suggests that I should update the probability that you are concealing information updating on which would lead me not to use the black box to "much higher".

Probability and radical uncertainty

David_Chapman11y30

Yes, I'm not at all committed to the metaprobability approach. In fact, I concocted the black box example specifically to show its limitations!

Solomonoff induction is extraordinarily unhelpful, I think... that it is uncomputable is only one reason.

I think there's a fairly simple and straightforward strategy to address the black box problem, which has not been mentioned so far...

4[anonymous]11y

Because it's output is not human-readable being the other? I mean, even if I've got a TARDIS to use as a halting oracle, an Inductive Turing Machine isn't going to output something I can actually use to make predictions about specific events such as "The black box gives you money under X, Y, and Z circumstances."

3Richard_Kennaway11y

Going back to the basic question about the black box: Too small to be worth considering. I might as well ask, what's the probability that I'll find $2 hidden half way up the nearest tree? Nothing has been claimed about the black box to specifically draw "it will pay you $2 for $1" out of hypothesis space.

Probability and radical uncertainty

David_Chapman11y20

That's good, yes!

How would you assign a probability to that?

4CoffeeStain11y

"How often do listing sorts of problems with some reasonable considerations result in an answer of 'None of the above' for me?" If "reasonable considerations" are not available, then we can still: "How often did listing sorts of problems with no other information available result in an answer of 'None of the above' for me?" Even if we suppose that maybe this problem bears no resemblance to any previously encountered problem, we can still (because the fact that it bears no resemblance is itself a signifier): "How often did problems I'd encountered for the first time have an answer I never thought of?"

6ialdabaoth11y

Ideally, by looking a the number of times that I've experienced out-of-context problems in the past. You can optimize further by creating models that predict the base amount of novelty in your current environment - if you have reason to believe that your current environment is more unusual / novel than normal, increase your assigned "none of the above" proportionally. (And conversely, whenever evidence triggers the creation of a new top-level heading, that top-level heading's probability should get sliced out of the "none of the above", but the fact that you had to create a top-level heading should be used as evidence that you're in a novel environment, thus slightly increasing ALL "none of the above" categories. If you're using hard-coded heuristics instead of actually computing probability tables, this might come out as a form of hypervigilance and/or curiosity triggered by novel stimulus.)

Probability and radical uncertainty

David_Chapman11y20

So... you think I am probably evil, then? :-)

I gave you the box (in the thought experiment). I may not have selected it from Thingspace at random!

In fact, there's strong evidence in the text of the OP that I didn't...

4Bayeslisk11y

I am pattern-matching from fiction on "black box with evil-looking inscriptions on it". Those do not tend to end well for anyone. Also, what do you mean by strong evidence against that the box is less harmful than a given random object from Thingspace? I can /barely sort of/ see "not a random object from Thingspace"; I cannot see "EV(U(spoopy creppy black box)) > EV(U(object from Thingspace))".

Probability and radical uncertainty

David_Chapman11y50

This is interesting—it seems like the project here would be to construct a universal, hierarchical ontology of every possible thing a device could do? This seems like a very big job... how would you know you hadn't left out important possibilities? How would you go about assigning probabilities?

(The approach I have in mind is simpler...)

2dspeyer11y

A universal ontology is intractable, no argument there. As is a tree of (meta)*-probabilities. My point was about how to regard the problem. As for an actual solution, we start with propositions like "this box has a nontrivial potential to kill, injure or madden me.". I can find a probability for that based on my knowledge of you and on what you've said. If the probability is small enough, I can subdivide that by considering another proposition.

3[anonymous]11y

I'm currently mostly wondering how I get the black box to do anything at all, and particularly how I can protect myself against the dangerous things it might be feasible for an eldritch box to do.

6ialdabaoth11y

At least one of the top-level headings should be a catch-all "None of the above", which represents your estimated probability that you left something out.

Probability and radical uncertainty

David_Chapman11y00

Well, regardless of the value of metaprobability, or its lack of value, in the case of the black box, it doesn't seem to offer any help in finding a decision strategy. (I find it helpful in understanding the problem, but not in formulating an answer.)

How would you go about choosing a strategy for the black box?

5CoffeeStain11y

My LessWrongian answer is that I would ask my mind that was created already in motion what the probability is, then refine it with as many further reflections as I can come up with. Embody an AI long enough in this world, and it too will have priors about black boxes, except that reporting that probability in the form of a number is inherent to its source code rather than strange and otherworldly like it is for us. The point that was made in that article (and in the Metaethics sequence as a whole) is that the only mind you have to solve a problem is the one that you have, and you will inevitably use it to solve problems unoptimally, where "unoptimal" if taken strictly means everything anybody has ever done. The reflection part of this is important, as it's the only thing we have control over, and I suppose could involve discussions about metaprobabilities. It doesn't really do it for me though, although I'm only just a single point in the mind design space. To me, metaprobability seems isomorphic to a collection of reducible considerations, and so doesn't seem like a useful shortcut or abstraction. My particular strategy for reflection would be something like that in dspeyer's comment, things such as reasoning about the source of the box, possibilities for what could be in the box that I might reasonably expect to be there. Depending on how much time I have, I'd be very systematic about it, listing out possibilities, solving infinite series on expected value, etc.

Probability and radical uncertainty

David_Chapman11y20

Well, I hope to continue the sequence... I ended this article with a question, or puzzle, or homework problem, though. Any thoughts about it?

0Transfuturist10y

I hope you continue the sequence as well. :V

2Bayeslisk11y

IMO the correct response is to run like hell from the box. In Thingspace, most things are very unfriendly, in much the same way that most of Mindspace contains unfriendly AIs.

Probability and radical uncertainty

David_Chapman11y10

So, how would you analyze this problem, more specifically? What do you think the optimal strategy is?

Manfred11y100

The problem of what to expect from the black box?

I'd think about it like this: suppose that I hand you a box with a slot in it. What do you expect to happen if you put a quarter into the slot?

To answer this we engage our big amount of human knowledge about boxes and people who hand them to you. It's very likely that nothing at all will happen, but I've also seen plenty of boxes that also emit sound, or gumballs, or temporary tattoos, or sometimes more quarters. But suppose that I have previously handed you a box that emits more quarters sometimes when y... (read more)

Welcome to Less Wrong! (5th thread, March 2013)

David_Chapman11y20

Hi, I have a site tech question. (Sorry if this is the wrong place to post that!—I couldn't find any other.)

I can't find a way to get email notifications of comment replies (i.e. when my inbox icon goes red). If there is one, how do I turn it on?

If there isn't one, is that a deliberate design feature, or a limitation of the software, or...?

Thanks (and thanks especially to whoever does the system maintenance here—it must be a big job.)

1TheOtherDave11y

There's no way I know of to get email notifications, and I've looked enough that I'm pretty confident one doesn't exist. No idea if it's a deliberate choice or a software limitation.

Probability, knowledge, and meta-probability

David_Chapman11y20

Then why use it instead of learning the standard terms and using those?

The standard term is A_p, which seemed unnecessarily obscure.

Re the figure, see the discussion here.

(Sorry to be slow to reply to this; I got busy and didn't check my LW inbox for more than a month.)

Probability, knowledge, and meta-probability

David_Chapman11y00

Thank you very much—link fixed!

That's a really funny quote!

Multi-armed bandit problems were intractable during WWII probably mainly because computers weren't available yet. In many cases, the best approach is brute force simulation. That's the way I would approach the "blue box" problem (because I'm lazy).

But exact approaches have also been found: "Burnetas AN and Katehakis MN (1996) also provided an explicit solution for the important case in which the distributions of outcomes follow arbitrary (i.e., nonparametric) discrete, univariate distributions." The blue box problem is within that class.

Probability, knowledge, and meta-probability

David_Chapman11y20

Thanks, yes! I.e. who is this "everyone else," and where do they treat it the same way Jaynes does? I'm not aware of any examples, but I have only a basic knowledge of probability theory.

It's certainly possible that this approach is common, but Jaynes wasn't ignorant, and he seemed to think it was a new and unusual and maybe controversial idea, so I kind of doubt it.

Also, I should say that I have no dog in this fight at all; I'm not advocating "Jaynes is the greatest thing since sliced bread", for example. (Although that does seem to be the opinion of some LW writers.)

Probability, knowledge, and meta-probability

David_Chapman11y00

Can you point me at some other similar treatments of the same problem? Thanks!

1Douglas_Knight11y

I ask you for a different treatment, so you ask me for a similar treatment? No, I don't see the point. Doesn't my request make sense, regardless of whether we agree on what is similar or different?

Probability, knowledge, and meta-probability

David_Chapman11y20

Thanks, that's really funny! "On the other hand" is my general approach to life, so I'm happy to argue with myself.

And yes, I'm steelmanning. I think this approach is an excellent one in some cases; it will break down in others. I'll present a first one in the next article. It's another box you can put coins in that (I'll claim) can't usefully be modeled in this way.

Here's the quote from Jaynes, by the way:

What are we doing here? It seems almost as if we are talking about the ‘probability of a probability’. Pending a better understanding of wha

David_Chapman11y00

Yes, meta-probabilities are probabilities, although somewhat odd ones; they obey the normal rules of probability. Jaynes discusses this in his Chapter 18; his discussion there is worth a read.

The statement "probability estimates are not, by themselves, adequate to make rational decisions" was meant to describe the entire sequence, not this article.

I've revised the first paragraph of the article, since it seems to have misled many readers. I hope the point is clearer now!

6roystgnr11y

I'm looking forward to the rest of your sequence, thanks! I was recently reading through a month-old blog post where one lousy comment was arguing against a strawman of Bayesian reasoning wherein you deal with probabilities by "mushing them all into a single number". I immediately recollected that the latest thing I saw on LessWrong was a fantastic summary of how you can treat mixed uncertainty as a probability-distribution-of-probability-distributions. I considered posting a belated link in reply, until I discovered that the lousy comment was written by David Chapman and the fantastic summary was written by David_Chapman. I'm not sure if later you're going to go off the rails or change my mind or what, but so far this looks like one of the greatest attempts at "steelmanning" that I've ever seen on the internet.

Probability, knowledge, and meta-probability

David_Chapman11y20

Are you claiming there's no prior distribution over sequences which reflects our knowledge?

No. Well, not so long as we're allowed to take our own actions into account!

I want to emphasize—since many commenters seem to have mistaken me on this—that there's an obvious, correct solution to this problem (which I made explicit in the OP). I deliberately made the problem as simple as possible in order to present the A_p framework clearly.

Are we talking about the Laplace vs. fair coins?

Not sure what you are asking here, sorry...

7Eliezer Yudkowsky11y

Heh! Yes, traditional causal models have structure beyond what is present in the corresponding probability distribution over those models, though this has to do with computing counterfactuals rather than meta-probability or estimate instability. Work continues at MIRI decision theory workshops on the search for ways to turn some of this back into probability, but yes, in my world causal models are things we assign probabilities to, over and beyond probabilities we assign to joint collections of events. They are still models of reality to which a probability is assigned, though. (See Judea Pearl's "Why I Am Only A Half-Bayesian".)

Probability, knowledge, and meta-probability

David_Chapman11y30

We could also try to summarize some features of such epistemic states by talking about the instability of estimates - the degree to which they are easily updated by knowledge of other events

Yes, this is Jaynes' A_p approach.

this will be a derived feature of the probability distribution, rather than an ontologically extra feature of probability.

I'm not sure I follow this. There is no prior distribution for the per-coin payout probabilities that can accurately reflect all our knowledge.

I reject that this is a good reason for probability theorists to

... (read more)

1Eliezer Yudkowsky11y

Are we talking about the Laplace vs. fair coins? Are you claiming there's no prior distribution over sequences which reflects our knowledge? If so I think you are wrong as a matter of math.

Probability, knowledge, and meta-probability

David_Chapman11y40

So, let me try again to explain why I think this is missing the point... I wrote "a single probability value fails to capture everything you know about an uncertain event." Maybe "simple" would have been better than "single"?

The point is that you can't solve this problem without somehow reasoning about probabilities of probabilities. You can solve it by reasoning about the expected value of different strategies. (I said so in the OP; I constructed the example to make this the obviously correct approach.) But those strategies c... (read more)

Probability, knowledge, and meta-probability

David_Chapman11y20

Glad you liked it!

I also get "stop after two losses," although my numbers come out slightly differently. However, I suck at this sort of problem, so it's quite likely I've got it wrong.

My temptation would be to solve it numerically (by brute force), i.e. code up a simulation and run it a million times and get the answer by seeing which strategy does best. Often that's the right approach. However, sometimes you can't simulate, and an analytical (exact, a priori) answer is better.

I think you are right about the sportsball case! I've updated my meta... (read more)

0Vaniver11y

The wikipedia article on the Beta distribution has a good discussion of possible priors to use. The Jeffreys prior is probably the one I'd use for Sportsball, but the Bayes-Laplace prior is generally acceptable as a representation of ignorance. The example I like to give is the uncertain digital coin- I generate some double p between 0 and 1 using a random number generator, and then write a function "flip" which generates another double, and compares it to p. This is analogous to your blue box, and if you're confident in the RNG means you have a tight meta-meta-probability curve, which justifies the uniform prior. Yeah, that seems like a good candidate for the Haldane prior to me.

Probability, knowledge, and meta-probability

David_Chapman11y20

Thanks! Fixed.

Probability, knowledge, and meta-probability

David_Chapman11y30

Yup, it's definitely wrong! I was hoping no one would notice. I thought it would be a distraction to explain why the two are different (if that's not obvious), and also I didn't want to figure out exactly what the right math was to feed to my plotting package for this case. (Is the correct form of the curve for the p=0 case obvious to you? It wasn't obvious to me, but this isn't my area of expertise...)

2Vaniver11y

I would have left it unexplained in the post, and then explained it in the comments when the first person asked about it. In my experience, causally remarked semi-obvious true facts like that ("why are these two not equally tall?" "Because the area underneath is what matters") are useful at convincing people of technical ability. I probably would have gone with the point mass approximation- i.e. a big circle at (0,.5), a line down to (0,0), a line over to (.9,0), and then a line up to a big circle at (.9,.5), then also a line from (.9,0) to (1,0). Using the Gaussian mixtures, though, I'd probably give them the same variance and just give the left one twice the weight of the right one, center them at 0 and .9, and then display only between 0 and 1. Using the pure functional form, that would look something like 2exp(-x^2/v)+exp(-(x-.9)^2/v). Now, this is assuming we have some sort of Gaussian prior. We could also have a beta prior, which is conjugate to the binomial distribution, which is nice because that fits our testbed. Gaussian might be appropriate because we've actually opened the system up and we think the measurement system it uses has Gaussian noise. I'm not sure I agree with the claim that the variance is the same; you could probably assert that chance the left one will pay out is 0 to arbitrarily high precision, and it seems likely the variance would depend on the number of plugs filled. That said, this doesn't have much impact, and saying "we'll approximate away the meta-meta-probability to simplify this example" seems like it goes against your general point, and is thus inadvisable.

Probability, knowledge, and meta-probability

David_Chapman11y00

Decisions are made on the basis of expected value, not probability.

Yes, that's the point here!

your analysis of the first bet ignores the value of the information gained from it in executing your options for further play thereafter.

By "the first bet" I take it that you mean "your first opportunity to put a coin in a green box" (rather than meaning "brown box").

My analysis of that was "you should put some coins in the box", exactly because of the information gain.

This statement indicates a lack of understandin

David_Chapman11y00

I don't think is demonstrated at all by this example.

Yes, I see your point (although I don't altogether agree). But, again, what I'm doing here is setting up analytical apparatus that will be helpful for more difficult cases later.

In the mean time, the LW posts I pointed to here may motivate more strongly the claim that probability alone is an insufficient guide to action.

Probability, knowledge, and meta-probability

David_Chapman11y20

I'm sure you know more about this than I do! Based on a quick Wiki check, I suspect that formally the A_p are one type of hyperprior, but not all hyperpriors are A_p (a/k/a metaprobabilities).

Hyperparameters are used in Bayesian sensitivity analysis, a/k/a "Robust Bayesian Analysis", which I recently accidentally reinvented here. I might write more about that later in this sequence.

0alex_zag_al11y

Yeah - from what I've seen, something mathematically equivalent to A_p distributions are commonly used, but that's not what they're called. Like, I think you might call the case in this problem "a Bernoulli random variable with an unknown parameter". (The Bernoulli random variable being 1 if it gives you $2, 0 if it gives you $0). And then the hyperprior would be the probability distribution of that parameter, I guess? I haven't really heard that word before. ET Jaynes, of course, would never talk like this because the idea of a random quantity existing in the real world is a mind projection fallacy. Thus, no "random variables". So he uses the A_p distribution as a way of thinking about the same math without the idea of randomness. Jaynes's A_p in this case corresponds exactly to the more traditional "the parameter of the Bernoulli random variable is p". (btw I have a purely mathematical question about the A_p distribution chapter, which I posted to the open thread: http://lesswrong.com/lw/ii6/open_thread_september_28_2013/9pbn if you know the answer I'd really appreciate it if you told me)

8Vaniver11y

When you use an underscore in a name, make sure to escape it first, like so: I suspect that formally the A\_p are one type of [hyperprior](http://en.wikipedia.org/wiki/Hyperprior), but not all hyperpriors are A\_p (a/k/a metaprobabilities). (This is necessary because underscores are yet another way to make things italic, and only applies to comments, as posts use different formatting.)

Probability, knowledge, and meta-probability

David_Chapman11y00

It may be helpful to read some related posts (linked by lukeprog in a comment on this post): Estimate stability, and Model Stability in Intervention Assessment, which comments on Why We Can't Take Expected Value Estimates Literally (Even When They're Unbiased). The first of those motivates the A_p (meta-probability) approach, the second uses it, and the third explains intuitively why it's important in practice.

Probability, knowledge, and meta-probability

David_Chapman11y00

Jeremy, I think the apparent disagreement here is due to unclarity about what the point of my argument was. The point was not that this situation can't be analyzed with decision theory; it certainly can, and I did so. The point is that different decisions have to be made in two situations where the probabilities are the same.

Your discussion seems to equate "probability" with "utility", and the whole point of the example is that, in this case, they are not the same.

6jeremysalwen11y

I guess my position is thus: While there are sets of probabilities which by themselves are not adequate to capture the information about a decision, there always is a set of probabilities which is adequate to capture the information about a decision. In that sense I do not see your article as an argument against using probabilities to represent decision information, but rather a reminder to use the correct set of probabilities.

Probability, knowledge, and meta-probability

David_Chapman11y30

Thanks, Jonathan, yes, that's how I understand it.

Jaynes' discussion motivates A_p as an efficiency hack that allows you to save memory by forgetting some details. That's cool, although not the point I'm trying to make here.

Probability, knowledge, and meta-probability

David_Chapman11y20

Luke, thank you for these pointers! I've read some of them, and have the rest open in tabs to read soon.

Probability, knowledge, and meta-probability

David_Chapman11y10

Jeremy, thank you for this. To be clear, I wasn't suggesting that meta-probability is the solution. It's a solution. I chose it because I plan to use this framework in later articles, where it will (I hope) be particularly illuminating.

I would take issue with the first section of this article in which you establish single probability (expected utility) calculations as insufficient for the problem.

I don't think it's correct to equate probability with expected utility, as you seem to do here. The probability of a payout is the same in the two situations.... (read more)

3jeremysalwen11y

Hmmm. I was equating them as part of the standard technique of calculating the probability of outcomes from your actions, and then from there multiplying by the utilities of the outcomes and summing to find the expected utility of a given action. I think it's just a question of what you think the error is in the original calculation. I find the error to be the conflation of "payout" (as in immediate reward from inserting the coin) with "payout" (as in the expected reward from your action including short term and long-term rewards). It seems to me that you are saying that you can't look at the immediate probability of payout which I agree with. But you seem to ignore the obvious solution of considering the probability of total payout, including considerations about your strategy. In that case, you really do have a single probability representing the likelihood of a single outcome, and you do get the correct answer. So I don't see where the issue with using a single probability comes from. It seems to me an issue with using the wrong single probability. And especially troubling is that you seem to agree that using direct probabilities to calculate the single probability of each outcome and then weighing them by desirability will give you the correct answer, but then you say which may be true, but I don't think is demonstrated at all by this example. Thank you for further explaining your thinking.

Welcome to Less Wrong! (5th thread, March 2013)

David_Chapman11y230

Hi!

I’ve been interested in how to think well since early childhood. When I was about ten, I read a book about cybernetics. (This was in the Oligocene, when “cybernetics” had only recently gone extinct.) It gave simple introductions to probability theory, game theory, information theory, boolean switching logic, control theory, and neural networks. This was definitely the coolest stuff ever.

I went on to MIT, and got an undergraduate degree in math, specializing in mathematical logic and the theory of computation—fields that grew out of philosophical investi... (read more)

What is the best paper explaining the superiority of Bayesianism over frequentism?

David_Chapman12y20

Can you recommend an explanation of the complete class theorem(s)? Preferably online. I've been googling pretty hard and I've turned up almost nothing. I'd like to understand what conditions they start from (suspecting that maybe the result is not quite as strong as "Bayes Rules!"). I've found only one paper, which basically said "what Wald proved is extremely difficult to understand, and probably not what you wanted."

Thank you very much!

5jsteinhardt12y

Maybe try this one? Let me know if that helps or if you're looking for something different. The complete class theorem states, informally: any Pareto optimal decision rule is a Bayesian decision rule (i.e. it can be obtained by choosing some prior, observing data, and then maximizing expected utility relative to the posterior). Roughly, the argument is that if I have a collection W of possible worlds that I could be in, and a value U(w) to taking a particular action in world w, then any Pareto optimal strategy implicitly assigns an "importance" p(w) to each world, and takes the action that maximizes the sum of p(w)*U(w). We can then show that this is equivalent to using the Bayesian decision rule with p(w) as the prior over W. The main thing needed to formalize this argument is the separating hyperplane theorem, which is what the linked paper does.

Grad Student Advice Repository

David_Chapman12y40

A collection of collections of advice for graduate students! http://vlsicad.ucsd.edu/Research/Advice/

Grad Student Advice Repository

David_Chapman12y50

A collection of advice for graduate students I put together some time ago: http://www.cs.indiana.edu/mit.research.how.to.html

It was meant specifically for people at the MIT AI Lab, but much of it is applicable to other STEM fields.

How to Be Happy

David_Chapman14y80

Regarding the development of agreeableness/empathy: there are meditation techniques specifically intended to do this. (They are variously called "Metta", "Lojong", "Tonglen", or (yuck) "loving kindness meditation"; all of which are pretty similar.) These originate in Mahayana Buddhism, but don't have any specifically religious content. They are often taught in conjunction with mindfulness meditation.

I don't know whether there have been any serious studies on these methods, but anecdotally they are highly effective... (read more)

1TheOtherDave14y

(nods) I began a Metta practice for mood management after my stroke (among a variety of other things) and found it very helpful. I still pull it out from time to time when I'm feeling particularly isolated. The usual caveats about other-optimizing apply, but within those limits I endorse this.

2lukeprog14y

Of course I'm keeping my eye out for literature on improving empathy. All the reviews I found so far said that we're not sure how to do that yet, because the studies do not give strong and clear results. Most of the literature is about trying to train medical workers to have empathy.