Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: gjm 29 April 2014 09:59:05PM 1 point [-]

The equilibrium probability might not be well defined. (E.g., if for whatever reason you form a sufficiently firm intention to falsify whatever the oracle tells you.)

And yes, if the oracle tells you something about your own future actions -- which it has to, to give you an equilibrium probability -- it's unsurprising that you're going to feel a loss of freedom. Either that, or disbelieve the oracle.

Comment author: MichaelBishop 30 April 2014 01:16:23PM 0 points [-]

What does it mean for a probability not to be well defined in this context? I mean, I think I share the intuition, but I'm not really comfortable with it either. Doesn't it seem strange that a probability could be well defined until I start learning more about it and trying to change it? How little do I have to care about the probability before it becomes well defined again?

Comment author: gjm 29 April 2014 04:54:06PM 2 points [-]

Part of my confusion is that knowing the probability I will lose my job seems certain to affect the probability that I lose my job.

Yes, and this may make the question you ask the oracle ill-posed. But you can avoid this while still making the oracle about as useful: it will tell you the probability that you lose your job in the absence of a specific response to what the oracle tells you.

Alternatively, to reduce those feedback effects we could adjust the question to reduce your influence over the thing you're being given information about. So, suppose you know that your job performance is good, and appreciated by your employer, and have no reason to think that's likely to change, but your job is at risk for reasons that have nothing to do with your performance: you're at a startup that might fail to find a good enough market, or a hedge fund that's taking risks that might wipe it out, or you're a political representative for a party that may be swept out of power on account of decisions taken by people other than you. If you knew everything relevant about the world you'd see that the probability of such a failure is either 0 or 50%, but in fact you have no idea, and the oracle will tell you which.

In either case we get back to something nearer to a pure value-of-information problem.

So, taking the "exogenous failure" version of the second approach, my answer to your question is something like this: If I lose my job with no warning, I guess it might take three months to find another comparably good job; if I have plenty of warning, I can line something up faster. I might pay the equivalent of ~ 1 month's take-home pay for the information. But this is still an answer based on the possibility of making a bad prediction not come to pass after all. If all I get is some advance warning that I'm going to lose my job without warning (this is reminding me of the paradox of the unexpected hanging...) then it's less useful; let's say ~ 2 weeks' pay. Note that these figures would all increase, perhaps by a lot, if my estimate of my re-employability were lower.

it appears I getting an offer to reduce the actual risk

Yes, I don't think this is a VoI problem as posed. But again we can make it one by modifying it. You have an estimate: your earnings over the next 10 years will be normally distributed with mean M and standard deviation S. The oracle will, in exchange for your payment, give you a new value of M (about which you are currently quite uncertain) along with a new smaller value of S. Your present uncertainty about the new M corresponds to the reduction in S.

Unfortunately you still have the problem from the first thought experiment, which I propose remedying in the same way: either the oracle gives you a prediction conditional on your acting as you would have without her help (so now if the income figure is depressingly low, that suggests you aren't going to get the promotion you hoped for and you should consider looking for another job elsewhere (etc.) instead), or else you are for some reason unable to do anything to make bad predictions not come true.

Let me try to answer this question too, now it's been made more answerable. Here's a simplified version of the fisrt of those options: before asking the oracle I predict income M-S or M+S with equal probability (std dev is S). The oracle gives me better probabilities so as to halve the standard deviation, which means 93.3% for one and 6.7% for the other. On the occasions when it gives me a "bad" prediction (it says M-S with probability 93.3%) I switch to plan B, which (optimistically) is about as good a priori as what I was previously intending to do, which means it restores the probabilities to 50%. So (my mean - M) has gone from zero to 1/2 (0.933 S - 0.067 S) + 1/2.0 = 0.433 S. In practice my plan B is probably worse a priori than my plan A, and I suspect other simplifications I've made have also made the oracle's information more valuable, so the right figure is probably somewhat less than 0.433 S (note: S here is our "three years' worth of income"). My gut feeling is that it's quite a lot less, e.g. because when the oracle gives you bad news you don't know which aspects of your current plans are responsible for it. The right answer might be more like 0.1 S, or ~ 4 months' income.

In the second version (where I'm somehow prohibited from doing anything to fix the problem, if the oracle gives a low estimate of my future earnings), again the value of the information is obviously lower. I suppose it would be useful information for pension planning. As with the first question, I'm handwavily going to estimate that the benefit is half as much in this case, so 2 months' income.

I should add that these figures for the second problem still feel rather high to me. If an oracle actually offered me that information, I am not at all sure I'd feel willing to pay even two months' income for it.

Comment author: MichaelBishop 29 April 2014 09:22:43PM 0 points [-]

+1 and many thanks for wading into this with me... I've been working all day and I'm still at work so can't necessarily respond in full...

I agree that these problems are a lot simpler if reducing my uncertainty about X cannot help me affect X. This is not a minor class of problems. I'd love to have better information for a lot of problems in this class. That said, many of the problems that it seems most worthwhile for me to spend my time and money reducing my uncertainty about are of the type where I have a non-trivial role in how they play out. Assuming I do have some causal power over X, I think I'd pay a lot more to know the "equilibrium" probability of X after I've digested the information the oracle gave me - anything else seems like stale information... but learning that equilibrium probability seems weird as well. If I'm surprised by what the oracle says, then I imagine I'd ask myself questions like: how am I likely to react in regard to this information... what was the probability before I knew this information such that the current probability is what it is... It feels like I'm losing freedom... to what extent is the experience of uncertainty tied to the experience of freedom?

The Value of Uncertainty Reduction - references to academic literature appreciated

2 MichaelBishop 29 April 2014 01:51PM

My intuition (probably widely shared) suggests that uncertainty about the future is stress inducing and reducing uncertainty about the future is helpful because it allows us to plan. So I started trying to invent thought experiments that would begin to help me quantify how much I (and others) value uncertainty reduction... and then I began to get confused. Below I'll share two examples focused on knowledge about the next five years of one's career but similar psychological/philosophical issues would arise in many other contexts.

Thought Experiment #1: the risk of job loss
Imagine the true probability you involuntarily lose your job at some point in the next 5 years is either 0% or 50% and that your current best guess is that you have a 25% chance of losing your job. For a price, an oracle will tell you whether the truth is 0% of 50%. How much will you pay?

If you think you can answer this question, please do so. Part of my confusion is that knowing the probability I will lose my job seems certain to affect the probability that I lose my job. If you told me the probability was 50% then I'd do a combination of working overtime and looking for other jobs that should reduce that probability, and if the probability remains 50% then I'm in a much less pleasant situation than I would be in the case that the probability is 50% but only because I'm assuming its a more reasonable 25%.

Thought Experiment #2: uncertainty about future earnings
Imagine your estimate of your total income over the next 10 years is unbiased, and that the random error in your estimate is normally distributed. (Admittedly a normally distributed error term is unrealistic in this problem but bear with me for simplicity). What's a reasonable standard deviation? Let's say 3 years worth of income. How much would you pay to reduce that standard deviation to 1.5 years of income?

Once again, go ahead and to answer this if you can, but I've got myself confused here as well... I'm trying to get at the present value of reducing uncertainty about the future, but in this example it appears I getting an offer to reduce the actual risk of *experiencing* a much lower than expected income at the expense of reducing the chances that I make a much higher than expected income, not just reducing uncertainty.

Any insight into what's going on with my thought experiments would be greatly appreciated. I see some parallels between them and Newcomb's Paradox, but I'm not sure what to make of Newcomb's Paradox either. If people have relevant references to the philosophy literature that's great...
relevant references to judgment and decision-making or economics literature would be even better.

Seeking examples of people smarter than me who got hung up

10 MichaelBishop 13 January 2013 04:40PM

I'm looking for historical examples of scientists who were 
a) very intelligent and still
b) continued to put themselves behind a theory in their discipline long after it was rejected.  Maybe they got too attached to it, refused to be wrong, got emotional, but they somehow let their hangups get in the way.

Maybe I'm being too demanding, but if you can resist, give me fewer cranks, pseudoscience, and wierd sociopolitical commitments and more theories that were credible until they became incredible to all but their big fancy until-then-respected proponent.

To get you started: 
* Fred Hoyle against the Big Bang
* Lord Kelvin and Hoyle on microbes from spce
* Tesla against relativity and other chunks of modern physics.
* Heaviside against relativity
* George Gaylord Simpson against plate tectonics
* Newton on alchemy


Posted on behalf of a friend. Thanks.

Comment author: gwern 05 May 2012 04:32:22PM 2 points [-]
Comment author: MichaelBishop 05 May 2012 08:18:22PM 1 point [-]

hmmm, I guess I missed that. Should I remove this post?

Test your forecasting ability, contribute to the science of human judgment

3 MichaelBishop 05 May 2012 03:07PM

As XFrequentist mentioned last August, "Intelligence Advanced Research Project Activity (IARPA) with the goal of improving forecasting methods for global events of national (US) interest. One of the teams (The Good Judgement Team) is recruiting volunteers to have their forecasts tracked. Volunteers will receive an annual honorarium ($150), and it appears there will be ongoing training to improve one's forecast accuracy (not sure exactly what form this will take)."

You can pre-register here.

Last year, approximately 2400 forecasters were assigned to one of eight experimental conditions.  I was the #1 forecaster in my condition.  It was fun, and I learned a lot, and eventually they are going to give me a public link so that I can brag about this until the end of time.  I'm participating again this year, though I plan to regress towards the mean.

I'll share the same info XFrequentist did last year below the fold because I think it's all still relevant.

continue reading »
Comment author: MichaelBishop 29 March 2012 03:04:44PM 0 points [-]

Economist Jeff Ely recently blogged an interesting example of a slippery slope. http://cheaptalk.org/2012/03/27/the-slippery-slope/

Comment author: gwern 28 March 2012 05:10:56PM *  6 points [-]

Surely more productive industrial researchers are generally paid more. Many firms even give explicit bonuses on a per patent basis.

Yes, but the bonuses I've heard of are in the hundreds to thousands of dollars range, at companies committed to patenting like IBM. This isn't going to make a big difference to lifetime incomes where the range is 1-3 million dollars although the data may be rich enough to spot these effects (and how many patents is even '4x'? 4 patents on average per person?), and I suspect these bonuses come at the expense of salaries & benefits. (I know that's how I'd regard it as a manager: shifting risk from the company to the employee.)

And I think you're forgetting that income did increase with each standard deviation by an amount somewhat comparable to my suggested numbers for patents, so we're not explaining why IQ did not increase income whatsoever, but why it increased it relatively little, why the patenters apparently captured relatively little of the value.

Comment author: MichaelBishop 28 March 2012 05:38:57PM 2 points [-]

Woh, I did allow myself to misread/misremember your initial comment a bit so I'll dial it back slightly. The fact that even at the highest levels IQ is still positively correlated to income is important, and its what I would have expected, so the overall story does not undermine my support for the hypothesis that at the highest IQ levels, higher IQ individuals produce more positive externalities. I apologize for getting a bit sloppy there.

I would guess that if you had data from people with the same job description at the same company the correlation between IQ, patents, and income would be even higher.

Comment author: NancyLebovitz 28 March 2012 04:36:01PM 0 points [-]

I've noticed that if I notice someone online as civilized and intelligent, the odds seem rather high that I'll be seeing them writing about having an ongoing problem with depression within months.

This doesn't mean that everyone I like (online or off) is depressed, but it seems like a lot. The thing is, I don't know whether the proportion is high compared to the general population, or whether depression and intelligence are correlated. (Some people have suggested this as an explanation for what I think I've noticed.)

I wonder whether there's a correlation between depression and being conflict averse.

Comment author: MichaelBishop 28 March 2012 05:07:13PM 0 points [-]

I wonder whether there's a correlation between depression and being conflict averse. I would guess that there is, and I'm sure there has been at least some academic study of it. This doesn't really address the issue, but its related.

I also think that keeping a blog or writing in odd corners of the internet may be associated with, possibly even caused by, depression.

Comment author: gwern 28 March 2012 04:11:27PM 4 points [-]

If they were contributions to open-source projects, that would be one thing.

Open-source contribution is even more gameable than patents: at least with patents there's a human involved, checking to some degree that there is at least a little new stuff in the patent, while no one and nothing stops you from putting a worthless repo up on Github reinventing wheels poorly.

But people doing work that generates patents which don't lead to higher income - that raises some questions for me.

The usual arrangement with, say, industrial researchers is that their employers receive the unpredictable dividends from the patents in exchange for forking over regular salaries in fallow periods...

Is it possible that extremely high IQ is associated with a tendency to become "addicted" to a game like patenting?

I don't see why you would privilege this hypothesis.

Comment author: MichaelBishop 28 March 2012 05:00:28PM *  1 point [-]

Let me put it this way. Before considering the Terman data on patents you presented, I already thought IQ would be positively correlated with producing positive externalities and that there was a mostly one way causal link from the former to the latter. I expected the correlation between patents and IQ. What was new to me was the lack of correlation between IQ and income, and the lack of correlation between patents and income. Correction added: there was actually a fairly strong correlation between IQ and income, just not between income and patents, (conditional on IQ I think). Surely more productive industrial researchers are generally paid more. Many firms even give explicit bonuses on a per patent basis. So for me, given my priors, the Terman data you presented shifts me slightly against correction: does not shift me for or against the hypothesis that at the highest IQ levels, higher IQ individuals continues to be associated with producing more positive externalities. ref Still, I think increasing people's IQ, even the already gifted, probably has strong positive externalities unless the method for increasing it also has surprising (to me) side-effects.

I agree that measuring open-source contributions requires more than merely counting lines of code written. But I did want to highlight the fact that the patent system is explicitly designed to increase the private returns for a given innovation. I don't think that there is a strong correlation between the companies/industries which are patenting the most, and the companies/industries, which are benefiting the world the most.

View more: Next