TL,DR: More information is never a bad thing.
As certain wise Paperclip Optimizer once said, information that someone is blackmailing you is bad. You're better off not having this information because it makes you blackmail-proof.
All your analysis gets thrown out of the window in case of signaling, game theory etc. There are probably a lot more other cases where it doesn't work.
As certain wise Paperclip Optimizer once said, information that someone is blackmailing you is bad.
Actually, no it isn't. What is bad for you is for the blackmailer to learn that you are aware of the blackmail.
Acquiring information is never bad, in and of itself. Allowing others to gain information can be bad for you. Speaking as an egoist, that is.
ETA: I now notice that gjm already made this point.
No, that isn't what taw is saying. The point is that having more information and being known to have it can be extremely bad for you. This is not a counterexample to the theorem, which considers two scenarios whose only difference is in how much you know, but in real-life applications that's very frequently not the case.
I don't think taw's blackmail example is quite right as it stands, but here's a slight variant that is. A Simple Blackmailer will publish the pictures if you don't give him the money. Obviously if there is such a person, and if there are no further future consequences, and if you prefer losing the money to losing your reputation, it is better for you to know about the blackmailer so you can give him the money. But now consider a Clever Blackmailer, who will publish the pictures if you don't give him the money and if he thinks you might give him the money if he doesn't. If there's a Clever Blackmailer and you don't know it (and he knows you don't know it) then he won't bother publishing because the threat has no force for you -- since you don't even know there is one. But if you learn of his existence and he knows this then he will publish the pictures unless you give him the money, so you have to give him the money. So, in this situation, you lose by discovering his existence. But only because he knows that you've discovered it.
I am unsure of what the point of posting this theorem was. Yes, it holds as stated, but it seems to have very little applicability to the real world. Your tl;dr version is "More information is never a bad thing", but that is clearly false if we're talking about real people making real decisions.
Because of all the simplifying assumptions, the theorem proved in the post has no bearing on the question posed in the title.
Here's the intuitive version:
Consider the set of all strategies, that is, functions from {possible sequences of observations} => {possible actions}
Each strategy has an expected utility.
Adding more information gets you more strategies, because all the old ones are still viable - you just ignore the new observation - and some additional strategies are viable.
Adding more options is never bad. (because the maximum of AuB is at least as big as the maximum of A)
In the example you choose it is blatantly intuitively obvious that making the observation has high expected utility, so its use as an intuition pump is minimal. Perhaps it would be better to find an example where it's not as immediately obvious?
Counter-example: http://web.archive.org/web/20090415130842/http://www.weidai.com/smart-losers.txt
Seems to me the proof does not go through because it only consider actions taken by the agent.
Or in other words, the expectation of a max of some random variables is always greater or equal to the max of the expectations.
You could call this 'standard knowledge' but it's not the kind of thing one bothers to commit to memory. Rather, one immediately perceives it as true.
TL,DR: More information is never a bad thing.
The average American who has never been to a hockey game could probably do better at naming someone who co-holds the record for the most combined points by brothers in the National Hockey League than the average person who is a casual fan and has been to one or two games.
You are assuming that the observation has no error margin.
Lets suppose that the priors are 51%A and 49%B and then your new observation says "55%A and 45%B" So - automatically you'd round your A-value up a little right?
but very few observations are going to be 100% accurate. Lets say this one has an error rate of 10% so actually it could be only 50%A and 50%B, but has given you a false positive of 55%A
Are you better off? or have you just introduced more error into your estimations?
Here's a simple theorem in utility theory that I haven't seen anywhere. Maybe it's standard knowledge, or maybe not.
TL,DR: More information is never a bad thing.
The theorem proved below says that before you make an observation, you cannot expect it to decrease your utility, but you can sometimes expect it to increase your utility. I'm ignoring the cost of obtaining the additional data, and any losses consequential on the time it takes. These are real considerations in any practical situation, but they are not the subject of this note.
First, an example to illustrate the principle. Suppose you are faced with two choices, A and B. One of them is right and one is wrong, and it's very important to make the right choice, because being right will confer some large positive utility U (you get to marry the princess), while the wrong choice will get you -U (eaten by a tiger). However, you're not sure which is the right choice. You estimate that there's a 51% chance that A is right, and 49% that B is right. So, you shut up and multiply, and choose A for an expected utility of 0.02U, right?
Suppose the choice does not have to be made immediately, and that you can do something to get better information about whether A or B is the right choice. Say you can make certain observations which will tell you with 99% certainty which is right. Your prior expectation of your posterior is equal to your prior, so before you make the observation, you expect a 50/98 chance of it telling you that A is right, and 48/98 that B is right.
You make the observation and then choose the course of action it tells you. Whether it says A or B, it's 99% likely to be right, so your expected utility from choosing according to the observation is 0.98U, an increase over not making the observation of 0.96U.
Clearly, you should make the observation. Even though you cannot expect what it will tell you, you can expect to greatly benefit from whatever it tells you.
Now the general case.
Theorem: Every act of observation has, before you make it, a non-negative expected utility.
Proof. Let the set of actions available to an agent be C. For each action c in C, the agent has a probability distribution over possible outcomes. Each outcome has a certain utility. For present purposes it is not necessary to distinguish between outcomes and their utility, so we shall consider the agent to have, for each action c, a probability distribution P_c(u) over utilities u. The expectation value int_u u P_c(u) of that distribution is the prior expected utility of the choice c, and the agent's rational choice, given no other information, is to choose that c which maximises int_u u P_c(u). The resulting utility is max_c int_u u P_c(u).
(I can't be bothered to fiddle with the system for getting mathematics typeset as images. The underscore indicates subscripts, int_x means integral with respect to x, and max_x means the maximum value over all x. Take care to backslash all the underscores if quoting any of this.)
Now suppose the agent makes an observation, with result o. This gives the agent a new probability distribution for each choice c over outcomes: P_c(u|o). It should choose the c that maximises int_u u P_c(u|o).
The agent also has a prior distribution of observations P(o). Before making the observation, the expected distribution of utility returned by doing c after the observation is int_o P(o) P_c(u|o). This is equal to P_c(u), as it should be, by the principle that your prior estimate of your posterior distribution of a variable must coincide with your prior distribution.
We therefore have the following expected utilities. If we choose the action without making the observation, the utility is
max_c int_u u P_c(u)
= max_c int_u u int_o P(o) P_c(u|o)
If we observe, then choose, we get
int_o P(o) max_c int_u u P_c(u|o)
The second of these is always at least as large as the first. Proof:
max_c int_u u int_o P(o) P_c(u|o)
= max_c int_o P(o) int_u u P_c(u|o)
<= max_c int_o P(o) max_c int_u u P_c(u|o)
= int_o P(o) max_c int_u u P_c(u|o)
ETA: In some cases, a non-zero amount of new information will make zero change to your expected utility. In the original example, suppose that your prior probabilities were 75% for A being right, and 25% for B. You make an additional and rather weak observation which, if it says "choose A" raises your posterior probability for A to 80%, while if it says "choose B", it only diminishes your posterior for A to 60%. In either case you still choose A and your expected utility (prior to actually making the observation) is unchanged.
Or informally, further research is only useful if there is a possibility of it telling you enough to change your mind.