All of EHeller's Comments + Replies

It wouldn't have made a lot of sense to predict any doublings for transistors in an integrated circuit before 1960, because I think that is when they were invented.

This claim doesn't make much sense from the outset. Look at your specific example of transistors. In 1965, an electronics magazine wanted to figure out what would happen over time with electronics/transistors so they called up an expert, the director of research of Fairchild semiconductor. Gordon Moore (the director of research), proceeded to coin Moore's law and tell them the doubling would continue for at least a decade, probably more. Moore wasn't an outsider, he was an expert.

You then generalize from an incorrect anecdote.

0Houshalter
I never said that every engineer at every point in time was pessimistic. Just that many of them were at one time. And I said it was a second hand anecdote, so take that for what it's worth.
EHeller120

I'm not sure the connotation of the term (i.e. a black person being successful at anything is so shocking it's entertainment value all on it's own) makes the statement any better. Especially when discussing, say, one of the most important American musicians of all time (among others).

I thought the heuristic was "if I think I passed the hotel, I was going too fast to notice. I better slow down so I see it when I come up on it, or so I might recognize a landmark/road that indicates I went too far." We slow down not because we are splitting the difference between turning around and continuing on. We slow down to make it easier to gather more information, a perfectly rational response.

1Pedro Callado
Yes, Thays what I think happens. I guess that stopping could be a "better" way of doing it, but if you're unsure the hotel might be ahead, keep going while trying to gather proper orientation may not be the worse. That would require lower speeds. If the example was about a frozen lake and you were driving over what you thought was thin ice, maybe being very careful with any momentum change should apply. I guess that may be the core underlying concern in sensitive situations.

Sure, not 100% unique to academia, there are also industrial research environments.

My phd was in physics, and there were lots of examples. Weird tricks for aligning optics benches, semi-classical models that gave good order of magnitude estimates despite a lack of rigour, which estimates from the literature were trust worthy (and which estimates were garbage). Biophysics labs and material science lab all sorts of rituals around sample and culture growth and preparation. Many were voodoo, but there were good reasons for a lot of them as well.

Even tr... (read more)

In STEM fields, there is a great deal of necessary knowledge that simply is not in journals or articles, and is carried forward as institutional knowledge passed around among grad students and professors.

Maybe someday someone clever will figure out how to disseminate that knowledge, but it simply isn't there yet.

0Risto_Saarelma
Based on Razib Khan's blog posts, many cutting edge researchers seem to be pretty active on Twitter where they can talk about their own stuff and keep up on what their colleagues are up to. Grad students on social media will probably respond to someone asking about their subfield if it looks like they know their basics and may be up to something interesting. The tiny bandwidth is of course a problem. "Professor Z has probably proven math lemma A" fits in a tweet, instruction on lab work rituals not so much. Clever people who don't want to pay for plane tickets and tuition might be pretty resourceful though, once they figure out they want to talk with each other to learn what they need to know.
0btrettel
Interesting point. Can you give an example of this knowledge? I'm working on a PhD myself (in engineering), but the main things I feel I get from this are access to top scholars, mentoring, structure, and the chance to talk with others who are interested in learning more and research. One could also have access to difficult to obtain equipment in academia, but a large corporation could also provide such equipment. In principle I don't think these things are unique to academia.

No, the important older theories lead to better theories.

Newton's gravitational physics made correct predictions of limited precision, and Newton's laws lead to the development of Navier-Stokes, kinetic theories of gasses,etc. Even phlogiston lead to the discovery of oxygen and the modern understanding of oxidation. You don't have to be 100% right to make useful predictions.

Vitalism, on the other hand, like astrology, didn't lead anywhere useful.

But quantum theory also makes correct predictions, and mainstream physics does not en masse advocate quackery. Vitalism never worked, and it lead the entire medical community to advocate actively harmful quackery for much of the 19th century.

0Gunnar_Zarncke
By this line of reasoning almost all past theories can the discredited. People use a theory to make predictions and act on them. Only later do you learn the shortcomings. If you don't have empiricism you don't even have a tool to systematically notice your error. I think this is a fully general counter argument.
3ChristianKl
As said above Vitalism worked in the sense that it suggest to treat organic and anorganic chemistry differently. Vitalism isn't a single idea. Aristoles get's labeled as a Vitalist but he didn't consider the vital force to be a prime element the way people in the 18th century did. He instead had the theory of the four bodily humors. Homeopathy that is supposed to strengthen the vital force is less harmful than draining blood from people during a cold as the four bodily humor theory predicted. If you say the theory of the existance of a vital force lead to harmful treatments and there treatments that you mean that aren't based on humorism?

No, vitalism wasn't just a dead end, it was a wrong alley that too many people spent time wandering down. Vital theories were responsible for a lot of the quack ideas of medical history.

0Gunnar_Zarncke
Quantum theories are responsible for a lot of the quack ideas too. I fear this isn't enough to make an idea ridiculous.

I don't think that is true? There is a huge contingent of evangelicals (last I checked, a bit under half of Americans believe in creationism), it only takes a few non-creationist but religious Christians to get to a majority.

0Tem42
I think you are missing a critical point -- most people seriously don't care about the age of the Earth, at all. So if you ask someone "did God create the Earth in its present form", you are not identifying whether or not someone is a young Earth creationist, but simply giving the prompt "do you believe in God enough to say 'yes' on a random survey?" One survey found that 25% of Americans don't know that the Earth orbits the sun. This seems like a non-religious question to me, and thus I am willing to take it as a general indicator of 'how much Americans care about basic science'. So I would split that 42% into two groups: 'Americans who strongly believe that God created the Universe in its present form' = 17% (ish), 'Americans who guessed wrong and/or would like to weakly signal that they are Christians' = 25% (ish). Most people just don't care enough to alieve about science. However, I suspect that more people do care enough to alieve about politics, and are willing to base their political ingroup on religion.
-1Jiro
Whether someone is an alieving Christian can be hard to determine because of where you set your threshhold--typically people act as though some things about Christianity are true but not others. But entirelyuseless brought it up in the context of the people who run the government and I think it's exceptionally clear that most of them aren't. I certainly doubt that the members of the Supreme Court who voted for gay marriage are either evangelicals or religious Christians.

There is a lot of statistical literature on optimal experimental design, and it's used all the time. Years ago at Intel, we spent a lot of time on optimal design of quality control measurements, and I have no doubt a lot of industrial scientists in other companies spend their time thinking about such things.

The problem is, information is a model dependent concept (derivatives of log-likelihood depend on the likelihood), so if your prior isn't fairly strong, there isn't a lot of improvement to be had. A lot of science is exploratory, trying to optimize ... (read more)

I don't understand the improvement you think is possible here. In a lot of cases, the math isn't the problem, the theory is known. The difficulty is usually finding a large enough sample size,etc.

You'd think so, but office hours and TA sections without attendance grades are very sparsely attended.

3anna_macdonald
When I was in college, I almost never went to office hours or TA hours... except for one particular class, where the professor was a probably-brilliant guy who was completely incapable of giving a straight explanation or answer to anything. TA hours were packed full; most of the class went, and the TA explained all the stuff the teacher hadn't.
4IlyaShpitser
Not in my class.

How hard your quals are depends on how well you know your field. I went to a top 5 physics program, and everyone passed their qualifying exams, roughly half of whom opted to take the qual their first year of grad school. Obviously, we weren't randomly selected though.

Fellowships are a crapshoot that depend on a lot of factors outside your control, but getting funding is generally pretty easy in the sciences. When you work as an "RA" you are basically just doing your thesis research. TAing can be time consuming, but literally no one cares if... (read more)

8gjm
I have heard rumours that students are actually people, and that they care about the quality of the teaching they receive.
2Gram_Stone
Do you incur debt if this happens, due to the cost of stipends and tuition waivers to the institution?

I don't think medicine is a junk investment when you consider the opportunity cost, at least in the US.

Consider my sister, a fairly median medical school graduate in the US. After 4 years of medical school (plus her undergrad) she graduated with 150k in debt (at 6% or so). She then did a residency for 3 years making 50k a year, give or take. After that she became an attending with a starting salary of $220k. At younger than 30, she was in the top 4% of salaries in the US.

The opportunity cost is maybe ~45k*4 years, 180k + direct cost of 150k or so.... (read more)

I don't see how Eliezer is correct here. Conservation of energy just isn't deeply related to the deeper structure of quantum mechanics in the way Harry suggests. It's not related to unitarity, so you can't do weird non-unitary things.

Hold on- aren't you saying the choice of experimental rule is VERY important (i.e. double blind vs. not double blind,etc)?

If so you are agreeing with VAuroch. You have to include the details of the experiment somewhere. The data does not speak for itself.

2Cyan
Of course experimental design is very important in general. But VAuroch and I agree that when two designs give rise to the same likelihood function, the information that comes in from the data are equivalent. We disagree about the weight to give to the information that comes in from what the choice of experimental design tells us about the experimenter's prior state of knowledge.

My point was only that nothing in the axioms prevents macroscopic superposition.

EHeller30

The part that is new compared to Cromwell's rule is that Yudkowsky doesn't want to give probability 1 to logical statements (53 is a prime number).

Because he doesn't want to treat 1 as a probability, you can't expect complete sets of events to have total probability 1, despite them being tautologies. Because he doesn't want probability 0, how do you handle the empty set? How do you assign probabilities to statements like "A and B" where A and B are logical exclusive? (the coin lands heads AND the coin lands tails).

Removing 0 and 1 from the math of probability breaks most of the standard manipulations. Again, it's best to just say "be careful with 0 and 1 when working with odds ratios."

EHeller40

I think the issue at hand is that 0 and 1 aren't special cases at all, but very important for the math of probability theory to work (try and construct a probability measure where some subset doesn't have probability 1 or 0).

This is incredibly necessary for the mathematical idea of probability ,and EY seems to be confusing "are 0 and 1 probabilities relevant to Bayesian agents?" with "are 0 and 1 probabilities?" (yes, they are, unavoidably, not as a special case!).

EHeller10

So there are obviously a lot of different things you could mean by "Copenhagen" or "in the back of a lot of copenhagenist minds" but the way it's usually used by physicists nowadays is to mean "the Von Neumann axioms" because that is what is in 90+% of the textbooks.

0TheAncientGeek
The von Neumann axioms aren't self interpreting . Physicists are trained to understand things in terms of mathematical formalisms and experimental results, but that falls over when dealing with interpretation. Interpretations canot be settled empirically, by definition,, and formulae are not self interpreting.
EHeller00

There is nothing in Copenhagen that forbids macroscopic superposition. The experimental results of macroscopic superposition in SQUIDs are usually calculated in terms of copenhagen (as are almost all experimental results).

1TheAncientGeek
That's mainly because Copenhagen never specified macrsoscopic ...but the idea of an unequivocal "cut" was at the back of a lot of copenhagenists minds, and it has been eaten away by various things over the years.
EHeller00

How are you defining territory here? If the territory is 'reality' the only place where quantum mechanics connects to reality is when it tells us the outcome of measurements. We don't observe the wavefunction directly, we measure observables.

I think the challenge of MWI is to make the probabilities a natural result of the theory, and there has been a fair amount of active research trying and failing to do this. RQM side steps this by saying "the observables are the thing, the wavefunction is just a map, not territory."

0TheMajor
See my reply to TheAncientGeek, I think it covers most of my thoughts on this matter. I don't think that your second paragraph captures the difference between RQM and MWI - the probabilities seem to be just as arbitrary in RQM as they are in any other interpretation. RQM gets some points by saying "Of course it's partially arbitrary, they're just maps people made that overfit to reality!", but it then fails to explain exactly which parts are overfitting, or where/if we would expect this process to go wrong.
0nyralech
To my very limited understanding, most of QM in general is completely unnatural as a theory from a purely mathematical point of view. If that is actually so, what precisely do you mean by "natural result of the theory"?
EHeller00

How would this affect a frequentist?

It doesn't the frequentist is already measuring with the sample distribution. That is how frequentism works.

I was mainly trying to convince you that nothing's actually wrong with having 33% false positive rate in contrived cases.

I mean it's not "wrong" but if you care about false positive rates and there is a method had has a 5% false positive rate, wouldn't you want to use that instead?

0ike
If for some reason low false positive rates were important, sure. If it's important enough to give up consistency.
EHeller00

No, there's a limit on that as well. See http://www.ejwagenmakers.com/2007/StoppingRuleAppendix.pdf

I can check my simulation for bugs. I don't have the referenced textbook to check the result being suggested.

It is my thesis that every optional stopping so-called paradox can be converted into a form without optional stopping, and those will be clearer as to whether the problem is real or not.

The first part of this is trivially true. Replace the original distribution with the sampling distribution from the stopped problem, and it's not longer a st... (read more)

0ike
You can see http://projecteuclid.org/euclid.aoms/1177704038, which proves the result. How would this affect a frequentist? I'm giving low data because those are the simplest kinds of cases to think of. If you had lots of data with the same distribution/likelihood, it would be the same. I leave it as an exercise to find a case with lots of data and the same underlying distribution ... I was mainly trying to convince you that nothing's actually wrong with having 33% false positive rate in contrived cases.
EHeller00

I think this is problem dependent.

In simulation, I start to asymptote to around 20%, with a coin flip, but estimating mean from a normal distribution (with the null being 0) with fixed variance I keep climbing indefinitely. If you are willing to sample literally forever it seems like you'd be able to convince the Bayesian that the mean is not 0 with arbitrary Bayes factor. So for large enough N in a sample, I expect you can get a factor of 3 for 99/100 of the Bayesians in cages (so long as that last Bayesian is really, really sure the value is 0).

But it doesn't change the results if we switch and say we fool 33% of the Bayesians with Bayes factor of 3. We are still fooling them.

4ike
No, there's a limit on that as well. See http://www.ejwagenmakers.com/2007/StoppingRuleAppendix.pdf If you can generate arbitrarily high Bayes factors, then you can reduce your posterior to .01, which means that it can only happen 1 in 100 times. You can never have a guarantee of always getting strong evidence for a false hypothesis. If you find a case that does, it will be new to me and probably change my mind. That doesn't concern me. I'm not going to argue for why, I'll just point out that if it is a problem, it has absolutely nothing to do with optional stopping. The exact same behavior (probability 1/3 of generating a Bayes factor of 3 in favor of a false hypothesis) shows up in the following case: a coin either always lands on heads, or lands on heads 1/3 of the time and tails 2/3 of the time. I flip the coin a single time. Let's say the coin is the second coin. There's a 33% chance of getting heads, which would produce a Bayes factor of 3 in favor of the 100%H coin. If there's something wrong with that, it's a problem with classic Bayes, not optional stopping. It is my thesis that every optional stopping so-called paradox can be converted into a form without optional stopping, and those will be clearer as to whether the problem is real or not.
EHeller00

Before I analyse this case, can you clarify whether the hypothesis happens to be true, false, or chosen at random? Also give these Bayesians' priors, and perhaps an example of the rule you'd use.

Again, the prior doesn't matter, they are computing Bayes factors. We are talking about Bayes factors. Bayes factors. Prior doesn't matter. Bayes factors. Prior.Doesn't.Matter. Bayes factors. Prior.Doesn't.Matter. Bayes.factor.

Let's say the null is true, but the frequentist mastermind has devised some data generating process that (let's say he has infinite data at his disposal) that can produce evidence in favor of competing hypothesis at a Bayes factor of 3, 99% of the time.

2ike
It matters here, because you said "So you might be able to create a rule that fools 99 out of the 100 Bayesians". The probability of getting data given a certain rule depends on which hypothesis is true, and if we're assuming the hypothesis is like the prior, then we need to know the prior to calculate those numbers. That's impossible. http://doingbayesiandataanalysis.blogspot.com/2013/11/optional-stopping-in-data-collection-p.html goes through the math. In fact, you can show easily that this can succeed at most 33% of the time. By definition, the Bayes factor is how likely the data is given one hypothesis, divided by how likely the data is given the other. The data in the class "results in a bayes factor of 3 against the null" has a certain chance of happening given that the null is true, say p. This class of course contains many individual mutually exclusive sets of data, each with a far lower probability, but they sum to p. Now, the chance of this class of possible data sets happening given that the null is not true has an upper bound of 1. Each individual probability (which collectively sum to at most 1) must be 3 times as much as the corresponding probability in the group that sums to p. Ergo, p is upper bounded by 33%.
EHeller00

I'm saying that all inferences are still correct. So if your prior is correct/well calibrated, then your posterior is as well. If you end up with 100 studies that all found an effect for different things at a posterior of 95%, 5% of them should be wrong.

But that is based on the posterior.

When I ask for clarification, you seem to be doing two things:

  1. changing the subject to posteriors
  2. asserting that a perfect prior leads to a perfect posterior.

I think 2 is uncontroversial, other than if you have a perfect prior why do any experiment at all? But i... (read more)

0Lumifer
Can't frequentists just pool their data and then generate a new confidence interval from the supersized sample?
0ike
By perfect I mean well calibrated. I don't see why knowing that your priors in general are well calibrated implies that more information doesn't have positive expected utility. Only in some cases, and only with regard to someone who knows more than the Bayesian. The Bayesian himself can't predict that the factor will be biased; the expected factor should be 1. It's only someone who knows better that can predict this. Before I analyse this case, can you clarify whether the hypothesis happens to be true, false, or chosen at random? Also give these Bayesians' priors, and perhaps an example of the rule you'd use.
EHeller00

That paper only calculates what happens to the bayes factor when the null is true. There's nothing that implies the inference will be wrong.

That is the practical problem for statistics (the null is true, but the experimenter desperately wants it to be false). Everyone wants their experiment to be a success. The goal of this particular form of p-hacking is to increase the chance that you get a publishable result. The goal of the p-hacker is to increase the probability of type 1 error. A publication rule based on Bayes factors instead of p-values is still susceptible to optional stopping.

You seem to be saying that a rule based on posteriors would not be susceptible to such hacking?

2ike
I'm saying that all inferences are still correct. So if your prior is correct/well calibrated, then your posterior is as well. If you end up with 100 studies that all found an effect for different things at a posterior of 95%, 5% of them should be wrong. So what I should say is that the Bayesian doesn't care about the frequency of type 1 errors. If you're going to criticise that, you can do so without regard to stopping rules. I gave an example in a different reply of hacking bayes factors, now I'll give one with hacking posteriors: Two kinds of coins: one fair, one 10%H/90%T. There are 1 billion of the fair ones, and 1 of the other kind. You take a coin, flip it 10 times, then say which coin you think it is. The Bayesian gets the biased coin, and no matter what he flips, will conclude that the coin is fair with overwhelming probability. The frequentist gets the coin, get ~9 tails, and says "no way is this fair". There, the frequentist does better because the Bayesian's prior is bad (I said there are a billion fair ones and only one biased one, but only looked at the biased ones). It doesn't matter if you always conclude with 95% posterior that the null is false when it is true, as long as you have 20 times as many cases that the null is actually false. Yes, this opens you up to being tricked; but if you're worried about deliberate deception, you should include a prior over that. If you're worried about publication bias when reading other studies, include a prior over that, etc.
EHeller00

It depends only on the prior. I consider all these "stopping rule paradoxes" disguised cases where you give the Bayesian a bad prior, and the frequentist formula encodes a better prior.

Then you are doing a very confusing thing that isn't likely to give much insight. Frequentist inference and Bayesian inference are different and it's useful to at least understand both ideas(even if you reject frequentism).

Frequentists are bounding their error with various forms of the law of large numbers, they aren't coherently integrating evidence. So sa... (read more)

0ike
Let me put it differently. Yes, your chance of getting a bayes factor of >3 is 1.8 with data peeking, as opposed to 1% without; but your chance of getting a higher factor also goes down, because you stop as soon as you reach 3. Your expected bayes factor is necessarily 1 weighted over your prior; you expect to find evidence for neither side. Changing the exact distribution of your results won't change that.
0ike
I think I understand frequentism. My claim here was that the specific claim of "the stopping rule paradox proves that frequentism does better than Bayes" is wrong, or is no stronger than the standard objection that Bayes relies on having good priors. What I meant is that you can get the same results as the frequentist in the stopping rule case if you adopt a particular prior. I might not be able to show that rigorously, though. That paper only calculates what happens to the bayes factor when the null is true. There's nothing that implies the inference will be wrong. There are a couple different version of the stopping rule cases. Some are disguised priors, and some don't affect calibration/inference or any Bayesian metrics.
EHeller20

In practice what p-hacking is about is convincing the world of an effect, so you are trying to create bias toward any data looking like a novel effect. Stopping rules/data peeking accomplish this just as much for Bayes as for frequentist inference (though if the frequentist knows about the stopping rule they can adjust in a way that bayesians can't), which is my whole point.

Whether or not the Bayesian calibration is overall correct depends not just on the Bayes factor but the prior.

3ike
It depends only on the prior. I consider all these "stopping rule paradoxes" disguised cases where you give the Bayesian a bad prior, and the frequentist formula encodes a better prior. You still wouldn't have more chances of showing a novel effect than you thought you would when you went into the experiment, if your priors are correct. If you say "I'll stop when I have a novel effect", do this many times, and then look at all the times you found a novel effect, 95% of the time the effect should actually be true. If this is wrong, you must have bad priors.
EHeller20

Reminds of this bit from a Wasserman paper http://ba.stat.cmu.edu/journal/2006/vol01/issue03/wasserman.pdf

van Nostrand: Of course. I remember each problem quite clearly. And I recall that on each occasion I was quite thorough. I interrogated you in detail, determined your model and prior and produced a coherent 95 percent interval for the quantity of interest.

Pennypacker: Yes indeed. We did this many times and I paid you quite handsomely.

van Nostrand: Well earned money I’d say. And it helped win you that Nobel.

Pennypacker: Well they retracted the Nobel a

... (read more)
EHeller00

What makes Bayesian "lose" in the cases proposed by Mayo and Simonsohn isn't the inference, it's the scoring rule. A Bayesian scores himself on total calibration, "number of times my 95% confidence interval includes the truth" is just a small part of it. You can generate an experiment that has a high chance (let's say 99%) of making a Bayesian have a 20:1 likelihood ratio in favor of some hypothesis. By conservation of expected evidence, the same experiment might have 1% chance of generating close to a 2000:1 likelihood ratio against t

... (read more)
2ike
The stopping rule won't change the expectation of the Bayes factor. If your prior is correct, then your 95% credibility interval will, in fact, be well calibrated and be correct 95% of the time. I argued at length on tumblr that most or all of the force of the stopping rule objection to Bayes is a disguised "you have a bad prior" situation. If you're willing to ask the question that way, you can generate similar cases without stopping rules as well. For instance, imagine there are two kinds of coins; ones that land on heads 100% of the time, and ones that land on heads 20% of the time. (The rest are tails.) You have one flip with the coin. Oh, one more thing: I tell you that there are 1 billion coins of the first kind, and only one of the second kind. You flip the coin once. It's easy to show that there's an overwhelming probability of getting a 20:1 likelihood in favor of the first coin. Why is this problematic? I can and have given a similar case for 95% credibility intervals as opposed to Bayes factors, which I'll write out if you're interested.
EHeller00

If you look at the paper, what you call optional stopping is what the authors called "data peeking."

In their simulations, the authors first took in a sample of 20 and calculated it, and then could selectively continue to add data up to 30 (stopping when they reach "effect" or 30 samples). The papers point is that this does skew the Bayes factor (doubles the chances of managing to get a Bayes factor > 3).

0ike
It skews the Bayes factor when the hypothesis is in fact not true. The times that the hypothesis is true should balance out to make the calibration correct overall.
EHeller70

It is true that optional stopping won't change Bayes rule updates (which is easy enough to show). It's also true that optional stopping does affect frequentist tests (different sampling distributions). The broader question is "which behavior is better?"

p-hacking is when statisticians use optional stopping to make their results look more significant (by not reporting their stopping rule). As it turns out you in fact can "posterior hack" Bayesians - http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2374040

Edit: Also Debrah Mayo's Er... (read more)

0Anders_H
I comment on this discussion here: http://lesswrong.com/r/discussion/lw/mke/on_stopping_rules/
2Richard_Kennaway
That is not my understanding of the term "optional stopping" (nor, more significantly, is it that of Jaynes). Optional stopping is the process of collecting data, computing your preferred measure of resultiness as you go, and stopping the moment it passes your criterion for reporting it, whether that is p<0.05, or a Bayes factor above 3, or anything else. (If it never passes the criterion, you just never report it.) That is but one of the large arsenal of tools available to the p-hacker: computing multiple statistics from the data in the hope of finding one that passes the criterion, thinking up more hypotheses to test, selective inclusion or omission of "outliers", fitting a range of different models, and so on. And of these, optional stopping is surely the least effective, for as Jaynes remarks in "Probability Theory as Logic", it is practically impossible to sample long enough to produce substantial support for a hypothesis deviating substantially from the truth. All of those other methods of p-hacking involve concealing the real hypothesis, which is the collection of all the hypotheses that were measured against the data. It is like dealing a bridge hand and showing that it supports astoundingly well the hypothesis that that bridge hand would be dealt. In machine learning terms, the hypothesis is being covertly trained on the data, then tested on how well it fits the data. No measure of the latter, whether frequentist or Bayesian, is a measure of how well the hypothesis will fit new data.
0Jacob Falkovich
Since I don't want this to spiral into another stopping rule argument, allow me to try and dissolve a confusing point that the discussions get stuck on. What makes Bayesian "lose" in the cases proposed by Mayo and Simonsohn isn't the inference, it's the scoring rule. A Bayesian scores himself on total calibration, "number of times my 95% confidence interval includes the truth" is just a small part of it. You can generate an experiment that has a high chance (let's say 99%) of making a Bayesian have a 20:1 likelihood ratio in favor of some hypothesis. By conservation of expected evidence, the same experiment might have 1% chance of generating close to a 2000:1 likelihood ratio against that same hypothesis. A frequentist could never be as sure of anything, this occasional 2000:1 confidence is the Bayesian's reward. If you rig the rules to view something about 95% confidence intervals as the only measure of success, then the frequentist's decision rule about accepting hypotheses at a 5% p-value wins, it's not his inference that magically becomes superior. Allow me to steal an analogy from my friend Simon: I'm running a Bayesian Casino in Vegas. Debrah Mayo comes to my casino every day with $31. She bets $1 on a coin flip, then bets $2 if she loses, then $4 and so on until she either wins $1 or loses all $31 if 5 flips go against her. I obviously think that by conservation of expected money in a coin flip this deal is fair, but Prof. Mayo tells me that I'm a sucker because I lose more days that I win. I tell her that I care about dollars, not days, but she replies that if she had more money in her pocket, she could make sure I have a losing day with arbitrarily high probability! I smile and ask her if she wants a drink.
EHeller50

The existence of the Higg's is one of the rare bits of physics that doesn't average out under renormalization.

The reason is that the Higgs is deeply related to the overall symmetry of the whole standard model- you start with a symmetry group SU(2)xU(1) and then the Higgs messes with the symmetry so you end up with just U(1) symmetry. What the theory predicts is relationships between the Higgs, the W and Z boson, but not the absolute scale. The general rule is RG flow respects symmetries, but other stuff gets washed out.

This is why the prediction was... (read more)

EHeller30

I think I'm communicating a little poorly. So start with atomic level physics- it's characterized by energy scales of 13.6 eV or so. Making measurements at that scale will tell you a lot about atomic level physics, but it won't tell you anything about lower level physics- there is an infinite number of of lower level physics theories that will be compatible with your atomic theory (which is why you don't need the mass of the top quark to calculate the hydrogen energy levels- conversely you can't find the mass of the top quark by measuring those levels).... (read more)

1hyporational
I think I'm the one communicating poorly since it seems I understood your first explanation, thanks for making it sure anyways and thanks for the link! When I was wondering about successful predictions in particle physics, I was in particular thinking about Higgs boson. We needed to build a massive "microscope" to detect it, yet could predict its existence four decades ago with much lower energy scale equipment, right?
EHeller50

The point of RG is that "higher level" physics is independent of most "lower level" physics. There are infinitely many low level theories that could lead to a plane flying.

There are infinitely many lower level theories that could lead to quarks behaving as they do,etc. So 1. you can't deduce low level physics from high level physics (i.e. you could never figure out quarks by making careful measurements of tennis balls), and you can never know if you have truly found the lowest level theory (there might be a totally different theory ... (read more)

0hyporational
Thanks, my reality got just a bit weirder. It's almost as if someone set up a convenient playground for us, but that must be my apophenia speaking. If there are infinite possibilities of lower level theories, are successful predictions in particle physics just a matter of parsimony? Is there profuse survival bias when it comes to hyping successful predictions?
EHeller10

The whole point of the renormalization group is that lower level models aren't more accurate, the lower level effects average out.

The multiple levels of reality are "parallel in a peculiar way" governed by RG. It might be "more complex" but it's also the backbone of modern physics.

0hyporational
I tried to read about RG but it went way over my head. Is the universe in principle inexplicable by lower level theories alone according to modern physics? Doesn't "averaging out" lose information? Are different levels of abstraction considered equally real by RG? Does this question even matter or is it in the realm of unobservables in the vein of Copenhagen vs MW interpretation?
EHeller10

Heck climate scientists aren't even that sparing about basic facts. They'll mention that CO2 is a greenhouse gas, but avoid any more technical questions. For example, I only recently found out that (in the absence of other factors or any feedback) temperature is a logarithmic function of CO2 concentration.

So this seems like you've never cracked open any climate/atmospheric science textbook? Because that is pretty basic info. It seems like you're determined to be skeptical despite not really spending much time learning about the state of the science. ... (read more)

EHeller00

It's a quite bit more general than Lagrangian mechanics. You can extend it to any functional that takes functions between two manifolds to complex numbers.

EHeller00

General question- does combining the 2013 and 2014 survey make sense, given that we expect a lot of overlap (same people surveyed)?

Also, why treat EA as a latent variable when it was directly surveyed? Shouldn't we just combine by saying if you answered Yes to an EA questions, you are EA?

4gwern
Yes, because the surveys have (since the 2012 survey, IIRC) asked if you took a previous survey. So I dropped anyone in the 2014 data who answered yes (s2014 <- subset(survey2014, PreviousSurveys!="Yes", select=...); as long as everyone was honest, there should be zero overlap/double-counting. I also included 2013 vs 2014 response as a covariate to allow estimating some of the heterogeneity. If you answer both yes and also claim to go to EA events, aren't you probably more of an EA than someone who says yes but doesn't attend? In any event, it doesn't make a difference. I just like using latent variables when I can.
EHeller20

Well, one thing was definitely changed was my approach to the coursework. I started taking a lot of notes as a memory aid, but then when I worked through problems I relied on what I remembered and refused to look things up in the text book or my notes. This forced me to figure out ways to solve problems in ways that made sense to me- it was really slow going at first but I slowly built up my own bag of tricks.

0John_Maxwell
Interesting. I think I've read research suggesting that answering questions is significantly better for learning than just reading material (similar to how Anki asks you questions instead of just telling you things). Val at CFAR likes to make the point that if you look at what students in a typical math class are actually practicing during class, they are practicing copying off the blackboard. In the same way maybe what most people are "actually practicing" when they do math homework is flipping though the textbook until they find an example problem that looks analogous to the one they're working on and imitating the structure of the example problem solution in order to do their homework.
EHeller00

Glancing at the data, it looks like the median EA at several ages gives 0 as well as the median non-EA. You might want to separate the 0 set from everything else and then answer two questions:

what percentage of EAs/non-EAs donate any money when they give, how much do EAs give, how much do non-EAs give.

I think this makes more sense then what is happening now- the lines don't seem to fit the data very well.

EHeller00

Can you give an example of the level where things suddenly become more difficult?

As I said in another post, I struggled quite a bit with early calculus classes, but breezed through later "more difficult" classes that built on them.

Also, I disagree with the math and stats thing. Many of the best statisticians I know have strong grounding in mathematics, as do many of the best data scientists I know.

2Shmi
I hit a wall in my string theory course, after having to apply a lot more effort than expected in a QFT course the year before. Didn't have that issue with GR at all. Well, maybe with some finer points involving algebraic topology, but nothing insurmountable.
EHeller50

Of course he would have gotten an A. The difference between being good and bad at math is whether you need to "spent all waking hours talking about calculus" to get an A.

Extrapolating from 1 course is silly. I worked like a demon to do mediocre (low Bs) in both calc 1 and physics 1, but somewhere towards the end of my freshman year of college something fell into place for me. By my first year of grad school I was breezing through quantum field theory and math methods for string theory courses with minimal effort.

6John_Maxwell
Fascinating... do you have any idea what might have "fallen in to place"? (I'm always eager to learn from people who became good, as opposed to people who were always good or always bad, because I figure the people who became good have the most to tell us about whatever components of being good are non-innate. For example, Elon Musk thinks he's been highly driven ever since he was a kid, which suggests that he doesn't have much to teach others about motivation.)
5Shmi
That's true. I've seen this go both ways, too. Though the priors are against a turn of events like yours, it does happen.
EHeller-20

I worked it out back of the envelope, and the probability of being kidnapped when you blink is only 1/5^^^5.

1dxu
Well, now I know you're underestimating how big 3^^^3 is (and 5^^^5, too). But let's say somehow you're right, and the probability really is 1/5^^^5. All I have to do is modify the thought experiment so that the planet has 5^^^5 people instead of 3^^^3. There, problem solved. So, new question: would you prefer that one person be horribly tortured for fifty years without hope or rest, or that 5^^^5 people get dust specks in their eyes?
EHeller00

Both numbers seem basically arbitrarily small (probability 0).

Since the planet has so many distinct people, and they blink more than once a day, you are essentially asserting that on that planet, multiple people are kidnapped and tortured for more than 50 years several times a day.

1dxu
Well, I mean, obviously a single person can't be kidnapped more than once every 50 years (assuming that's how long each torture session lasts), and certainly not several times a day, since he/she wouldn't have finished being tortured quickly enough to be kidnapped again. But yes, the general sentiment of your comment is correct, I'd say. The prospect of a planet with daily kidnappings and 50-year-long torture sessions may seem strange, but that sort of thing is just what you get when you have a population count of 3^^^3.
EHeller00

However, 3^^^3 just so unimaginably enormous that blinking for even the tiniest fraction of a second increases the probability that you will be captured by a madman during that blink and tortured for 50 years by more than 1/3^^^3.

This seems pretty unlikely to be true.

0dxu
I think you underestimate the magnitude of 3^^^3 (and thereby overestimate the magnitude of 1/3^^^3).
EHeller20

I think the bigger issue with metamed is that they are trying to fill a role that already exists in the market. UpToDate is a bigger, more ambitious company (instead of generating per-patient reports they have a huge database of research summaries curated by actual experts for each disease) that almost all academic medical centers already pay for.

Load More