gwern comments on The Logic of the Hypothesis Test: A Steel Man - Less Wrong

5 Post author: Matt_Simpson 21 February 2013 06:19AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (37)
Tags: testing induction hypothesis statistics frequentist

Comments (37)

You are viewing a single comment's thread.

Comment author: gwern 21 February 2013 05:22:10PM 0 points [-]

What's interesting to me about this process is that it almost tries to avoid induction altogether. Only the move from step 4 to 5 seems anything like an inductive argument. The rest is purely deductive - though admittedly it takes a couple premises in order to quantify just how likely our sample was and that surely has something to do with induction. But it's still a bit like solving the problem of induction by sweeping it under the rug then putting a big heavy deduction table on top so no one notices the lumps underneath.

One interesting thing here is that you start with a null vs other hypothesis, but that's because you're doing a two-sample z/t-test. But what's going on when you do a one sample z/t-test and get out a confidence interval?

Comment author: Matt_Simpson 21 February 2013 06:54:54PM 0 points [-]

The one sided hypothesis test is still null vs. other because it uses the full parameter space, i.e. it's H0: mu <= c vs. Ha: mu > c. We present it to undergrads as H0: mu = c vs. Ha: mu > c in order to simplify (I think that's the reason anyway) but really we're testing the former. The Karlin-Rubin theorem justifies this.

Comment author: gwern 21 February 2013 11:32:48PM 0 points [-]

I don't follow... that sounds like you're giving the definition of a one-tailed hypothesis test. What does that have to do with a constant c? Suppose I do this in R:

R> set.seed(12345); t.test(rnorm(20))
One Sample t-test
data: rnorm(20)
t = 0.4103, df = 19, p-value = 0.6861
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
-0.3138 0.4668
sample estimates:
mean of x
0.07652

And get a 95% CI of (-0.3138-0.4668); if my null hypothesis (H0) is my mu or sample mean (0.07652), then you say my Ha is mu > c, or 0.07652 > c. What is this c?

Comment author: Matt_Simpson 22 February 2013 04:06:27AM *  0 points [-]

So rereading your first comment, I realize you said one-sample vs. two-sample hypothesis test and not one-sided vs. two-sided (ore one-tailed vs. two-tailed). If that's what you meant, I don't follow your first comment. The t-test I gave in the post is a one-sample test - and I don't understand how the difference between the two is relevant here.

But to answer your question anyway:

I don't follow... that sounds like you're giving the definition of a one-tailed hypothesis test. What does that have to do with a constant c? Suppose I do this in R:

And get a 95% CI of (-0.3138-0.4668); if my null hypothesis (H0) is my mu or sample mean (0.07652), then you say my Ha is mu > c, or 0.07652 > c. What is this c?

c is the value you're testing as the null hypothesis. In that R-code, R assumes that c=0 so that H0: mu=c and Ha: mu=/=c. For the R code:

t.test(data, alternative="greater", mu=c)

You perform a t test with H0: mu<=c and Ha: mu>c.

Comment author: gwern 22 February 2013 10:01:52PM 0 points [-]

I'm interested in the calculated confidence interval, not the p-value necessarily. Noodling around some more, I think I'm starting to understand it more: the confidence interval isn't calculated with respect to the H0 of 0 which the R code defaults to, it's calculated based purely on the mean (and then an H0 of 0 is assumed to spit out some p-value)

R> set.seed(12345); t.test(rnorm(20,100,15))
One Sample t-test
data: rnorm(20, 100, 15)
t = 36.16, df = 19, p-value < 2.2e-16
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
95.29 107.00
sample estimates:
mean of x
101.1
R>
R> 107-95.29
[1] 11.71
R> 107 - (11.71/2)
[1] 101.1

Hm... I'm trying to fit this assumption into your framework....

  1. Either h0, true mean = sample mean; or ha, true mean != sample mean
  2. construct the test statistic: 't = sample mean - sample mean / s/sqrt(n)'
  3. 't = 0 / s/sqrt(n)'; t = 0
  4. ... a confidence interval
Comment author: Matt_Simpson 26 February 2013 08:36:25PM 0 points [-]

A 95% confidence interval is sort of like testing H0:mu=c vs Ha:mu=\=c for all values of c at the same time. In fact if you reject the null hypothesis for a given c when c is outside your calculated confidence interval and fail to reject otherwise, you're performing the exact same t-test with the exact same rejection criteria as the usual one (that is if the p-value is less than 0.05).

The formula for the test statistic is (generally) t = (estimate - c)/(standard error of estimate) while the formula for a confidence interval is (generally) estimate +/- t^(standard error of estimate) where t^ is a quantile of the t distribution with appropriate degrees of freedom, chosen according to your desired confidence level. t^* and the threshold for rejecting the null in a hypothesis test are intimately related. If you google "confidence intervals and p values" I'm sure you'll find a more polished and detailed explanation of this than mine.

Powered by Reddit Powered by Reddit