gwern comments on The Logic of the Hypothesis Test: A Steel Man - Less Wrong

5 Post author: Matt_Simpson 21 February 2013 06:19AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (37)
Tags: testing induction hypothesis statistics frequentist

Comments (37)

You are viewing a single comment's thread. Show more comments above.

Comment author: gwern 22 February 2013 10:01:52PM 0 points [-]

I'm interested in the calculated confidence interval, not the p-value necessarily. Noodling around some more, I think I'm starting to understand it more: the confidence interval isn't calculated with respect to the H0 of 0 which the R code defaults to, it's calculated based purely on the mean (and then an H0 of 0 is assumed to spit out some p-value)

R> set.seed(12345); t.test(rnorm(20,100,15))
One Sample t-test
data: rnorm(20, 100, 15)
t = 36.16, df = 19, p-value < 2.2e-16
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
95.29 107.00
sample estimates:
mean of x
101.1
R>
R> 107-95.29
[1] 11.71
R> 107 - (11.71/2)
[1] 101.1

Hm... I'm trying to fit this assumption into your framework....

  1. Either h0, true mean = sample mean; or ha, true mean != sample mean
  2. construct the test statistic: 't = sample mean - sample mean / s/sqrt(n)'
  3. 't = 0 / s/sqrt(n)'; t = 0
  4. ... a confidence interval
Comment author: Matt_Simpson 26 February 2013 08:36:25PM 0 points [-]

A 95% confidence interval is sort of like testing H0:mu=c vs Ha:mu=\=c for all values of c at the same time. In fact if you reject the null hypothesis for a given c when c is outside your calculated confidence interval and fail to reject otherwise, you're performing the exact same t-test with the exact same rejection criteria as the usual one (that is if the p-value is less than 0.05).

The formula for the test statistic is (generally) t = (estimate - c)/(standard error of estimate) while the formula for a confidence interval is (generally) estimate +/- t^(standard error of estimate) where t^ is a quantile of the t distribution with appropriate degrees of freedom, chosen according to your desired confidence level. t^* and the threshold for rejecting the null in a hypothesis test are intimately related. If you google "confidence intervals and p values" I'm sure you'll find a more polished and detailed explanation of this than mine.

Powered by Reddit Powered by Reddit