You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

gwern comments on The Logic of the Hypothesis Test: A Steel Man - Less Wrong Discussion

5 Post author: Matt_Simpson 21 February 2013 06:19AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (37)
Tags: testing induction hypothesis statistics frequentist

Comments (37)

You are viewing a single comment's thread. Show more comments above.

Comment author: gwern 22 February 2013 10:01:52PM 0 points [-]

I'm interested in the calculated confidence interval, not the p-value necessarily. Noodling around some more, I think I'm starting to understand it more: the confidence interval isn't calculated with respect to the H0 of 0 which the R code defaults to, it's calculated based purely on the mean (and then an H0 of 0 is assumed to spit out some p-value)

R> set.seed(12345); t.test(rnorm(20,100,15))
One Sample t-test
data: rnorm(20, 100, 15)
t = 36.16, df = 19, p-value < 2.2e-16
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
95.29 107.00
sample estimates:
mean of x
101.1
R>
R> 107-95.29
[1] 11.71
R> 107 - (11.71/2)
[1] 101.1

Hm... I'm trying to fit this assumption into your framework....

  1. Either h0, true mean = sample mean; or ha, true mean != sample mean
  2. construct the test statistic: 't = sample mean - sample mean / s/sqrt(n)'
  3. 't = 0 / s/sqrt(n)'; t = 0
  4. ... a confidence interval
Comment author: Matt_Simpson 26 February 2013 08:36:25PM 0 points [-]

A 95% confidence interval is sort of like testing H0:mu=c vs Ha:mu=\=c for all values of c at the same time. In fact if you reject the null hypothesis for a given c when c is outside your calculated confidence interval and fail to reject otherwise, you're performing the exact same t-test with the exact same rejection criteria as the usual one (that is if the p-value is less than 0.05).

The formula for the test statistic is (generally) t = (estimate - c)/(standard error of estimate) while the formula for a confidence interval is (generally) estimate +/- t^(standard error of estimate) where t^ is a quantile of the t distribution with appropriate degrees of freedom, chosen according to your desired confidence level. t^* and the threshold for rejecting the null in a hypothesis test are intimately related. If you google "confidence intervals and p values" I'm sure you'll find a more polished and detailed explanation of this than mine.

Powered by Reddit Powered by Reddit