You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

gwern comments on The Logic of the Hypothesis Test: A Steel Man - Less Wrong Discussion

5 Post author: Matt_Simpson 21 February 2013 06:19AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (37)
Tags: testing induction hypothesis statistics frequentist

Comments (37)

You are viewing a single comment's thread. Show more comments above.

Comment author: gwern 21 February 2013 11:32:48PM 0 points [-]

I don't follow... that sounds like you're giving the definition of a one-tailed hypothesis test. What does that have to do with a constant c? Suppose I do this in R:

R> set.seed(12345); t.test(rnorm(20))
One Sample t-test
data: rnorm(20)
t = 0.4103, df = 19, p-value = 0.6861
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
-0.3138 0.4668
sample estimates:
mean of x
0.07652

And get a 95% CI of (-0.3138-0.4668); if my null hypothesis (H0) is my mu or sample mean (0.07652), then you say my Ha is mu > c, or 0.07652 > c. What is this c?

Comment author: Matt_Simpson 22 February 2013 04:06:27AM *  0 points [-]

So rereading your first comment, I realize you said one-sample vs. two-sample hypothesis test and not one-sided vs. two-sided (ore one-tailed vs. two-tailed). If that's what you meant, I don't follow your first comment. The t-test I gave in the post is a one-sample test - and I don't understand how the difference between the two is relevant here.

But to answer your question anyway:

I don't follow... that sounds like you're giving the definition of a one-tailed hypothesis test. What does that have to do with a constant c? Suppose I do this in R:

And get a 95% CI of (-0.3138-0.4668); if my null hypothesis (H0) is my mu or sample mean (0.07652), then you say my Ha is mu > c, or 0.07652 > c. What is this c?

c is the value you're testing as the null hypothesis. In that R-code, R assumes that c=0 so that H0: mu=c and Ha: mu=/=c. For the R code:

t.test(data, alternative="greater", mu=c)

You perform a t test with H0: mu<=c and Ha: mu>c.

Comment author: gwern 22 February 2013 10:01:52PM 0 points [-]

I'm interested in the calculated confidence interval, not the p-value necessarily. Noodling around some more, I think I'm starting to understand it more: the confidence interval isn't calculated with respect to the H0 of 0 which the R code defaults to, it's calculated based purely on the mean (and then an H0 of 0 is assumed to spit out some p-value)

R> set.seed(12345); t.test(rnorm(20,100,15))
One Sample t-test
data: rnorm(20, 100, 15)
t = 36.16, df = 19, p-value < 2.2e-16
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
95.29 107.00
sample estimates:
mean of x
101.1
R>
R> 107-95.29
[1] 11.71
R> 107 - (11.71/2)
[1] 101.1

Hm... I'm trying to fit this assumption into your framework....

  1. Either h0, true mean = sample mean; or ha, true mean != sample mean
  2. construct the test statistic: 't = sample mean - sample mean / s/sqrt(n)'
  3. 't = 0 / s/sqrt(n)'; t = 0
  4. ... a confidence interval
Comment author: Matt_Simpson 26 February 2013 08:36:25PM 0 points [-]

A 95% confidence interval is sort of like testing H0:mu=c vs Ha:mu=\=c for all values of c at the same time. In fact if you reject the null hypothesis for a given c when c is outside your calculated confidence interval and fail to reject otherwise, you're performing the exact same t-test with the exact same rejection criteria as the usual one (that is if the p-value is less than 0.05).

The formula for the test statistic is (generally) t = (estimate - c)/(standard error of estimate) while the formula for a confidence interval is (generally) estimate +/- t^(standard error of estimate) where t^ is a quantile of the t distribution with appropriate degrees of freedom, chosen according to your desired confidence level. t^* and the threshold for rejecting the null in a hypothesis test are intimately related. If you google "confidence intervals and p values" I'm sure you'll find a more polished and detailed explanation of this than mine.

Powered by Reddit Powered by Reddit