"Journal" might be a wrong concept, this service looks like something that could be decoupled from publishing.
I imagine it would not contain articles, only information about reliability of statements. Such as "result R published in article A was replicated successfully, here are the relevant data". So when someone reads an article, they could come to this service to check its reliability.
How should such thing be organized?
Wikis, Consumer Reports, Cochrane Reports, etc. The virtue of journals would be to reduce switching costs for academia.
I know there are already journals which require pre-registration; perhaps one could investigate how successful they have been. (We've known about these issues for a long time, but they haven't been fixed - which suggests incentive problems....)
Would one get banned from this journal for registering the testing of 100 only slightly different hypotheses at p=0.01?
Luke's recent post mentioned that The Lancet has a policy encouraging the advance registration of clinical trials, while mine examined an apparent case study of data-peeking and on-the-fly transformation of studies. But how much variation is there across journals on such dimensions? Are there journals that buck the standards of their fields (demanding registration, p=0.01 rather than p=0.05 where the latter is typical in the field, advance specification of statistical analyses and subject numbers, etc)? What are some of the standouts? Are there fields without any such?
I wonder if there is a niche for a new open-access journal, along the lines of PLoS, with standards strict enough to reliably exclude false-positives. Some possible titles: