I can't help with the object level determination, but I think you may be overrating both the balance and import of the second-order evidence.
As far as I can tell, Yudkowsky is a (?dramatically) pessimistic outlier among the class of "rationalist/rationalist-adjacent" SMEs in AI safety, and probably even more so relative to aggregate opinion without an LW-y filter applied (cf.). My impression of the epistemic track-record is Yudkowsky has a tendency of staking out positions (both within and without AI) with striking levels of confidence but not commensurate...
> the class of "rationalist/rationalist-adjacent" SMEs in AI safety,
What's an SME?
Looking back, my sense remains that we basically succeeded—i.e., that we described the situation about as accurately and neutrally as we could have. If I'm wrong about this... well, all I can say is that it wasn't for lack of trying.
I think CFAR ultimately succeeded in providing a candid and good faith account of what went wrong, but the time it took to get there (i.e. 6 months between this and the initial update/apology) invites adverse inferences like those in the grandparent.
A lot of the information ultimately disclosed in March would definitely h...
The malaria story has fair face validity if one observes the wider time series (e.g.). Further, the typical EA 'picks' for net distribution are generally seen as filling around the edges of the mega-distributors.
FWIW: I think this discussion would be clearer if framed in last-dollar terms.
If Gates et al. are doing something like last dollar optimisation, trying to save as many lives as they can allocating across opportunities both now and in the future, leaving the right now best marginal interventions on the table would imply they expect to ex...
I also buy the econ story here (and, per Ruby, I'm somewhat pleasantly surprised by the amount of reviewing activity given this).
General observation suggests that people won't find writing reviews that intrinsically motivating (compare to just writing posts, which all the authors are doing 'for free' with scant chance of reward, also compare to academia - I don't think many academics find peer review/refereeing one of the highlights of their job). With apologies for the classic classical econ joke, if reviewing was so valuable, ho...
Sure - there's a fair bit of literature on 'optimal stopping' rules for interim results in clinical trials to try and strike the right balance.
It probably wouldn't have helped much for Salk's dilemma: Polio is seasonal and the outcome of interest is substantially lagged from the intervention - which has to precede the exposure, and so the 'window of opportunity' is quickly lost; I doubt the statistical methods for conducting this were well-developed in the 50s; and the polio studies were already some of the largest trial...
Mostly I just find it fascinating that as late as the 1950s, the need for proper randomized blind placebo controls in clinical trials was not universally accepted, even among scientific researchers. Cultural norms matter, especially epistemic norms.
This seems to misunderstand the dispute. Salk may have had an overly optimistic view of the efficacy of his vaccine (among other foibles your source demonstrates), but I don't recall him being a general disbeliever in the value of RCTs.
Rather, his objection is consonant with consensus guidelines for medical...
I'm afraid I couldn't follow most of this, but do you actually mean 'high energy' brain states in terms of aggregate neural activity (i.e. the parentheticals which equate energy to 'firing rates' or 'neural activity')? If so, this seems relatively easy to assess for proposed 'annealing prompts' - whether psychedelics/meditation/music/etc. tend to provoke greater aggregate activity than not seems open to direct calorimetry, leave alone proxy indicators.
Yet the steers on this tend very equivocal (e.g. the ev...
I think this post is referring to "high energy" not in terms of electrochemical neural activity but instead as a metaphor for optimization in machine learning.
Machine learning is the process of minimizing an error function. We can conceptualize this error function as a potential gradient such as a gravity well or electrostatic potential. Minimizing the energy of a particle in this potential gradient is mathematically equivalent to minimizing the error function. The advantage of referring to this as "energy" instead of "error" is it lets you borrow other te
...Thanks for this excellent write-up!
I'm don't have relevant expertise in either AI or SC2, but I was wondering whether precision might still be a bigger mechanical advantage than the write-up notes. Even if humans can (say) max out at 150 'combat' actions per minute, they might misclick, not be able to pick out the right unit in a busy and fast battle to focus fire/trigger abilities/etc, and so on. The AI presumably won't have this problem. So even with similar EAPM (and subdividing out 'non-combat' EAPM which need not be...
Combining the two doesn't solve the 'biggest problems of utilitarianism':
1) We know from Arrhenius's impossibility theorems you cannot get an axiology which can avoid the repugnant conclusion without incurring other large costs (e.g. violations of transitivity, dependence of irrelevant alternatives). Although you don't spell out 'balance utilitarianism' enough to tell what it violates, we know it - like any other population axiology - will have very large drawbacks.
2) 'Balance utilitarianism' seems a long way fr...
(Very minor inexpert points on military history, I agree with the overall point there can be various asymmetries, not all of which are good - although, in fairness, I don't think Scott had intended to make this generalisation.)
1) I think you're right the German army was considered one of the most effective fighting forces on a 'man for man' basis (I recall pretty contemporaneous criticism from allied commanders on facing them in combat, and I think the consensus of military historians is they tended to outfight American, British, and Ru...
It's perhaps worth noting that if you add in some chance of failure (e.g. even if everyone goes stag, there's a 5% chance of ending up -5, so Elliott might be risk-averse enough to decline even if they knew everyone else was going for sure), or some unevenness in allocation (e.g. maybe you can keep rabbits to yourself, or the stag-hunt-proposer gets more of the spoils), this further strengthens the suggested takeaways. People often aren't defecting/being insufficiently public spirited/heroic/cooperative if they aren't 'going to hun...
A marginalist analysis that assumes that the person making the decision doesn’t know their own intentions & is just another random draw of a ball from an urn totally misses this factor.
Happily, this factor has not been missed by either my profile or 80k's work here more generally. Among other things, we looked at:
...[I wrote the 80k medical careers page]
I don't see there as being a 'fundamental confusion' here, and not even that much of a fundamental disagreement.
When I crunched the numbers on 'how much good do doctors do' it was meant to provide a rough handle on a plausible upper bound: even if we beg the question against critics of medicine (of which there are many), and even if we presume any observational marginal response is purely causal (and purely mediated by doctors), the numbers aren't (in EA terms) that exciting in terms of di...
Something that nets out to a small or no effect because large benefits and harms cancel out is very different (with different potential for impact) than something like, say, faith healing, where you can’t outperform just by killing fewer patients. A marginalist analysis that assumes that the person making the decision doesn’t know their own intentions & is just another random draw of a ball from an urn totally misses this factor.
It looks generally redundant in most cases to me: Given how pervasive IQ-correlations are, I think most people can get a reasonable estimate of their IQ by observing their life history so far. E.g.
Obviously, none of these are perfect signals, but I think taking them together usually gives a reasonable steer to a credible range not dramatically larger than test-restest correlations of an IQ test. An IQ test would still provide additional info...
Googling around phrases like 'perception of intelligence' seems to be a keyword for a relevant literature. On a very cursory skim (i.e. no more than what you see here) it seems to suggest "people can estimate intelligence of strangers better than chance (but with plenty of room for error and bias), even with limited exposure". E.g.:
Perceived Intelligence Is Associated with Measured Intelligence in Men but Not Women (Note in this study the assessment was done purely on looking at a photograph of someone's face)
As you say, Bob's good epistemic reputation should count when he says something that appears wild, especially if he has a track record that endorses him in these cases ("We've thought he was crazy before, but he proved us wrong"). Maybe one should think of Bob as an epistemic 'venture capitalist', making (seemingly) wild epistemic bets which are right more often than chance (and often illuminating even if wrong), even if they aren't right more often than not, and this might be enough to warrant further attention ("we...
FWIW: I'm not sure I've spent >100 hours on a 'serious study of rationality'. Although I have been around a while, I am at best sporadically active. If I understand the karma mechanics, the great majority of my ~1400 karma comes from a single highly upvoted top level post I wrote a few years ago. I have pretty sceptical reflexes re. rationality, the rationality community, etc., and this is reflected in that (I think) the modal post/comment I make is critical.
On the topic 'under the hood' here:
I sympathise with the desire to...
I know of a lot of people who continued studying and being interested in the forecasting perspective. I think the primary reason why there has been less writing from that is just that LessWrong was dead for a while, and so we've seen less writeups in general. (I also think there were some secondary factors that also contributed, but that the absence of a publishing platform was the biggest)
There seem some foundational questions to the 'Rationality project', and (reprising my role as querulous critic) are oddly neglected in the 5-10 year history of the rationalist community: conspicuously, I find the best insight into these questions comes from psychology academia.
Is rationality best thought of as a single construct?
It roughly makes sense to talk of 'intelligence' or 'physical fitness' because performance in sub-components positively correlate: although it is hard to say which of an elite ultramarathoner, Judoka,...
On Functional Decision Theory (Wolfgang Schwarz)
I recently refereed Eliezer Yudkowsky and Nate Soares's "Functional Decision Theory" for a philosophy journal. My recommendation was to accept resubmission with major revisions, but since the article had already undergone a previous round of revisions and still had serious problems, the editors (understandably) decided to reject it. I normally don't publish my referee reports, but this time I'll make an exception because the authors are well-known figures from outside academia, and I...
Relevant excerpt for why exactly it was rejected:
The standards for deserving publication in academic philosophy are relatively simple and self-explanatory. A paper should make a significant point, it should be clearly written, it should correctly position itself in the existing literature, and it should support its main claims by coherent arguments. The paper I read sadly fell short on all these points, except the first. (It does make a significant point.) [...]
I still think the paper could probably have been published after a few rounds of major revisions...
I'm not gonna go comment on his blog because his confusion about the theory (supposedly) isn't related to his rejection of the paper, and also because I think talking to a judge about the theory out of band would bias their judgement of the clarity of the writing in future (it would come to seem more clear and readable to them than it is, just as it would to me) and is probably bad civics, but I just have to let this out because someone is wrong on the internet, damnit
FDT says you should not pay because, if you were the kind of person who doesn&#...
I'm someone who both prefers and practises the 'status quo'.
My impression is the key feature of this is limited (and author controlled) sharing. (There are other nifty features for things like gdocs - e.g. commenting 'on a line' - but this practice predates gdocs). The key benefits for 'me as author' are these:
1. I can target the best critics: I usually have a good idea of who is likely to help make my work better. If I broadcast, the mean quality of feedback almost certainly goes down.
2. I can leverage existing relatio...
if Alice sees Bob make good remarks etc., she’s more interested in ‘running a draft by him’ next time, or to respond positively if Bob asks her to look something over
This dynamic contributes to anxiety for me to comment in Google Docs, and makes it less fun than public commenting (apparently the opposite of many other people). I feel like if I fail to make a good contribution, or worse, make a dumb comment, I won't be invited to future drafts and will end up missing a lot of good arguments, or entire articles because many drafts don't get published unti
...I don't see 'comments going to waste' issue as the greatest challenge
I think this underestimates the challenge. Empirically, people don't crosspost those comments. Periodically saying "hey it'd be good if you crossposted those private comments" won't change the underlying incentive structure.
(Similarly, the fact that one 'could' keep an eye out for posts and comments from outsiders won't change the fact that people generally don't)
Once again I plead that when you see that an expert community looks like they don't know what their doing, it is usually more accurate to 'reduce confidence' in your understanding rather than their competence. The questions were patently not 'about forms', and covered pretty well the things I would have in mind (I'm a doctor, and I have fairly extensive knowledge of medical ethics).
To explain:
I don't see the 'why aren't you winning?' critique as that powerful, and I'm someone who tends critical of rationality writ-large.
High-IQ societies and superforecasters select for demonstrable performance at being smart/epistemically rational. Yet on surveying these groups you see things like, "People generally do better-than-average by commonsense metrics, some are doing great, but it isn't like everyone is a millionaire". Given the barrier to entry to the rationalist community is more, "sincere interest" ...
Another thing I'd be particularly interested in is longer term follow-up. It would be impressive if the changes to conscientiousness etc. observed in the 2015 study persist now.
I'd be hesitant to defend Great Man theory (and so would apply similar caution) but I think it can go some way, especially for defending a fragility of history hypothesis.
In precis (more here):
1. Conception of any given person seems very fragile. If parents decide to conceive an hour earlier or later (or have done different things earlier in the day, etc. etc.), it seems likely another one of the 100 million available sperm fuses than the one which did. The counterpart seems naturally modelled by a sibling, and siblings are considerably different from...
I not sure t-tests are the best approach to take compared to something non-parametric, given smallish sample, considerable skew, etc. (this paper's statistical methods section is pretty handy). Nonetheless I'm confident the considerable effect size (in relative terms, almost a doubling) is not an artefact of statistical technique: when I plugged the numbers into a chi-squared calculator I got P < 0.001, and I'm confident a permutation technique or similar would find much the same.
0: We agree potentially hazardous information should only be disclosed (or potentially discovered) when the benefits of disclosure (or discovery) outweigh the downsides. Heuristics can make principles concrete, and a rule of thumb I try to follow is to have a clear objective in mind for gathering or disclosing such information (and being wary of vague justifications like ‘improving background knowledge’ or ‘better epistemic commons’) and incur the least possible information hazard in achieving this.
A further heuristic which seems right to me is one shoul...
Thanks for writing this. How best to manage hazardous information is fraught, and although I have some work in draft and under review, much remains unclear - as you say, almost anything could have some some downside risk, and never discussing anything seems a poor approach.
Yet I strongly disagree with the conclusion that the default should be to discuss potentially hazardous (but non-technical) information publicly, and I think your proposals of how to manage these dangers (e.g. talk to one scientist first) generally err too lax. I provide the substance of...
Thanks for this and subsequent comment which generally helped me to update my views on the problem and become even more cautious in discussing things.
Some thoughts appeared in my mind while reading, maybe I will have more thoughts later:
1. It looks like that all the talk about infohazards could be boiled down to just one thesis: "biorisk is much more serious x-risk than AI safety, but we decided not to acknowledge it, as it could be harmful".
2. Almost all work in AI safety is based on "red-teaming": someone comes with an idea X how to...
This seems right to me, and at least the 'motte' version of growth mindset accepts that innate ability may set pretty hard envelopes on what you can accomplish regardless of how energetic/agently you pursue self improvement (and this can apply across a range of ability - although it seems cruel and ludicrous to suggest someone with severe cognitive impairment can master calculus, it also seems misguided to suggest someone in middle age can become a sports star if they really go for it). As you say, taking growth mindset 'too far' has a ...
A healthy topology of the field should have approximately power-law distribution of hub sizes. This should be true also for related research fields we are trying to advance, like AI alignment or x-risk. If the structure is very far from that (e.g. one or two very big hubs, than nothing, than a lot of two orders of magnitude smaller groups fighting for mere existence), the movement should try to re-balance, supporting growth of medium-tier hubs.
Although my understanding of network science is abecedarian, I'm unsure of both whether this feature is diag...
+1
It also risks a backfire effect. If one is in essence a troll happy to sneer at what rationalists do regardless of merit (e.g. "LOL, look at those losers trying to LARP enders game!"), seeing things like Duncan's snarky parenthetical remarks would just spur me on, as it implies I'm successfully 'getting a rise' out of the target of my abuse.
It seems responses to criticism that is unpleasant or uncharitable are best addressed specifically to the offending remarks (if they're on LW2, this seems like pointing out the fall...
I also think I got things about right, but I think anyone else taking an outside view would've expected roughly the same thing.
I think you might be doing yourself a disservice. I took the majority of contemporary critcism was more directed towards (in caricature) 'this is going to turn into a nasty cult!' than (what I took your key insight to be) 'it will peter out because the commander won't actually have the required authority'.
So perhaps the typical 'anyone else' would have alighted on the wrong outside view, or ...
Bravo - I didn't look at the initial discussion, or I would have linked your pretty accurate looking analysis (on re-skimming, Deluks also had points along similar lines). My ex ante scepticism was more a general sense than a precise pre-mortem I had in mind.
Although I was sufficiently sceptical of this idea to doubt it was 'worth a shot' ex ante,(1) I was looking forward to being pleasantly surprised ex post. I'm sorry to hear it didn't turn out as well as hoped. This careful and candid write-up should definitely be included on the 'plus' side of the ledger for this project.
With the twin benefits of no skin in the game and hindsight. I'd like to float another account which may synthesize a large part of 'why it didn't work'.
Although I understand DAB wasn'...
I assume the legal 'fact on the ground' is that the participants of DAB were co-signatories on a lease, making significant financial contributions, with no mechanism for the designated 'commander' to kick people out unilaterally.
This is approximately correct--not all of us were on the lease, and not all of us were making significant financial contributions. But someone who was on the lease and was making a significant financial contribution could have made it highly difficult to evict them, even if everyone else in the house wanted them...
This new paper may be of relevance (H/T Steve Hsu). The abstract:
The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking. Sometimes, we are willing to admit that a certain degree of luck could also play a role in achieving significant material success. But, as a matter of fact, it is rather common to underestimate the importance of external forces in individual successful storie...
I endorse Said's view, and I've written a couple of frontpage posts.
I also add that I think Said is a particularly able and shrewd critic, and I think LW2 would be much poorer if there was a chilling effect on his contributions.
Let's focus on the substance, please.
I'm also mystified at why traceless deletition/banning are desirable properties to have on a forum like this. But (with apologies to the moderators) I think consulting the realpolitik will spare us the futile task of litigating these issues on the merits. Consider it instead a fait accompli with the objective to attract a particular writer LW2 wants by catering to his whims.
For whatever reason, Eliezer Yudkowsky wants to have the ability to block commenters and have the ability to do traceless deletion on his own work, and he's been quite clear t...
Yeah, I didn't want to make this a thread about discussing Eliezer's opinion, so I didn't put that front and center, but Eliezer only being happy to crosspost things if he has the ability to delete things was definitely a big consideration.
Here is my rough summary of how this plays into my current perspective on things:
1. Allowing users to moderate their own posts and set their own moderation policies on their personal blogs is something I wanted before we even talked to Eliezer about LW2 the first time.
2. Allowing users to moderate ...
FWIW, I struggle to navigate the front page to look at good posts (I struggle to explain why - I think I found 'frontpage etc.' easier for earlier versions). What I do instead is look at the comments feed and click through to articles that way, which seems suboptimal, as lots of comments may not be a very precise indicator of quality.
FWIW, this aptly describes my own adverse reaction to the OP. "I have this great insight, but I not only can't explain it to you, but I'm going to spend the balance of my time explaining why you couldn't understand it if I tried to explain it" sounds awfully close to bulveristic stories like, "If only you weren't blinded by sin, you too would see the glory of the coming of the lord".
That the object level benefits offered seem to be idiographic self-exhaltations augur still poorer (i.e. I cut through confusion so m...
I was unaware of the range restriction, which could well compress SD. That said, if you take the '9' scorers as '9 or more', then you get something like this (using 20-25)
Mean value is around 7 (6.8), 7% get 9 or more, suggesting 9 is at or around +1.5SD assuming normality, so when you get a sample size in the thousands, you should start seeing scores at 11 or so (+3SD) - I wouldn't be startled to find Ben has this level of ability. But scores at (say) 15 or higher (+6SD) should only be seen with extraordinarily rarely.
If you use ...
I'm aware of normalisation, hence I chose things which have some sort of 'natural cardinal scale' (i.e. 'how many Raven's do you get right' doesn't really work, but 'how many things can you keep in mind at once' is better, albeit imperfect).
Not all skew entails a log-normal (or some similar - assumedly heavy tailed) distribution. This applies to your graph for digit span you cite here. The mean of the data is around 5, and the SD is around 2. Having ~11% at +1SD (7) and about 3% at +2SD (9) is a lot closer to n...
Sorry you disliked the post so much. But you might have liked it more had you looked at the bit titled 'community benefits to immodesty', where I talk about the benefits of people striking out outside expert consensus (but even if they should act 'as if' their contra-expert take was correct, they should nonetheless defer to it for 'all things considered' views).
No. I chose him as a mark of self-effacement. When I was younger I went around discussion forums about philosophy, and people commonly named themselves after ancient greats like Socrates, Hume, etc. Given Thrasymachus's claim to fame is being rude, making some not great arguments, and getting spanked by Socrates before the real discussion started (although I think most experts think Socrates's techne based reply was pretty weak), I thought he would be a more accurate pseudonym,
Sorry for misreading your original remark. Happy to offer the bet in conditional, i.e.:
Conditional on CFAR producing results of sufficient quality for academic publication (as judged by someone like Christiano or Karnofsky) these will fail to demonstrate benefit on a pre-specified objective outcome measure
Thanks for your reply. Given my own time constraints I'll decline your kind offer to discuss this further (I would be interested in reading some future synthesis). As consolation, I'd happily take you up on the modified bet. Something like:
Within the next 24 months CFAR will not produce results of sufficient quality for academic publication (as judged by someone like Christiano or Karnofsky) that demonstrate benefit on a pre-specified objective outcome measure
I guess 'demonstrate benefit' could be stipulated as 'p<0.05 on some a
I hope readers will forgive a 'top level' reply from me, it's length, and that I plan to 'tap out' after making it (save for betting). As pleasant as this discussion is, other demands pull me elsewhere. I offer a summary of my thoughts below - a mix of dredging up points I made better 3-4 replies deep than I managed in the OP, and to reply to various folks at CFAR. I'd also like to bet (I regret to decline Eli's offer for reasons that will become apparent, but I hope to make some agreeable counter-offers).
I persist in thr
I also notice that I can't predict whether you'll look at the "prioritize discussion based on the slope of your possible update combined with the other party's belief" version that I give here and say "okay, but that's not double crux" or "okay, but the motion of double crux doesn't point there as efficiently as something else" or "that doesn't seem like the right step in the dance, tho."
I regret it is unclear what I would say given what I have written, but it is the former ("okay,
Hello Dan,
I'm not sure whether these remarks are addressed 'as a reply' to me in particular. That you use the 'marginal tax rate in the UK' example I do suggests this might be meant as a response. On the other hand, I struggle to locate the particular loci of disagreement - or rather, I see in your remarks an explanation of double crux which includes various elements I believe I both understand and object to, but not reasons that argue against this belief (e.g. "you think double crux involves X, but actually it is X*, and thus
Thanks for presenting this helpful data. If you'll forgive the (somewhat off topic) question, I understand both that you are responsible for evaluation of CFAR, and that you are working on a new evaluation. I'd be eager to know what this is likely to comprise, especially (see various comments) what evidence (if any) is expected to be released 'for public consumption'?
I see the concerns as these:
- The four corners of the agreement seem to define 'disparagement' broadly, so one might reasonably fear (e.g.) "First author on an eval especially critical of OpenAI versus its competitors", or "Policy document highly critical of OpenAI leadership decisions" might 'count'.
- Given Altman's/OpenAI's vindictiveness and duplicity, and the previous 'safeguards' (from their perspective) which give them all the cards in terms of folks being able to realise the value of their equity, "They will screw me out of a lot of money if I do someth
... (read more)