It may be useful to know that if events all obey the Markov property (they are probability distributions, conditional on some set of causal parents), then the Reichenbach Common Cause Principle follows (by d-separation arguments) as a theorem. So any counterexamples to RCCP must violate the Markov property as well.
There's also a lot of interesting discussion here.
The idea that "Agents are systems that would adapt their policy if their actions influenced the world in a different way." works well on mechanised CIDs whose variables are neatly divided into object-level and mechanism nodes: we simply check for a path from a utility function F_U to a policy Pi_D. But to apply this to a physical system, we would need a way to obtain such a partition those variables. Specifically, we need to know (1) what counts as a policy, and (2) whether any of its antecedents count as representations of "influence" on the world (and af...
Nice. I've previously argued similarly that if going for tenure, AIS researchers might places that are strong in departments other than their own, for inter-departmental collaboration. This would have similar implications to your thinking about recruiting students from other departments. But I also suggested we should favour capital cities, for policy input, and EA hubs, to enable external collaboration. But tenure may be somewhat less attractive for AIS academics, compared to usual, in that given our abundant funding, we might have reason to favour Top-5 ...
Feature suggestion. Using highlighting for higher-res up/downvotes and (dis)agreevotes.
Sometimes you want to indicate what part of a comment you like or dislike, but can't be bothered writing a comment response. In such cases, it would be nice if you could highlight the portion of text that you like/dislike, and for LW to "remember" that highlighting and show it to other users. Concretely, when you click the like/dislike button, the website would remember what text you had highlighted within that comment. Then, if anyone ever wants to see that highlighting...
Hi Orellanin,
In the early stages, I had in mind that the more info any individual anon-account revealed, the more easily one could infer what time they spent at Leverage, and therefore their identity. So while I don't know for certain, I would guess that I created anonymoose to disperse this info across two accounts.
When I commented on the Basic Facts post as anonymoose, It was not my intent to contrive a fake conversation between two entities with separate voices. I think this is pretty clear from anonymoose's comment, too - it's in the same bulleted and ...
"A Russian nuclear strike would change the course of the conflict and almost certainly provoke a "physical response" from Ukraine's allies and potentially from the North Atlantic Treaty Organization, a senior NATO official said on Wednesday.
Any use of nuclear weapons by Moscow would have "unprecedented consequences" for Russia, the official said on the eve of a closed-door meeting of NATO's nuclear planning group on Thursday.
Speaking on condition of anonymity, he said a nuclear strike by Moscow would "almost certainly be drawing a physical response from ma...
The reasoning is that retaliating is US doctrine - they generally respond to hostile actions in-kind, to deter them. If Ukraine got nuked, the level of outrage would place intense pressure on Biden to do something, and the hawks would become a lot louder than the doves, similar to after the 9/11 attacks. In the case of Russia, the US has exhausted most non-military avenues already. And US is a very militaristic country - they have many times bombed countries (Syria, Iraq, Afghanistan, Libya) for much less. So military action just seems very likely. (Involv...
I think your middle number is clearly too low. The risk scenario does not require that NATO trigger article 5 necessarily, but just that they carry out a strategically significant military response, like eliminating Russia's Black Sea Fleet, nuking, or creating a no-fly zone. And Max's 80% makes more sense than your 50% for he union of these possibilities, because it is hard to imagine that the US would stand down without penalising the use of nukes.
I would be at maybe .2*.8*.15=.024 for this particular chain of events leading to major US-Russia nuclear war.
It seems like you're saying that the practical weakness of forecasters vs experts is their inability to make numerous causal forecasts. Personally, I think the causal issue is the main issue, whereas you think it is that the predictions are so numerous. But they are not always numerous - sometimes you can affect big changes by intervening at a few pivot points, such as at elections. And the idea that you can avoid dealing with causal interventions by conditioning on every parent is usually not practical, because conditioning on every parent/confounder mean...
You could test this to some extent by asking the forecasters to predict more complicated causal questions. If they lose most of their edge, then you may be right.
I don't think the capital being locked up is such a big issue. You can just invest everyone's money in bonds, and then pay the winner their normal return multiplied by the return of the bonds.
A bigger issue is that you seem to only be describing conditional prediction markets, rather than ones that truly estimate causal quantities, like P(outcome|do(event)). To see this, note that the economy will go down IF Biden is elected, whereas it is not decreased much by causing Biden to be elected. The issue is that economic performance causes Biden to be unpopular...
You say two things that seem in conflict with one another.
...[Excerpt 1] If a system is well-described by a causal diagram, then it satisfies a complex set of statistical relationships. For example ... To an evidential decision theorist, these kinds of statistical relationships are the whole story about causality, or at least about its relevance to decisions.
[Excerpt 2] [Suppose] that there is a complicated causal diagram containing X and Y, such that my beliefs satisfy all of the statistical relationships implied by that causal diagram. EDT recommends
Many people don't realize how effective migraine treatments are. High-dose aspirin, tryptans, and preventers all work really well, and can often reduce migraine severity by 50-90%.
Also, most don't yet realise how effective semaglutide is for weight loss, due to the fact that weight loss drugs have generally been much less effective, or had much worse side-effects previously.
Balding treatments (finasteride and topical minoxodil) are also pretty good for a lot of people.
Another possibility is that most people were reluctant to read, summarise, or internalise Putin's writing on Ukraine due to finding it repugnant, because they aren't decouplers.
Off the top of my head, maybe it's because Metaculus' presents medians, and the median user neither investigates the issue much, nor trusts those who do (Matt Y, Scott A) and just roughly follows base rates. I also feel there was some wishful thinking, and that to some extent, the fullness of the invasion was at least somewhat intrinsically surprising.
Yeah I think in practice auctioning every day or two would be completely adequate - that's much less than the latency involved in dealing with lawyers and other aspects of the process. So now I'm mostly just curious about whether there's a theory built up for these kinds of problems in the continuous time case.
We do not have a scientific understanding of how to tell a superintelligent machine to "solve problem X, without doing something horrible as a side effect", because we cannot describe mathematically what "something horrible" actually means to us...
Similar to how utility theory (from von Neumann and so on) is excellent science/mathematics despite our not being able to state what utility is. AI Alignment hopes to tell us how to align AI, not the target to aim for. Choosing the target is also a necessary task, but it's not the focus of the field.
In terms of trying to formulate rigorous and consistent definitions, a major goal of the Causal Incentives Working Group is to analyse features of different problems using consistent definitions and a shared framework. In particular, our paper "Path-specific Objectives for Safer Agent Incentives" (AAAI-2022) will go online in about month, and should serve to organize a handful of papers in AIS.
Exactly. Really, the title should be "Six specializations makes you world-class at a combination of skills that is probably completely useless." Really, productivity is a function of your skills. The fact that you are "world class" in a random combination of skills is only interesting if people are systematically under-estimating the degree to which random skills can be usefully combined. If there are reasons to believe that, then I would be interested in reading about it.
Transformer models (like GPT-3) are generators of human-like text, so they can be modeled as quantilizers. However, any quantiliser guarantees are very weak, because they quantilise with very low q, equal to the likelihood that a human would generate that prompt.
The most plausible way out seems to be for grantmakers to grant money conditionally on work being published as open source. Some grantmakers may benefit from doing this, despite losing some publication prestige, because the funded work will be read more widely, and the grantmaker will look like they are improving the scientific process. Researchers lose some prestige, but gain some funding. Not sure how well this has worked so far, but perhaps we could get to the world where this works, if we're not already there.
Elsevier journals allow individual authors to make their own published paper "immediately and permanently free for everyone to read and download" for a fee. In the last Elsevier journal I submitted a paper to, the fee was $2,400.
I think this means that a grant conditioned on open-access publishing would just mean that authors will have to pay the fee if they publish in an Elsevier journal -- this makes it more like a tax (paid out of grant money) than a ban. Not sure if that would make it more or less effective, on net, though.
It would be useful to have a clarification of these points, to know how different of an org you actually encountered, compared to the one I did when I (briefly) visited in 2014.
It is not true that people were expected to undergo training by their manager.
OK, but did you have any assurance that the information from charting was kept confidential from other Leveragers? I got the impression Geoff charted people who he raised money from, for example, so it at least raises the question whether information gleaned from debugging might be discussed with that pers...
Thanks for your courage, Zoe!
Personally, I've tried to maintain anonymity in online discussion of this topic for years. I dipped my toe into openly commenting last week, and immediately received an email that made it more difficult to maintain anonymity - I was told "Geoff has previously speculated to me that you are 'throwaway', the author of the 2018 basic facts post". Firstly, I very much don't appreciate my ability to maintain anonymity being narrowed like this. Rather, anonymity is a helpful defense in any sensitive online discussion, not least this o...
...What's frustrating about still hearing noisy debate on this topic, so many years later, is that Leverage being a really bad org seems overdetermined at this point. On the one hand, if I ranked MIRI, CFAR, CEA, FHI, and several startups I've visited, in terms of how reality-distorting they can be, Leverage would score ~9, while no other would surpass ~7. (It manages to be nontransparent and cultlike in other ways too!). While on the other hand, their productive output was... also like a 2/10? It's indefensible. But still only a fraction of the relevan
As in, 5+ years ago, around when I'd first visited the Bay, I remember meeting up 1:1 with Geoff in a cafe. One of the things I asked, in order to understand how he thought about EA strategy, was what he would do if he wasn't busy starting Leverage. He said he'd probably start a cult, and I don't remember any indication that he was joking whatsoever. I'd initially drafted my comment as "he told me, unjokingly", except that it's a long time ago, so I don't want to give the impression that I'm quite that certain.
He's also told me, deadpan, that he would like to be starting a cult if he wasn't running Leverage.
Your comparison does a disservice to the human's sample efficiency in two ways:
So you seem to have it backwards when you say that the comparison that everyone is making is the "bad" one.
I think this becomes a lot clearer if we distinguish between total and marginal thinking. GPT-3's total sample efficiency for predicting text is poor:
But on-the-margin, it's very sample efficient at learning to perform new text-related tasks:
Essentially, what's happen...
That does help, thanks. However, now that I understand better what people are saying, I think it's wrong:
The comparison they are making is as follows:
| GPT-3 | Human |
| Pre-trained on 3x10^11 tokens of text | Pre-trained on 3x10^8 tokens of text (fermi estimate based on WMP 300 so maybe 500 tokens per minute, 10 hours per week reading, 52 weeks a year, over 20 years of life) |
| Able to read a new fact once or twice and then learn it / remember it. | Able to read a new fact once or twice and then learn it / remember it |
However, I think this is a bad comparison, because it igno...
The implication seems to be that this RFP is for AIS work that is especially focused on DL systems. Is there likely to be a future RFP for AIS research that applies equally well to DL and non-DL systems? Regardless of where my research lands, I imagine a lot of useful and underfunded research fits in the latter category.
To me, his main plausible x important claim is that performance is greatly improved by subject specialisation from age <5. The fact that many geniuses enter their fields late doesn't falsify this, since that isn't humdrum at all - barely one in a million kids specialise in that way. I think that people who enter a field such as CS at age 30, rather than at age 20 do have a mild disadvantage, maybe 0.5SD. So I wouldn't be surprised if starting at age 4, rather than at age 20 gave you another 1-2SD advantage. Of course, subjects like CS do benefit a lot f...
Thanks for these thoughts about the causal agenda. I basically agree with you on the facts, though I have a more favourable interpretation of how they bear on the potential of the causal incentives agenda. I've paraphrased the three bullet points, and responded in reverse order:
3) Many important incentives are not captured by the approach - e.g. sometimes an agent has an incentive to influence a variable, even if that variable does not cause reward attainment.
-> Agreed. We're starting to study "side-effect incentives" (improved name pending), whic...
This covers pretty well the altruistic reasons for/against working on technical AI safety at a frontier lab. I think the main reason for working at a frontier lab, however, is not altruistic. It's that it offers more money and status than working elsewhere - so it would be nice to be clear-eyed about this.
To be clear, on balance, I think it's pretty reasonable to want to work at a frontier lab, even based on the altruistic considerations alone.
What seems harder to justify altruistically, however, is why so many of us work on, and fund the same kinds ... (read more)