All of Larks's Comments + Replies
Thanks very much for your feedback, though I confess I'm not entirely sure where to go with it. My interpretation is you have basically two concerns:
- This policy doesn't really directly regulate algorithmic progress, e.g. if it happened on smaller amounts of compute.
- Algorithmic theft/leakage is easy.
The first one is true, as I alluded in the problems section. Part of my perspective here is coming from a place of skepticism about regulatory competence - I basically believe we can get regulators to control total compute usage, and to evaluate specific models ...
1You probably should have said 'yes' when asked if it was AI-written.
This started happening in Hawaii, and to a lesser extent in Arizona. The resolution, apart from reducing net metering subsidies, has been to increased the fixed component of the bill (which pays for the grid connection) and reduce the variable component. My impression is this has been a reasonably effective solution, assuming people don't want to cut their connection entirely.
I agree with you that basically anything in the stock market has much less counterparty risk than that. I disagree with basically all non-trading examples you give.
It's not just the stock market, it's true for the bond market, the derivatives market, the commodities market... financial markets, a category which includes prediction markets, cannot function effectively with counterparty risk anything like 5%.
My sense is around 1/20 Ubers don't show up, or if they show up, fail to do their job in some pretty obvious and clear way.
If the Uber...
In most domains except the most hardened part of the stock market counterparty risk is generally >5%.
This seems quite wrong to me:
- High Yield Corporate Bond OAS spreads are <5% according to bloomberg, and most of that is economic risk, not "you will get screwed by a change of rules" risk.
- Trades on US stock exchanges almost always succeed, many more 9s than just one.
- If I buy a product in a box in a supermarket the contents of the box match the label >>95% of the time.
- Banks make errors with depositor balances <<5% of the time.
- Most employers
A bit dated but have you read Robin's 2007 paper on the subject?
...Prediction markets are low volume speculative markets whose prices offer informative forecasts on particular policy topics. Observers worry that traders may attempt to mislead decision makers by manipulating prices. We adapt a Kyle-style market microstructure model to this case, adding a manipulator with an additional quadratic preference regarding the price. In this model, when other traders are uncertain about the manipulator’s target price, the mean target price has no effect on prices, and
Yes, sorry for being unclear. I meant to suggest that this argument implied 'accelerate agents and decelerate planners' could be the desirable piece of differential progress.
1This post seems like it was quite influential. This is basically a trivial review to allow the post to be voted on.
L'Ésswrong, c'est moi.
1
1I agree in general, but think the force of this is weaker in this specific instance because NonLinear seems like a really small org. Most of the issues raised seem to be associated with in-person work and I would be surprised if NonLinear ever went above 10 in-person employees. So at most this seems like one order of magnitude in difference. Clearly the case is different for major corporations or orgs that directly interact with many more people.
Note that one of the reasons why I cared about getting this report out was that Nonlinear was getting more influential as a middleman in the AI Safety funding ecosystem, through which they affected many people's lives and I think had influence beyond what a naive headcount would suggest. The Nonlinear network had many hundreds of applications.
As a personal example, I also think Lightcone, given that its at the center of a bunch of funding stuff, and infrastructure work, should also be subject to greater scrutiny than specific individuals, given the number of individuals that are affected by our work. And we are about the same size as Nonlinear, I think.
I think there will be some degree to which clearly demonstrating that false accusations were made will ripple out into the social graph naturally (even with the anonymization), and will have consequences. I also think there are some ways to privately reach out to some smaller subset of people who might have a particularly good reason to know about this.
If this is an acceptable resolution, why didn't you just let the problems with NonLinear ripply out into the social graph naturally?
I think that's a good question, and indeed I think that should be the default thing that happens!
In this case we decided to do something different because we received a lot of evidence that Nonlinear was actively suppressing negative information about them. As Ben's post states, the primary reason we got involved with this was that we heard Nonlinear was actively pressuring past employees to not say bad things about them, and many employees we talked to fely very scared of retribution if they told anyone about this, even privately, as long as it could some...
If most firms have these clauses, one firm doesn't, and most people don't understand this, it seems possible that most people would end up with a less accurate impression of their relative merits than if all firms had been subject to equivalent evidence filtering effects.
In particular, it seems like this might matter for Wave if most of their hiring is from non-EA/LW people who are comparing them against random other normal companies.
1I would typically aim for mid-December, in time for the American charitable giving season.
After having written an annual review of AI safety organisations for six years, I intend to stop this year. I'm sharing this in case someone else wanted to in my stead.
Reasons
- It is very time consuming and I am busy.
- I have a lot of conflicts of interests now.
- The space is much better funded by large donors than when I started. As a small donor, it seems like you either donate to:
- A large org that OP/FTX/etc. support, in which case funging is ~ total and you can probably just support any.
- A large org than OP/FTX/etc. reject in which case there is a high chance
I will miss the annual updates. I didn't care about the question of who to donate to, but it was always good for catching up on research I missed in the flood, and a great starting point for beginners - I could just tell them to read a dozen papers from each annual review which sounded interesting and they'd have a great overview of how things were going.
Thank you for the work you've put in!
Thanks!
Alignment research: 30
Could you share some breakdown for what these people work on? Does this include things like the 'anti-bias' prompt engineering?
It includes the people working on the kinds of projects I listed under the first misconception. It does not include people working on things like the mitigation you linked to. OpenAI distinguishes internally between research staff (who do ML and policy research) and applied staff (who work on commercial activities), and my numbers count only the former.
I would expect that to be the case for staff who truly support faculty. But many of them seem to be there to directly support students, rather than via faculty. The number of student mental health coordinators (and so on) you need doesn't scale with the number of faculty you have. The largest increase in this category is 'student services', which seems to be definitely of this nature.
Thanks very much for writing this very diligent analysis.
I think you do a good job of analyzing the student/faculty ratio, but unless I have misread it seems like this is only about half the answer. 'Support' expenses rose by even more than 'Instruction', and the former category seems less linked to the diversity of courses offered than to things like the proliferation of Deans, student welfare initiatives, fancy buildings, etc.
Thanks, that's very kind of you!
Is your argument about personnel overlap that one could do some sort of mixed effect regression, with location as the primary independent variable and controls for individual productivity? If so I'm so somewhat skeptical about the tractability: the sample size is not that big, the data seems messy, and I'm not sure it would capture necessarily the fundamental thing we care about. I'd be interested in the results if you wanted to give it a go though!
More importantly, I'm not sure this analysis would be that useful. Geography-based-priors only really seem us...
Thanks, fixed in both copies.
Thanks, fixed.
Should be fixed, thanks.
Changed in both copies as you request.
- I prioritized posts by named organizations.
- Diffractor does not list any institutional affiliations on his user page.
- No institution I noticed listed the post/sequence on their 'research' page.
- No institution I contacted mentioned the post/sequence.
- No post in the sequence was that high in the list of 2021 Alignment Forum posts, sorted by karma.
- Several other filtering methods also did not identify the post
However upon reflection it does seem to be MIRI-affiliated so perhaps should have been affiliated; if I have time I may review and edit it in later.
Notice that in MIRI's summary of 2020 they wrote "From our perspective, our most interesting public work this year is Scott Garrabrant’s Cartesian frames model and Vanessa Kosoy’s work on infra-Bayesianism."
13 years later: did anyone end up actually making such a book?
The labels on the life satisfaction chart appear to be wrong; January 2021 comes before December 2020.
Well, with hemispherectomy, those problems are no more. Hemispherectomy is a procedure where half of the brain is removed. It has been performed multiple times without any apparent complications (example).
I was skeptical until I read the example. Now I am convinced!
It's hard to sell 1 million eggs for one price, and 1 million for another price.
Are you sure this is the case? It's common for B2B transactions to feature highly customised and secret pricing and discounts. And in this case they're not selling the same product from the customer's point of view: one buyer gets a million ethical eggs, while another gets a million ordinary (from their point of view) eggs.
Thanks for writing this; ordered.
A teacher in year 9 walked up to a student who was talking, picked them up and threw them out of an (open) first floor window.
Worth clarifying for US readers that 'first floor' in the UK would be 'second floor' in the US, because UK floor indexing starts at zero. So this event is much worse than it sounds.
Thanks, added.
At the moment, the poor person and the rich person are both buying things. If the rich person buys more vaccine, that means they will buy less of the other things, so the poor person will be able to have more of them. So the question is about the ratios of how much the two guys care about the vaccine and how much they care about the other thing... and the answer is the rich guy will pay up for the vaccine when his vaccine:other ratio is higher than the other guys. This is the efficient allocation.
It might be the case that it is separately desirable to redi...
At the moment, the poor person and the rich person are both buying things. If the rich person buys more vaccine, that means they will buy less of the other things, so the poor person will be able to have more of them. So the question is about the ratios of how much the two guys care about the vaccine and how much they care about the other thing... and the answer is the rich guy will pay up for the vaccine when his vaccine:other ratio is higher than the other guys.
This is only true if the rich person is already spending as much money as possible, so an incr...
Hey Daniel, thanks very much for the comment. In my database I have you down as class of 2020, hence out of scope for that analysis, which was class of 2018 only. I didn't include 2019 or 2020 classes because I figured it takes time to find your footing, do research, write it up etc., so absence of evidence would not be very strong evidence of absence. So please don't consider this as any reflection on you. Ironically I actually did review one of your papers in the above - this one - which I did indeed think was pretty relevant! (Cntrl-F 'Hendrycks' to find the paragraph in the article). Sorry if this was not clear from the text.
Larks, excellent name choice for your AttackBot.
Thanks! I figured it was in the spirit of a DefectBot to defect linguistically as well, and there was a tiny chance someone might be doing naive string-matching.
You will have to wait for next time's obituary I'm afraid! I think Isusr should have a good grasp on the philosophical and ethical traditions I was attempting to channel with CooperateBot - while the insights are deep, I think the lengthy code is quite clear on the matter.
I actually have no idea - I guess we are just two naturally very cooperative people!
Cool competition! It makes me wish I had had more time to put into CooperateBot. At present I would say it instantiated a relatively naive view of cooperation, and could do much better if I invested more time considering the true nature of generosity. Looking at the obituary I suspect that CooperateBot may not last much longer.
Holding constant the total amount of taxes you pay, it is better not to get a refund. This is the perspective you should take at the beginning of the year.
Holding constant the amount of taxes you have already paid, it is better to get a refund. This is the perspective you should take at the end of the year.
I attempted to produce a rough estimate of this here (excerpted below):
... One (BERI funded!) study suggested that banning large gatherings reduced r0 by around 28%.
Unfortunately, protests seem in many ways ideal for spreading the disease. They involve a large number of people in a relatively small area for an extended period of time. Even protests which were advertised as being socially distanced often do not end up that way. While many people wear masks, photos of protests make clear that many do not, and those that are are often using cloth masks...
I still found this helpful as it allowed me to exit my directional Yang and Buttigieg positions with negative transaction cost.
I would like to add that I think this is bad (and have the codes). We are trying to build social norms around not destroying the world; you are blithely defecting against that.
This case is more complicated than the corporate cases because the powerful person (me) was getting merely the appearance of what she wanted (a genuine relationship with a compatible person), not the real thing. And because the exploited party was either me or Connor, not a third party like bank customers. No one thinks the Wells Fargo CEO was a victim the way I arguably was.
I think you have misunderstood the Wells Fargo case. These fake accounts generally didn't bring in any material revenue; they were just about internal 'number of new a...
Hmm. I'm not sure I fully understand the Wells Fargo case but I interpreted it as a concern between four parties:
1. The people who got fake accounts signed up for them.
2. The employees doing the fake signups
3. A middle management tier, which set quotas
4. A higher level management tier that (presumaby?) wanted middle management to actually be making money.
So, the people being defrauded are not the customers, but the higher management tier, basically. (But, also, this entire thing might just be a weird game that middle management tiers play with each ot...
Is this very different from founding a pharmaceutical company?
Critch wrote a related paper:
...Existing multi-objective reinforcement learning (MORL) algorithms do not account for objectives that arise from players with differing beliefs.Concretely, consider two players with different beliefs and utility functions who may cooperate to build a machine that takes actions on their behalf. A representation is needed for how much the machine’s policy will prioritize each player’s interests over time. Assuming the players have reached common knowledge of their situation, this paper derives a recursion that any Pareto optimal
Thanks very much! Yeah, I agree political will seems like a big issue. But I also hear people saying that they don't know what to push for, so I wanted to try to offer a concrete example of a system that wasn't as destructive to any constituency's interests as e.g. a total pause.