If you go around an EA conference saying, "I don't think it's worthwhile to work on AI safety research, because most of the research groups I've seen haven't put out useful work," every attendee will find a reason you should work on it anyway, and probably switch fields to try to join one of those research groups.

This sounds false to me. Have you tried it? 

Thanks very much! Yeah, I agree political will seems like a big issue. But I also hear people saying that they don't know what to push for, so I wanted to try to offer a concrete example of a system that wasn't as destructive to any constituency's interests as e.g. a total pause.

Thanks very much for your feedback, though I confess I'm not entirely sure where to go with it. My interpretation is you have basically two concerns:

1. This policy doesn't really directly regulate algorithmic progress, e.g. if it happened on smaller amounts of compute.
2. Algorithmic theft/leakage is easy.

The first one is true, as I alluded in the problems section. Part of my perspective here is coming from a place of skepticism about regulatory competence - I basically believe we can get regulators to control total compute usage, and to evaluate specific models according to pre-established evals, but I'm not sure I'd trust them to be able to determine "this is a new algorithmic advance, we need to evaluate it". To the extent you had less libertarian priors you could try to use something like the above scheme for algorithms as well, but I wouldn't expect it to work so well, as you lack the cardinal structure of compute size.

In terms of theft/leakage, you're right this plan doesn't discuss it much, and I agree it's worth working on. 

You probably should have said 'yes' when asked if it was AI-written.

This started happening in Hawaii, and to a lesser extent in Arizona. The resolution, apart from reducing net metering subsidies, has been to increased the fixed component of the bill (which pays for the grid connection) and reduce the variable component. My impression is this has been a reasonably effective solution, assuming people don't want to cut their connection entirely.

I agree with you that basically anything in the stock market has much less counterparty risk than that. I disagree with basically all non-trading examples you give. 

It's not just the stock market, it's true for the bond market, the derivatives market, the commodities market... financial markets, a category which includes prediction markets, cannot function effectively with counterparty risk anything like 5%. 

My sense is around 1/20 Ubers don't show up, or if they show up, fail to do their job in some pretty obvious and clear way. 

If the Uber doesn't show up I'm not sure that's counterparty risk: you haven't paid anything, so it seems more like them declining the contract. The equivalent for a prediction market would be if you hit 'buy' and the button didn't work, not for when you have paid the money and then don't get the result taken from you. That's much less bad than if the trade went through and then was settled incorrectly.

I think that's false, at least the statistics on wage theft seemed quite substantial to me. I am kind of confused how to interpret these, but various different studies on Wikipedia suggest wage theft on-average to be around 5%-15% (higher among lower-income workers).

I think those studies have significant methodological flaws, though unfortunately I can't remember the specific issues off the top off my head, so this may not be very convincing to you.

I agree this is true for gas and water (and mostly true for electricity, though PG&E is terrible and Berkeley really has a lot of outages). 

According to the first google hit, PG&E said the average customer suffered 255.9 minutes of outage in 2013, which is a lot higher than I expected, but is still only 100*255.9/(60*24*365) = 0.05%

In most domains except the most hardened part of the stock market counterparty risk is generally >5%.

This seems quite wrong to me:

• High Yield Corporate Bond OAS spreads are <5% according to bloomberg, and most of that is economic risk, not "you will get screwed by a change of rules" risk.
• Trades on US stock exchanges almost always succeed, many more 9s than just one.
• If I buy a product in a box in a supermarket the contents of the box match the label >>95% of the time.
• Banks make errors with depositor balances <<5% of the time.
• Most employers manage to pay fortnightly wages on time without missing one or more paycheques per year.
• Once you're seated in an Uber or Taxi they take you to your destination almost all the time.
• Your utility company fulfills its obligations to supply your house >>95% of the time under all but the most extreme circumstances. 
• Most employees turn up >95% of non-holiday days, and most students maintain >95% attendance.

A bit dated but have you read Robin's 2007 paper on the subject?

Prediction markets are low volume speculative markets whose prices offer informative forecasts on particular policy topics. Observers worry that traders may attempt to mislead decision makers by manipulating prices. We adapt a Kyle-style market microstructure model to this case, adding a manipulator with an additional quadratic preference regarding the price. In this model, when other traders are uncertain about the manipulator’s target price, the mean target price has no effect on prices, and increases in the variance of the target price can increase average price accuracy, by increasing the returns to informed trading and thereby incentives for traders to become informed.

Yes, sorry for being unclear. I meant to suggest that this argument implied 'accelerate agents and decelerate planners' could be the desirable piece of differential progress.