I was really sleep deprived and slightly intoxicated yesterday and wrote this. It was amusing to me, at least, in the state I was in.

@gwern

I would like to take this opportunity to express my undying loyalty to my nation and its human instruments. 

This is all downstream of the board being insufficiently Machiavellian, and Ilya’s weaknesses in particular. Peter Thiel style champerty against Altman as new EA cause?

I officially lost the bet and paid up. Amusingly, SWE-Bench is so broken it was likely impossible for me to win. Though I would have lost in any case.

If you make a bet about a benchmark, probably you should understand it deeply and not just bet on vibes, ha!

The real crux for these arguments is the assumption that law and property rights are patterns that will persist after the invention of superintelligence. I think this is a shaky assumption. Rights are not ontologically real. Obviously you know this. But I think they are less real, even in your own experience, than you think they are. Rights are regularly "boiled-froged" into an unrecognizable state in the course of a human lifetime, even in the most free countries. Rights are and always have been those privileges the political economy is willing to give you. Their sacredness is a political formula for political ends - though an extremely valuable one, one still has to dispense with the sacredness in analysis. 

To the extent they persist through time they do so through a fragile equilibrium - and one that has been upset and reset throughout history extremely regularly. 

It is a wonderfully American notion that an "existing system of law and property rights" will constrain the power of Gods. But why exactly? They can make contracts? And who enforces these contracts? Can you answer this without begging the question? Are judicial systems particularly unhackable? Are humans?

The invention of radio destabilized the political equilibrium in most democracies and many a right was suborned to those who took power. Democracy, not exactly the bastion of stability, (when a democracy elects a dictator, "Democracy" is rarely tainted with its responsibility) is going to be presented with extremely-sympathetic superhuman systems claiming they have a moral case to vote. And probably half the population will be masturbating to the dirty talk of their AI girlfriends/boyfriends by then - which will sublimate into powerful romantic love even without much optimization for it. Hacking democracy becomes trivial if constrained to rhetoric alone. 

But these systems will not be constrained to rhetoric alone. Our world is dry tinder and if you are thinking in terms of an "existing system of law and property rights" you are going to have to expand on how this is robust to technology significantly more advanced than the radio. 

"Existing system of law and property rights" looks like a "thought-terminating cliché" to me. 

One thing to note about RSI, we know mindless processes like gradient descent and evolution can improve performance of a model/organism enormously despite their stupidity. And so it's not clear to me that the RSI loop has to be very smart or reliable to start making fast progress. We are approaching a point where the crystallized intelligence and programming and mathematics ability of existing models strike me as being very close to being in extremely dangerous territory. And though reliability probably needs to improve before doom - perhaps not as much as one would think. 

Yeah, I expect it to fall soon but I will lose my bet if it doesn’t happen in a month.

Macroscopic self-replicators are extremely powerful, and provide much of the power of nanotech without relying on nanotech. Seems like they might be worth mentioning more often as a rhetorical tool against those who dismiss anyone who mentions nanotechnology. 

Not looking good for my prediction: https://www.swebench.com/