Maybe I'm missing something important, but I think AGI won't be much like a resource and also I don't think we'll see rentrier entities. I'm not saying it will be better though.

The key thing about oil or coal is that it's already there, you roughly know how much it's worth and this value won't change much whatever you do (or don't do). With AI this is different, because all the time you'll have many competitors trying to create a new AI that is either stronger or cheaper than yours. It's not that the deeper you dig the more & better oil you get.

So you can't really become a rentier - you must spend all resources on charging forward, because if you don't, you'll be left behind forever. If we assume there's no single entity that takes lead and restricts competition forever, this might lead to some version of ascended economy. That's probably even worse for an average human: AGI rentrier won't care about you, but also won't care much about your small field full of potatoes that will hopefully let you survive one winter more.

Oh yeah. How do I know I'm angry? My back is stiff and starts to hurt.

The second reason that I don't trust the neighbor method is that people just... aren't good at knowing who a majority of their neighbors are voting for. In many cases it's obvious (if over 70% of your neighbors support one candidate or the other, you'll probably know). But if it's 55-45, you probably don't know which direction it's 55-45 in.

My guess is that there's some postprocessing here. E.g. if you assume that the "neighbor" estimate is wrong but without the refusal problem, and you have the same data from the previous election, then you could estimate the shift of opinions and apply that to other pools that ask about your vote. Or you could ask some additional question like "who did your neighbours vote for in the previous election" and compare that to the real data (ideally per county or so). I would be very surprised if they based the bets just on the raw results.

My initial answer to your starting question was "I disagree with this statement because they likely used 20 different ways of calculating the p-value and selected the one that was statistically significant". Also https://en.m.wikipedia.org/wiki/Replication_crisis

I don't think there's a lot of value in distinguishing 3000 and 1,000,000 and probably for any aggregate you'll want to show this will just be "later than 2200" or something like that. But yes this way they can't make a statement that this will be 1,000,000 which is some downside.

I'm not a big fan of looking at the neighbors to decide whether this is a missing answer or high estimate (it's OK to not want to answer this one question). So some N/A or -1 should be ok.

(Just to make it clear, I'm not saying this is an important problem)

I think you can also ask them to put some arbitrarily large number (3000 or 10000 or so) and then just filter out all the numbers above some threshold.

By what year do you think the Singularity will occur? Answer such that you think, conditional on the Singularity occurring, there is an even chance of the Singularity falling before or after this year. If you think a singularity is so unlikely you don't even want to condition on it, leave this question blank

You won't be able to distinguish people who think singularity is super unlikely from people who just didn't want to answer this question for whatever reason (maybe they forgot or thought it's boring or whatever).

Anyway, in such a world some people would probably evolve music that is much more interesting to the public

I wouldn't be so sure.

I think the current diversity of music is largely caused by artists' different lived experiences. You feel something, this is important for you, you try to express that via music. As long as AIs don't have anything like "unique experiences" on the scale of humans, I'm not sure if they'll be able to create music that is that diverse (and thus interesting).

I assume the scenario you described, not a personal AI trained on all your life. With that, it could work.

(Note that I mostly think about small bands, not popular-music-optimised-for-wide-publicity).

Are there other arguments for active skepticism about Multipolar value fragility? I don’t have a ton of great stories

The story looks for me roughly like this:

1. The agents won't have random values - they will be somehow scattered around the current status quo.
2. Therefore the future we'll end up in should not be that far from the status quo.
3. Current world is pretty good.

(I'm not saying I'm sure this is correct, it's just roughly how I think about that)

This is interesting! Although I think it's pretty hard to use that in a benchmark (because you need a set of problems assigned to clearly defined types and I'm not aware of any such dataset).

There are some papers on "do models know what they know", e.g. https://arxiv.org/abs/2401.13275 or https://arxiv.org/pdf/2401.17882.