Is there any information on how long the LLM spent on taking the tests? Any idea? I'd like to know the comparison with human times. (I realize it can depend on hardware, etc but would just like some general idea.)

Someone like Paul Graham or Tyler Cowen is noticing more smarter kids, because we now have much better systems for putting the smarter kids into contact with people like Paul Graham and Tyler Cowen.

I'd guess very smart kids are getting more numerous and smarter at the elite level since I'd guess just about everything is improving at the most competitive level. Unfortunately it doesn't seem like there's much interest in measuring this, e.g. hundreds of kids tie for the maximum score possible on SATs (1600) instead of designing a test that won't max out. 

(Btw, one cool thing I learned about recently is that some tests use dynamic scoring where if you get questions correct the system asks you harder questions.)

Governments are not social welfare maximizers

Most people making up governments, and society in general, care at least somewhat about social welfare.  This is why we get to have nice things and not descend into chaos.

Elected governments have the most moral authority to take actions that effect everyone, ideally a diverse group of nations as mentioned in Daniel Kokotajlo's maximal proposal comment.

I'm having difficulty following the code for the urn scenario. Can it be something like?

def P():
    # Initialize the world with random balls (or whatever)
    num_balls = 1000
    urn = [random.choice(["red", "white"]) for i in range(num_balls)]

    # Run the world
    history = []
    total_loss = 0
    for i in range(len(urn)):
        ball = urn[i]
        probability_of_red = S(history)
        if probability_of_red == 1 and ball != 'red' or probability_of_red == 0 and ball == 'red':
            print("You were 100% sure of a wrong prediction. You lose for all eternity.")
            return  # avoid crashing in math.log()
        if ball == 'red':
            loss = math.log(probability_of_red)
        else:
            loss = math.log(1 - probability_of_red)
        total_loss += loss
        history.append(ball)
        print(f"{ball:6}\tPrediction={probability_of_red:0.3f}\tAverage log loss={total_loss / (i + 1):0.3f}")
 

If we define S() as:

def S(history):
    if not history:
        return 0.5
    reds = history.count('red')
    prediction = reds / float(len(history))

    # Should never be 100% confident
    if prediction == 1:
        prediction = 0.999
    if prediction == 0:
        prediction = 0.001

    return prediction

The output will converge on Prediction = 0.5 and Average log loss as log(0.5). Is that right?