All of Tomás B.'s Comments + Replies

I was not wrong? I argued with Connor about this back in the day on EAI. If you take the PISA scores at face value and account for their higher population, China has like 30x more people above 145 IQ. Steve Hsu has a nice post about this^[1]: 

This appears to be very antimemetic but seems to be true and the world sure is behaving like this is true. 

If human capital matters, they have more and better human capital than America. But human capital will be obsolete soon. 

And I do think much of the idea that China cannot catch up is based on some i... (read more)

Update: Claude 3.6 is clearly capable of writing jokes. Even if I tell it to write jokes a maximally-alien creature would write if they lived in a maximally alien environment it now seems able to reliably produce strings I qualify as jokes.

interesting reading this 3 years later. I occasionally paste a bug report directly into cursor and provided I am right about which file the bug is in, it often one-shots them. i remain confused about why rsi isn't critical by now.

one data-point: i have been generating songs with lyrics I like and it’s most of my music consumption now.

the song you generated has a slop vibe i am not a fan of - but we are all wireheaded in different ways. however, if I generate hundreds of songs I usually get what I want. focusing on simple lyrics helps a lot and “no autotune” in the prompt helps too.

I was really sleep deprived and slightly intoxicated yesterday and wrote this. It was amusing to me, at least, in the state I was in.

@gwern

I would like to take this opportunity to express my undying loyalty to my nation and its human instruments. 

This is all downstream of the board being insufficiently Machiavellian, and Ilya’s weaknesses in particular. Peter Thiel style champerty against Altman as new EA cause?

I officially lost the bet and paid up. Amusingly, SWE-Bench is so broken it was likely impossible for me to win. Though I would have lost in any case.

If you make a bet about a benchmark, probably you should understand it deeply and not just bet on vibes, ha!

The real crux for these arguments is the assumption that law and property rights are patterns that will persist after the invention of superintelligence. I think this is a shaky assumption. Rights are not ontologically real. Obviously you know this. But I think they are less real, even in your own experience, than you think they are. Rights are regularly "boiled-froged" into an unrecognizable state in the course of a human lifetime, even in the most free countries. Rights are and always have been those privileges the political economy is willing to give yo... (read more)

One thing to note about RSI, we know mindless processes like gradient descent and evolution can improve performance of a model/organism enormously despite their stupidity. And so it's not clear to me that the RSI loop has to be very smart or reliable to start making fast progress. We are approaching a point where the crystallized intelligence and programming and mathematics ability of existing models strike me as being very close to being in extremely dangerous territory. And though reliability probably needs to improve before doom - perhaps not as much as one would think. 

Yeah, I expect it to fall soon but I will lose my bet if it doesn’t happen in a month.

Macroscopic self-replicators are extremely powerful, and provide much of the power of nanotech without relying on nanotech. Seems like they might be worth mentioning more often as a rhetorical tool against those who dismiss anyone who mentions nanotechnology. 

Not looking good for my prediction: https://www.swebench.com/

Curious for an update now that we have slight-better modals. In my brain-dead webdev use-cases, Claude 3.5 has passed some threshold of usability. 

The most important thing this article did was make legible Gerard's history on Uncyclopedia - which one of his allies will inevitably use to destroy him. 

I think about anticipated future experiences. All future slices of me have the same claim to myself. 

Significant evidence for data contamination of MATH benchmark: https://arxiv.org/abs/2402.19450

https://x.com/OriolVinyalsML/status/1791521517211107515

Rumours are GPT-5 has been finished awhile. 

The trans IQ connection is entirely explained by woman’s clothing being less itchy.

I was in the middle of writing a frustrated reply to Matthew's comment when I realized he isn't making very strong claims. I don't think he's claiming your scenario is not possible. Just that not all power seeking is socially destructive, and this is true just because most power seeking is only partially effective. Presumably he agrees that in the limit of perfect power acquisition most power seeking would indeed be socially destructive. 

Every toaster a Mozart, every microwave a Newton, every waffle iron a Picasso.

Was this all done through Suno? You guys are much better at prompting it than I am.

The bet is with a friend and I will let him judge.

I agree that providing an api to God is a completely mad strategy and we should probably expect less legibility going forward. Still, we have no shortage of ridiculously smart people acting completely mad.

This seems to be as good of a place as any to post my unjustified predictions on this topic, the second of which I have a bet outstanding on at even odds.

1. Devin will turn out to be just a bunch of GPT-3.5/4 calls and a pile of prompts/heuristics/scaffolding so disgusting and unprincipled only a team of geniuses could have created it.
2. Someone will create an agent that gets 80%+ on SWE-Bench within six months. 

I am not sure if 1. being true or false is good news. Both suggest we should update towards large jumps in coding ability very soon.

Regarding RSI, ... (read more)

After spending several hours trying to get Gemini, GPT-4 and Claude 3 to make original jokes - I now think I may be wrong about this. Still could be RLHF, but it does seem like an intelligence issue. @janus are the base models capable of making original jokes?

Looks to me he's training on the test set tbh. His ambition to get an IQ of 195 is admirable though. 

I very much doubt this will work. I am also annoyed you don't share your methods. If you can provide me with a procedure that raises my IQ by 20 points in a manner that convinces me this is a real increase in g, I will give you one hundred thousand dollars.

Hsu on China's huge human capital advantage:

Returning to Summers' calculation, and boldly extrapolating the normal distribution to the far tail (not necessarily reliable, but let's follow Larry a bit further), the fraction of NE Asians at +4SD (relative to the OECD avg) is about 1 in 4k, whereas the fraction of Europeans at +4SD is 1 in 33k. So the relative representation is about 8 to 1. (This assumed the same SD=90 for both populations. The Finnish numbers might be similar, although it depends crucially on whether you use the smaller SD=80.) Are these re

I would like to register a prediction. I believe a GPT-4-level model that has been RLHFd for humour will be super-human or near superhuman at humour.  At least in the 99th percentile of professional comedians. My intuition is humour is much easier than people think, and current models fail at it mostly because the forms of RLHF existing models use pushed them into humourlessness . 

Conditional on this being true, he must be very certain we are close to median human performance, like on the order of one to three years.  I don't think this amount of capital can be efficiently expended in the chips industry unless human capital is far less important than it once was. And it will not be profitable, basically, unless he thinks Winning is on the table in the very near term.

I feel 5 trillion must be a misprint. This is like several years worth of American tax revenues. Conditional on this being true I would take this as significant evidence that what they have internally is unbelievably good. Perhaps even an AI with super-persuasion!

It is such a ridiculous figure, I suspect it must be off by at least an OOM. 

My take on self-driving taking forever is driving is near AGI complete. Humans drive roughly a million miles between fatal accidents; it would not be particularly surprising if in these million miles (where you are interacting with intelligent agents) you inevitably encounter near AGI-complete problems. Indeed, as the surviving self-driving companies are all moving to end-to-end approaches, self-driving research is begining to resemble AGI research more and more.

Any evidence for it working? Seriously doubt.

Nope. Sadly. And if there were, your intellect would not be impressive for such tools would reach fixation.

If it’s any consolation, all the brilliant people able to make many multiples of your salary due to being born with a better brain - while almost to a man being incredibly smug about it - will soon be losing intellectual death matches with toaster ovens.

https://youtu.be/OGxdgNJ_lZM

And OpenAI has explicitly said this is what they want to do! Their Superalignment strat looks suspiciously like "gunning for RSI".

It does seem to me a little silly to give competitors API access to your brain. If one has enough of a lead, one can just capture your competitors markets. 

I think I may be almost crazy enough to volunteer for such a procedure, ha, should you convince me. 

This comment was assuming causal variants are known, which I admit is a big gimme. More of a first-principles type eye-balling.   

So I made this comment awhile back, though I admit being ignorant on how good modern somatic gene therapy is:

I think somatic gene therapy, while technically possible in principal, is extremely unpromising for intelligence augmentation. Creating a super-genius is almost trivial with germ-line engineering. Provided we know enough causal variants, one needs to only make a low-hundreds number of edits to one cell to make someone smarter than any human that has ever lived. With somatic gene therapy you would almost certainly have to alter billions of cells to get anywhere.

Am I just wrong here? Is somatic gene therapy really robust and error-free enough to safely edit billions of cells? 

This is a very good, and very scary point - another thing that could provide, at least the appearance of, a discontinuity. One symptom of this this scenario would be a widespread, false belief that "open source" models are SOTA.

Might be good to brainstorm other symptoms to prime ourselves to recognize when we are in this scenario. Complete hiring-freezes/massive layoffs at the firms in question, aggressive expansion into previously-unrelated markets, etc. 

One argument I've had for self-driving being hard is: humans drive many millions of miles before they get in fatal accidents. In this long tail, would it be that surprising if there were AGI complete problems within it? My understanding is Waymo and Cruise both use teleoperation in these cases. And one could imagine automating this, a God advising the ant in your analogy. But still, at that point you're just doing AGI research.