Popular Comments
Recent Discussion
This is a linkpost: link to post.
Originally this was part of a much larger work. However, I realized that I don't think I've seen the specific argument around GPU salvage spelled out. Since I'm also using a new writing format, I figured I could get feedback on both the content and format at the same time.
That said, despite the plausible novelty of one of the arguments I don't think it will be especially interesting to LW, since it is based on oddly specific assumptions: this makes more sense in the context of a broad AI risk argument. It also feels kind of obvious?
The format is the interesting bit. For motivation, sometimes people have opposing reactions to AI risk arguments:
- "Your argument is nice and concise, but it doesn't
The post setup skips the "AIs are loyal to you" bit, but it does seem like this line of thought broadly aligns with the post.
I do think this does not require ASI, but I would agree that including it certainly doesn't help.
Summary: AGI isn't super likely to come super soon. People should be working on stuff that saves humanity in worlds where AGI comes in 20 or 50 years, in addition to stuff that saves humanity in worlds where AGI comes in the next 10 years.
Thanks to Alexander Gietelink Oldenziel, Abram Demski, Daniel Kokotajlo, Cleo Nardo, Alex Zhu, and Sam Eisenstat for related conversations.
My views on when AGI comes
AGI
By "AGI" I mean the thing that has very large effects on the world (e.g., it kills everyone) via the same sort of route that humanity has large effects on the world. The route is where you figure out how to figure stuff out, and you figure a lot of stuff out using your figure-outers, and then the stuff you...
In particular, even if the LLM were being continually trained (in a way that's similar to how LLMs are already trained, with similar architecture), it still wouldn't do the thing humans do with quickly picking up new analogies, quickly creating new concepts, and generally reforging concepts.
Is this true? How do you know? (I assume there's some facts here about in-context learning that I just happen to not know.)
It seems like eg I can teach an LLM a new game in one session, and it will operate within the rules of that game.
Every now and then, some AI luminaries
- (1) propose that the future of powerful AI will be reinforcement learning agents—an algorithm class that in many ways has more in common with MuZero (2019) than with LLMs; and
- (2) propose that the technical problem of making these powerful future AIs follow human commands and/or care about human welfare—as opposed to, y’know, the Terminator thing—is a straightforward problem that they already know how to solve, at least in broad outline.
I agree with (1) and strenuously disagree with (2).
The last time I saw something like this, I responded by writing: LeCun’s “A Path Towards Autonomous Machine Intelligence” has an unsolved technical alignment problem.
Well, now we have a second entry in the series, with the new preprint book chapter “Welcome to the Era of Experience” by...
My intuition says reward hacking seems harder to solve than this (even in EEA), but I'm pretty unsure. One example is, under your theory, what prevents reward hacking through forming a group and then just directly maxing out on mutually liking/admiring each other?
When applying these ideas to AI, how do you plan to deal with the potential problem of distributional shifts happening faster than we can edit the reward function?
Come get old-fashioned with us, and let's read the sequences at Lighthaven! We'll show up, mingle, do intros, and then get to beta test an app Lightcone is developing for LessOnline. Please do the reading beforehand - it should be no more than 20 minutes of reading. And BRING YOUR LAPTOP!!! You'll need it for the app.
This group is aimed for people who are new to the sequences and would enjoy a group experience, but also for people who've been around LessWrong and LessWrong meetups for a while and would like a refresher.
This meetup will also have dinner provided! We'll be ordering pizza-of-the-day from Sliver (including 2 vegan pizzas). Please RSVP to this event so we know how many people to have food for.
This week we'll be...
Hear me out, I think the most forbidden technique is very useful and should be used, as long as we avoid the "most forbidden aftertreatment:"
- An AI trained on interpretability techniques must not be trained on capabilities after (or during) it is trained on interpretability techniques, otherwise it will relearn bad behaviour—in a more sneaky way.
- An AI trained on interpretability techniques cannot be trusted any more than old version of itself, which hasn't been trained on interpretability techniques yet. Evaluations must be performed on the old version of itself.
- An AI company which trains its AI on interpretability techniques, must publish the old version (which hasn't been trained on them), with the same availability as the new version.
The natural selection argument:
The reason why the most forbidden technique is forbidden,...
If there turns out not to be an AI crash, you get a 1/(1+7) * $25,000 = $3,125
If there is an AI crash, you transfer $25k to me.
If you believe that AI is going to keep getting more capable, pushing rapid user growth and work automation across sectors, this is near free money. But to be honest, I think there will likely be an AI crash in the next 5 years, and on average expect to profit well from this one-year bet.
If I win, I want to give the $25k to organisers who can act fast to restrict the weakened AI corps in the wake of the crash. So bet me if you're highly confident that you'll win or just want to hedge the community against the...
Remmelt, if you wish, I'm happy to operationalize a bet. I think you're wrong.
American democracy currently operates far below its theoretical ideal. An ideal democracy precisely captures and represents the nuanced collective desires of its constituents, synthesizing diverse individual preferences into coherent, actionable policy.
Today's system offers no direct path for citizens to express individual priorities. Instead, voters select candidates whose platforms only approximately match their views, guess at which governmental level—local, state, or federal—addresses their concerns, and ultimately rely on representatives who often imperfectly or inaccurately reflect voter intentions. As a result, issues affecting geographically dispersed groups—such as civil rights related to race, gender, or sexuality—are frequently overshadowed by localized interests. This distortion produces presidential candidates more closely aligned with each other's socioeconomic profiles than with the median voter.
Traditionally, aggregating individual preferences required simplifying complex desires into binary candidate selections,...
wildly parallel thinking and prototyping. i'd hop on a call.
[Thanks to Steven Byrnes for feedback and the idea for section §3.1. Also thanks to Justis from the LW feedback team.]
Remember this?
Or this?
The images are from WaitButWhy, but the idea was voiced by many prominent alignment people, including Eliezer Yudkowsky and Nick Bostrom. The argument is that the difference in brain architecture between the dumbest and smartest human is so small that the step from subhuman to superhuman AI should go extremely quickly. This idea was very pervasive at the time. It's also wrong. I don't think most people on LessWrong have a good model of why it's wrong, and I think because of this, they don't have a good model of AI timelines going forward.
1. Why Village Idiot to Einstein is a Long Road: The Two-Component
...Remember that we have no a priori reason to suspect that there are jumps in the future; humans perform sequential reasoning differently, so comparisons to the brain are just not informative.
In what way do we do it differently than the reasoning models?
If it’s worth saying, but not worth its own post, here's a place to put it.
If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.
If you're new to the community, you can start reading the Highlights from the Sequences, a collection of posts about the core ideas of LessWrong.
If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the Concepts section.
The Open Thread tag is here. The Open Thread sequence is here.
Ah but you don't even need to name selection pressures to make interesting progress. As long as you know some kinds of characteristics powerful AI agents might have: eg goals, self models... then we can start to ask--what goals/self models will the most surviving AGIs have?
and you can make progress on both, agnostic of environment. but then, once you enumerate possible goals/self models, then we can start to think about which selection pressures might influence those characteristics in good directions and which levers we can pull today to shape those pressures.
