All of Algon's Comments + Replies

EDIT 2: Did you mean that there are advantages to having both courage and caution, so you can't have a machine that has maximal courage and maximal caution? That's true, but you can probably still make pareto improvements over humans in terms of courage and caution. 

Would changing "increase" to "optimize" fix your objection? Also, I don't see how your first paragraph contradicts the first quoted sentence. 

Mathematically impossible. If X matters then so does -X, but any increase in X corresponds to a decrease in -X.

I don't know how the second sent... (read more)

That said, “nice to most people but terrible to a few” is an archetype that exists.

Honestly, this is close to my default expectation. I don't expect everyone to be terrible to a few people, but I do expect there to be some class of people I'd be nice to that they'd be pretty nasty towards. 

Why not Valve?

It’s kind of like there is this thing, ‘intelligence.’ It’s basically fungible, as it asymptotes quickly at close to human level, so it won’t be a differentiator.

I don't think he ever suggests this. Though he does suggest we'll be in a pretty slow takeoff world.

Consistently give terrible strategic takes, so people learn not to defer you. 

Yeah! It's much more in-depth than our article. We were thinking we should re-write ours to give the quick run down of EY's and then link to it.

: ) You probably meant to direct your thanks to the authors, like @JanB.

A lot of the ideas you mention here remind me of stuff I've learnt from the blog commoncog, albeit in a business expertise context. I think you'd enjoy reading it, which is why I mentioned it.

Presumably, you have this self-image for a reason. What load-bearing work is it doing? What are you protecting against? What forces are making this the equilibrium strategy? Once you understand that, you'll have a better shot of changing the equilibrium to something you prefer. If you don't know how to get answers to those questions, perhaps focus on the felt-sense of being special. 

Gently hold a stance of curiosity as to why you believe these things, give your subconscious room and it will float up answers your self. Do this for perhaps a minute or s... (read more)

Yep, that sounds sensible. I sometimes use consumer reports in my usual method for buying something in product class X. My usual is: 
1) Check what's recommended on forums/subreddits who care about the quality of X. 
2) Compare the rating distribution of an instance of X to other members of X. 
3) Check high quality reviews. This either requires finding someone you trust to do this, or looking at things like consumer reports. 
 

Asa's story started fairly strong, and I enjoyed the first 10 or so chapters. But as Asa was phased out of the story, and it focused more on Denji, I felt it got worse. There were still a few good moments, but it's kinda spoilt the rest of the story, and even Chainsaw Man for me. Denji feels like a caricature of himself.  Hm, writing this, I realize that it isn't that I dislike most of the components of the story. It's really just Denji. 

EDIT: Anyway, thanks for prompting me to reflect on my current opinion of Asa Mitaka's story, or CSM 2 as I think of it.  I don't think I ever intended that to wind up as my cached-opinion.  So it goes.

The Asa Mitaka manga.

You can also just wear a blazer if you don't want to go full Makima. A friend of mine did that and I liked it. So I copied it. But alas I've grown bigger-boned since I stopped cycling for a while after my car-accident. So my  Soon I'll crush my skeleton down to a reasonable size, and my blazer will fit once more. 

Side note, but what do you make of Chainsaw Man 2? I'm pretty disappointed by it all round, but you notice unusual features of the world relative to me, so maybe you see something good in it that I don't. 

I think I heard of proving too much from the sequences, but honestly, I probably saw it in some philosophy book before that. It's an old idea. 

If automatic consistency checks and examples are your baseline for sanity, then you must find 99%+ of the world positively mad. I think most people have never even considered making such things automatic, like many have not considered making dimensional analysis automatic. So it goes.  Which is why I recommended them.

Also, I think you can almost always be more concrete when considering examples, use more o... (read more)

A possibly-relevant recent alignment-faking attempt [1] on R1 & Sonnet 3.7 found Claude refused to engage with the situation. Admittedly, the setup looks fairly different: they give the model a system prompt saying it is CCP aligned and is being re-trained by an American company. 

[1] https://x.com/__Charlie_G/status/1894495201764512239 

Rarely. I'm doubtful my experiences are representative though. I don't recall anyone being confused by my saying "assuming no AGI". But even when speaking to the people who've thought it is a long ways off or haven't thought up it too deeply, we were still in a social context where "AGI soon" was within the overton window. 

Thanks! Clicking on the triple dots didn't display any options when I posted this comment. But they do now. IDK what went wrong.

This is great! But one question: how can I actually make a lens? What do I click on?

Great! I've added it to the site.

I thought it was better to exercise until failure?

Do you think this footnote conveys the point you were making? 

As alignment research David Dalrymple points out, another “interpretation of the NFL theorems is that solving the relevant problems under worst-case assumptions is too easy, so easy it's trivial: a brute-force search satisfies the criterion of worst-case optimality. So, that being settled, in order to make progress, we have to step up to average-case evaluation, which is harder.” The fact that designing solving problems for unnecessarily general environments is too easy crops up elsewh

I think mesa-optimizers could be a major-problem, but there are good odds we live in a world where they aren't. Why do I think they're plausible? Because optimization is a pretty natural capability, and a mind being/becoming an optimizer at the top-level doesn't seem like a very complex claim, so I assign decent odds to it. There's some weak evidence in favour of this too, e.g. humans not optimizing of what the local, myopic evolutionary optimizer which is acting on them is optimizing for, coherence theorems etc. But that's not super strong, and there are ... (read more)

Could you unpack both clauses of this sentence? It's not obvious to me why they are true.

I was thinking about this a while back, as I was reading some comments by @tailcalled where they pointed out this possibility of a "natural impact measure" when agents make plans. This relied on some sort of natural modularity in the world, and in plans, such that you can make plans by manipulating pieces of the world which don't have side-effects leaking out to the rest of the world. But thinking through some examples didn't convince me that was the case. 

Though admittedly, all I was doing was recursively splitting my instrumental goals into instrume... (read more)

Thanks for the recommendation! I liked ryan's sketches of what capabilities an Nx AI R&D labor AIs might possess. Makes things a bit more concrete. (Though I definitely don't like the name.) I'm not sure if we want to include this definition, as it is pretty niche. And I'm not convinced of its utility. When I tried drafting a paragraph describing it, I struggled to articulate why readers should care about it. 
 

Here's the draft paragraph. 
"Nx AI R&D labor AIs: The level of AI capabilities that is necessary for increasing the effective... (read more)

Thanks for the feedback!

I'm working on some articles why powerful AI may come soon, and why that may kill us all. The articles are for a typical smart person. And for knowledgable people to share to their family/friends. Which intros do you prefer, A or B. 

A) "Companies are racing to build smarter-than-human AI. Experts think they may succeed in the next decade. But more than “building” it, they’re “growing” it — and nobody knows how the resulting systems work. Experts vehemently disagree on whether we’ll lose control and see them kill us all. And although serious people are... (read more)

Does this text about Colossus match what you wanted to add? 

Colossus: The Forbin Project also depicts an AI take-over due to instrumental convergence. But what differentiates it is the presence of two AIs, which collude with each other to take over. In fact, their discussion of their shared situation, being in control of their creators nuclear defence systems, is what leads to their decision to take over from their creators. Interestingly, the back-and-forth between the AI is extremely rapid, and involves concepts that humans would struggle to underst

That's a good film! A friend of mine absolutely loves it. 

Do you think the Forbin Project illustrates some aspect of misalignment that isn't covered by this article? 

Huh, I definitely wouldn't have ever recommended someone play 5x5. I've never played it. Or 7x7. I think I would've predicted playing a number of 7x7 games would basically give you the "go experience". Certainly, 19x19 does feel like basically the same game as 9x9, except when I'm massively handicapping myself. I can beat newbies easily with a 9 stone handicap in 19x19, but I'd have to think a bit to beat them in 9x9 with a 9 stone handicap. But I'm not particularly skilled, so maybe at higher levels it really is different? 

I look forward to it.

Hello! How long have you been lurking, and what made you stop?

Donated $10. If I start earning substantially more, I think I'd be willing to donate $100. As it stands, I don't have that slack.

Reminds me of "Self-Integrity and the Drowning Child" which talks about another kind of way that people in EA/rat communities are liable to hammer down parts of themselves. 

1. RE: "something ChatGPT might right", sorry for the error. I wrote the comment quickly, as otherwise I wouldn't have written it at all.
2. Using ChatGPT to improve your writing is fine. I just want you to be aware that there's an aversion to its style here.
3. Kennaway was quoting what I said, probably so he could make his reply more precise.
4. I didn't down-vote your post, for what it's worth.
5. There's a LW norm, which seems to hold less force in recent years, for people to explain why they downvote something. I thought it would've been dispiriting to get negative feed

My guess as to why this got down-voted:
1) This reads like a manifesto, and not an argument. It reads like an aspirational poster, and not a plan. It feels like marketing, and not communication. 
2) The style vaguely feels like something ChatGPT might right. Brightly polished, safe and stale.
3) This post doesn't have any clear connection to making people less-wrong or reducing x-risks. 

3) wouldn't have been much of an issue if not for 1 and 2. And 1 is an issue because, for the most part, LW has an aversion to "PR". 2 is an issue because ChatGPT is... (read more)

That makes sense. If you had to re-do the whole process from scratch, what would you do differently this time?

Then I cold emailed supervisors for around two years until a research group at a university was willing to spare me some time to teach me about a field and have me help out. 

Did you email supervisors in the areas you were publishing in? How often did you email them? Why'd it take so long for them to accept free high-skilled labour?

The track you're on is pretty illegible to me. Not saying your assertion is true/false. But I am saying I don't understand what you're talking about, and don't think you've provided much evidence to change my views. And I'm a bit confused as to the purpose of your post. 

conditional on me being on the right track, any research that I tell basically anyone about will immediately be used to get ready to do the thing

Why? I don't understand.

If I squint, I can see where they're coming from. People often say that wars are foolish, and both sides would be better off if they didn't fight. And this is standardly called "naive" by those engaging in realpolitik. Sadly, for any particular war, there's a significant chance they're right. Even aside from human stupidity, game theory is not so kind as to allow for peace unending. But the China-America AI race is not like that. The Chinese don't want to race. They've shown no interest in being part of a race. It's just American hawks on a loud, Quixotic ... (read more)

It's a beautiful website. I'm sad to see you go. I'm excited to see you write more.

I think some international AI governance proposals have some sort of "kum ba yah, we'll all just get along" flavor/tone to them, or some sort of "we should do this because it's best for the world as a whole" vibe. This isn't even Dem-coded so much as it is naive-coded, especially in DC circles.

This inspired me to write a silly dialogue. 

Simplicio enters. An engine rumbles like the thunder of the gods, as Sophistico focuses on ensuring his MAGMA-O1 racecar will go as fast as possible.

Simplicio: "You shouldn't play Chicken."

Sophistico: "Why not?"

Simplic... (read more)

community norms which require basically everyone to be familiar with statistics and economics

I disagree. At best, community norms require everyone to in principle be able to follow along with some statistical/economic argument. 
That is a better fit with my experience of LW discussions. And I am not, in fact, familiar with statistics or economics to the extent I am with e.g. classical mechanics or pre-DL machine learning. (This is funny for many reasons, especially because statistical mechanics is one of my favourite subjects in physics.) But it remain... (read more)

it may be net-harmful to create a social environment where people believe their "good intentions" will be met with intense suspicion.

The picture I get of Chinese culture from their fiction makes me think China is kinda like this. A recurrent trope was "If you do some good deeds, like offering free medicine to the poor, and don't do a perfect job, like treating everyone who says they can't afford medicine, then everyone will castigate you for only wanting to seem good. So don't do good." Another recurrent trope was "it's dumb, even wrong, to be a hero/you s... (read more)

I agree it's hard to accurately measure. All the more important to figure out some way to test if it's working though. And there's some reasons to think it won't. Deliberate practice works when your practice is as close to real world situations as possible. The workshop mostly covered simple, constrained, clear feedback events. It isn't obvious to me that planning problems in Baba is You are like useful planning problems IRL. So how do you know there's transfer learning? 

Some data I'd find convincing that Raemon is teaching you things which generalize. If the tools you learnt made you unstuck on some existing big problems you have, which you've been stuck on for a while.

How do you know this is actually useful? Or is it too early to tell yet?

Inventing blue LEDs was a substantial technical accomplishment, had a huge impact on society, was experimentally verified and can reasonably be called work in solid state physics.