All of lc's Comments + Replies

Wow, I didn't realize.

The two guys from Epoch on the recent Dwarkesh Patel podcast repeatedly made the argument that we shouldn't fear AI catastrophe, because even if our successor AIs wanted to pave our cities with datacenters, they would negotiate a treaty with us instead of killing us. It's a ridiculous argument for many reasons but one of them is that they use abstract game theoretic and economic terms to hide nasty implementation details

Formatting is off for most of this post.

"Treaties" and "settlements" between two parties can be arbitrarily bad. "We'll kill you quickly and painlessly" is a settlement.

The doubling down is delusional but I think you're simplifying the failure of projection a bit. The inability of markets and forecasters to predict Trump's second term is quite interesting. A lot of different models of politics failed.

The indexes above seem to be concerned only with state restrictions on speech. But even if they weren't, I would be surprised if the private situation was any better in the UK than it is here.

Very hard to take an index about "freedom of expression" seriously when the United States, a country with constitutional guarantees of freedom of speech and freedom of the press, is ranked lower than the United Kingdom, which prosecutes hundreds of people for political speech every year.

The future is hard to predict, especially under constrained cognition.

AGI is still 30 years away - but also, we're going to fully automate the economy, TAM 80 trillion

I find it a little suspicious that the recent OpenAI model releases became leaders on the MASK dataset (when previous iterations didn't seem to trend better at all), but I'm hopeful this represents deeper alignment successes and not simple tuning on the same or a similar set of tasks.

Works now for me

If anybody here knows someone from CAIS they need to setup their non-www domain name. Going to https://safe.ai shows a github landing page

Interesting, but non-sequitur. That is, either you believe that interest rates will predictably increase and there's free money on the table, and you should just say so, or not, and this anecdote doesn't seem to be relevant (similarly, I made money buying NVDA around that time, but I don't think that proves anything).

I am saying so! The market is definitely not pricing in AGI; doesn't matter if it comes in 2028, or 2035, or 2040. Though interest rates are a pretty bad way to arb this; I would just buy call options on the Nasdaq.

Perhaps, but shouldn't LLMs

The outside view, insofar is that is a well-defined thing...

It's not really a well-defined thing, which is why the standard on this site is to taboo those words and just explain what your lines of evidence are, or the motivation for any special priors if you have them.

If AGI were arriving in 2030, the outside view says interest rates would be very high (I'm not particularly knowledgeable about this and might have the details wrong but see the analysis here, I believe the situation is still similar), and less confidently I think the S&P's value would pr

I don't predict a superintelligent singleton (having fused with the other AIs) would need to design a bioweapon or otherwise explicitly kill everyone. I expect it to simply transition into using more efficient tools than humans, and transfer the existing humans into hyperdomestication programs

+1, this is clearly a lot more likely than the alignment process missing humans entirely IMO

Now, one could reasonably counter-argue that the yin strategy delivers value somewhere else, besides just e.g. "probability of a date". Maybe it's a useful filter for some sort of guy...

I feel like you know this is the case and I'm wondering why you're even asking the question. Of course it's a filter; the entire mating process is. Women like confidence, and taking these mixed signals as a sign of attraction is itself a sign of confidence. Walking up to a guy and asking him to have sex immediately would also be more "efficient" by some deranged standards, but the point of flirting is that you get to signal social grace, selectivity, and a willingness to walk away from the interaction.

Close friend of mine, a regular software engineer, recently threw tens of thousands of dollars - a sizable chunk of his yearly salary - at futures contracts on some absurd theory about the Japanese Yen. Over the last few weeks, he coinflipped his money into half a million dollars. Everyone who knows him was begging him to pull out and use the money to buy a house or something. But of course yesterday he sold his futures contracts and bought into 0DTE Nasdaq options on another theory, and literally lost everything he put in and then some. I'm not sure but I... (read more)

Typically I operationalize "employable as a software engineer" as being capable of completing tasks like:

• "Fix this error we're getting on BetterStack."
• "Move our Redis cache from DigitalOcean to AWS."
• "Add and implement a cancellation feature for ZeroPath scans."
• "Add the results of this evaluation to our internal benchmark."

These are pretty representative examples of the kinds of tasks your median software engineer will be getting and resolving on a day to day basis.

No chatbot or chatbot wrapper can complete tasks like these for an engineering team at... (read more)

They strengthen chip export restrictions, order OpenBrain to further restrict its internet connections, and use extreme measures to secure algorithmic progress, like wiretapping OpenBrain employees—this catches the last remaining Chinese spy

Wiretapping? That's it? Was this spy calling Xi from his home phone? xD

As a newly minted +100 strong upvote, I think the current karma economy accurately reflects how my opinion should be weighted

I have Become Stronger

My strong upvotes are now giving +1 and my regular upvotes give +2.

test

Just edited the post because I think the way it was phrased kind of exaggerated the difficulties we've been having applying the newer models. 3.7 was better, as I mentioned to Daniel, just underwhelming and not as big a leap as either 3.6 or certainly 3.5.

If you plot a line, does it plateau or does it get to professional human level (i.e. reliably doing all the things you are trying to get it to do as well as a professional human would)?

It plateaus before professional human level, both in a macro sense (comparing what ZeroPath can do vs. human pentesters) and in a micro sense (comparing the individual tasks ZeroPath does when it's analyzing code). At least, the errors the models make are not ones I would expect a professional to make; I haven't actually hired a bunch of pentesters and asked them to do the s... (read more)

We use different models for different tasks for cost reasons. The primary workhorse model today is 3.7 sonnet, whose improvement over 3.6 sonnet was smaller than 3.6's improvement over 3.5 sonnet. When taking the job of this workhorse model, o3-mini and the rest of the recent o-series models were strictly worse than 3.6.

I haven't read the METR paper in full, but from the examples given I'm worried the tests might be biased in favor of an agent with no capacity for long term memory, or at least not hitting the thresholds where context limitations become a problem:

For instance, task #3 here is at the limit of current AI capabilities (takes an hour). But it's also something that could plausibly be done with very little context; if the AI just puts all of the example files in its context window it might be able to write the rest of the decoder from scratch. It might not... (read more)

There was a type of guy circa 2021 that basically said that gpt-3 etc. was cool, but we should be cautious about assuming everything was going to change, because the context limitation was a key bottleneck that might never be overcome. That guy's take was briefly "discredited" in subsequent years when LLM companies increased context lengths to 100k, 200k tokens.

I think that was premature. The context limitations (in particular the lack of an equivalent to human long term memory) are the key deficit of current LLMs and we haven't really seen much improvement at all.

If AI executives really are as bullish as they say they are on progress, then why are they willing to raise money anywhere in the ballpark of current valuations?

The story is that they need the capital to build the models that they think will do that.

Moral intuitions are odd. The current government's gutting of the AI safety summit is upsetting, but somehow less upsetting to my hindbrain than its order to drop the corruption charges against a mayor. I guess the AI safety thing is worse in practice but less shocking in terms of abstract conduct violations.

It helps, but this could be solved with increased affection for your children specifically, so I don't think it's the actual motivation for the trait.

The core is probably several things, but note that this bias is also part of a larger package of traits that makes someone less disagreeable. I'm guessing that the same selection effects that made men more disagreeable than women are also probably partly responsible for this gender difference.

I suspect that the psychopath's theory of mind is not "other people are generally nicer than me", but "other people are generally stupid, or too weak to risk fighting with me".

That is true, and it is indeed a bias, but it doesn't change the fact that their assessment of whether others are going to hurt them seems basically well calibrated. The anecdata that needs to be explained is why nice people do not seem to be able to tell when others are going to take advantage of them, but mean people do. The posts' offered reason is that generous impressions of ... (read more)

This post is about a suspected cognitive bias and why I think it came to be. It's not trying to justify any behavior, as far as I can tell, unless you think the sentiment "people are pretty awful" justifies bad behavior in of itself.

The game theory is mostly an extended metaphor rather than a serious model. Humans are complicated.

Elon already has all of the money in the world. I think he and his employs are ideologically driven, and as far as I can tell they're making sensible decisions given their stated goals of reducing unnecessary spend/sprawl. I seriously doubt they're going to use this access to either raid the treasury or turn it into a personal fiefdom. It's possible that in their haste they're introducing security risks, but I also think the tendency of media outlets and their sources will be to exaggerate those security risks. I'd be happy to start a prediction market about this if a regular feels very differently.

If Trump himself was spearheading this effort I would be more worried.

Anthropic has a bug bounty for jailbreaks: https://hackerone.com/constitutional-classifiers?type=team

If you can figure out how to get the model to give detailed answers to a set of certain questions, you get a 10k prize. If you can find a universal jailbreak for all the questions, you get 20k.

Yeah, one possible answer is "don't do anything weird, ever". That is the safe way, on average. No one will bother writing a story about you, because no one would bother reading it.

You laugh, but I really think a group norm of "think for yourself, question the outside world, don't be afraid to be weird" is part of the reason why all of these groups exist. Doing those things is ultimately a luxury for the well-adjusted and intelligent. If you tell people over and over to question social norms some of those people will turn out to be crazy and conclude crime and violence is acceptable.

I don't know if there's anything to do about that, but it is a thing.

So, to be clear, everyone you can think of has been mentioned in previous articles or alerts about Zizians so far? Because I have only been on the periphery of rationalist events for the last several years, but in 2023 I can remember sending this^[1] post about rationalist crazies into the San Antonio LW groupchat. A trans woman named Chase Carter, who doesn't generally attend our meetups, began to argue with me that Ziz (who gets mentioned in the article as an example) was subject to a "disinformation campaign" by rationalists, her goals were actually... (read more)

I know you're not endorsing the quoted claim, but just to make this extra explicit: running terrorist organizations is illegal, so this is the type of thing you would also say if Ziz was leading a terrorist organization, and you didn't want to see her arrested.

Why did 2 killings happen within the span of one week?

According to law enforcement the two people involved in the shootout received weapons and munitions from Jamie Zajko, and one of them also applied for a marriage certificate with the person who killed Curtis Lind. Additionally I think it's also safe to say from all of their preparations that they were preparing to commit violent acts.

So my best guess is that:

• Teresa Youngblut and/or Felix Bauckholt were co-conspirators with the other people committing violent crimes
• They were preparing to commit fur

I think an accident that caused a million deaths would do it.

I think this post is quite good, and gives a heuristic important to modeling the world. If you skipped it because of title + author, you probably have the wrong impression of its contents and should give it a skim. Its main problem is what's left unsaid.

Some people in the comments reply to it that other people self-deceive, yes, but you should assume good faith. I say - why not assume the truth, and then do what's prosocial anyways?

You're probably right, I don't actually know many/haven't actually interacted personally with many trans people. But also, I'm not really talking about the Zizians in particular here, or the possibility of getting physically harmed? It just seems like being trans is like taking LSD, in that it makes a person ex-ante much more likely to be someone who I've heard of having a notoriously bizarre mental breakdown that resulted in negative consequences for the people they've associated themselves with.

This is the craziest shit I have ever read on LessWrong, and I am mildly surprised at how little it is talked about. I get that it's very close to home for a lot of people, and that it's probably not relevant to either rationality as a discipline or the far future. But like, multiple unsolved murders by someone involved in the community is something that I would feel compelled to write about, if I didn't get the vague impression that it'd be defecting in some way.

Most of the time when people publicly debate "textualism" vs. "intentionalism" it smacks to me of a bunch of sophistry to achieve the policy objectives of the textualist. Even if you tried to interpret English statements like computer code, which seems like a really poor way to govern, the argument that gets put forth by the guy who wants to extend the interstate commerce clause to growing weed or whatever is almost always ridiculous on its own merits.

The 14th amendment debate is unique, though, in that the letter of the amendment goes one way, and the sta... (read more)

What’s reality? I don’t know. When my bird was looking at my computer monitor I thought, ‘That bird has no idea what he’s looking at.’ And yet what does the bird do? Does he panic? No, he can’t really panic, he just does the best he can. Is he able to live in a world where he’s so ignorant? Well, he doesn’t really have a choice. The bird is okay even though he doesn’t understand the world. You’re that bird looking at the monitor, and you’re thinking to yourself, ‘I can figure this out.’ Maybe you have some bird ideas. Maybe that’s the best you can do

Sarcasm is when we make statements we don't mean, expecting the other person to infer from context that we meant the opposite. It's a way of pointing out how unlikely it would be for you to mean what you said, by saying it.

There are two ways to evoke sarcasm; first by making your statement unlikely in context, and second by using "sarcasm voice", i.e. picking tones and verbiage that explicitly signal sarcasm. The sarcasm that people consider grating is usually the kind that relies on the second category of signals, rather than the first. It becomes more fu... (read more)

This all seems really clearly true.

Just reproduced it; all I have to do is subscribe to a bunch of people and this happens and the site becomes unusable: