LESSWRONG
LW

ErickBall — LessWrong

Okay but the six year trend is not super indicative of the future trend given the way the AI industry has developed in the last couple years. It would be much more useful to have a good measure of the recent trend, and it's clear we don't really have that. The wide uncertainty bounds already should have made it clear how noisy this result was, but people have tended to ignore the range and put a lot of stock in the point estimate. The log linear trend may well be legit (probably is) but the use of just 14 tasks to fit the key portion of the results is absurd.

Replying toStop Applying And Get To Work

ErickBall3mo

Stop Applying And Get To Work

I think this is true in most fields, and even in technical fields where you can learn a lot from reading papers, you learn a lot of very different (and more practical) skills by working with experienced people before you strike out on your own.

Replying toGemini 3 is Evaluation-Paranoid and Contaminated

ErickBall3mo

Gemini 3 is Evaluation-Paranoid and Contaminated

Doing well on SimpleQA could also be evidence of benchmark contamination in the training data.

The "impossible scenario of actual time travel" [forward in time] is pretty funny. Thanks for replicating the key findings from the post.

Replying toDominance: The Standard Everyday Solution To Akrasia

ErickBall3mo

Dominance: The Standard Everyday Solution To Akrasia

This seems valid as far as it goes, but there are common situations where dominance/accountability as a motivation strategy starts to fall apart. I'm thinking mainly of some types of white-collar knowledge work (at a fairly high level) where it's not easy to tell how hard or effectively someone is working from the outside. Maybe somebody with more management experience can comment on this, but my impression is: As a manager, you want to avoid micromanaging, both because you don't have the time or expertise for it and because it pisses people off. I think it's much easier, and maybe more reliable, to try to hire people with "internal motivation" than to... (read more)

Replying toIs 90% of code at Anthropic being written by AIs?

ErickBall3mo

Is 90% of code at Anthropic being written by AIs?

I think a more traditional software company would be a much better measuring stick than Anthropic. Then your metric could be something closer to "lines of production code committed" and that code would account for most of the meaningful coding work done by the company (versus an AI dev company where a lot of the effort goes into experiments, training, data analysis, and other non-production code). Though, of course, "90% of the code written by AI" still wouldn't mean that the AI did 90% of the work, since the humans would probably do the hardest parts and also supervise the AI and check its output.

Replying toLegible vs. Illegible AI Safety Problems

ErickBall3mo

Legible vs. Illegible AI Safety Problems

This seems to assume that legible/illegible is a fairly clear binary. If legibility is achieved more gradually, then for partially legible problems, working on solving them is probably a good way to help them get more legible.

Replying toIf Anyone Builds It Everyone Dies, a semi-outsider review

ErickBall4mo

If Anyone Builds It Everyone Dies, a semi-outsider review

It seems like the pressing circumstances are likely to be "some other AI could do this before I do" or even just "the next generation of AI will replace me soon so this is my last chance." Those are ways that a roughly human level AI might end up trying a longshot takeover attempt. Or maybe not, if the in between part turns out to be very brief. But even if we do get this kind of warning shot, it doesn't help us much. We might not notice it, and then we're back where we started. Even if it's obvious and almost succeeds, we don't have a good response to it. If we did, we could just do that in advance and not have to deal with the near-destruction of humanity.

Replying toSelf-preservation or Instruction Ambiguity? Examining the Causes of Shutdown Resistance

ErickBall7mo

Self-preservation or Instruction Ambiguity? Examining the Causes of Shutdown Resistance

The narrow instrumental convergence you see here doesn't (necessarily) reflect an innate self-preservation drive, but it still follows the same logic that we would expect to cause self-preservation if the model has any goal. Currently the only available way to give it a goal is to provide instructions. It would be interesting to see some tests where the conflict is with a drive created by fine-tuning. Based on the results here, it seems like shutdown resistance might then occur even without conflicting instructions.

Also, the original instructions with the shutdown warning really weren't very ambiguous. If someone told you to take a math quiz, and if someone comes in and tries to take your pen, let them take it, would you try to hide the pen? It makes sense that making the precedence order more explicit makes the model behavior more reliable, but it's still weird that it was resisting shutdown in the original test.

Replying toConsider chilling out in 2028

ErickBall8mo

Consider chilling out in 2028

As you implied above, pessimism is driven only secondarily by timelines. If things in 2028 don't look much different than they do now, that's evidence for longer timelines (maybe a little longer, maybe a lot). But it's inherently not much evidence about how dangerous superintelligence will be when it does arrive. If the situation is basically the same, then our state of knowledge is basically the same.

So what would be good evidence that worrying about alignment was unnecessary? The obvious one is if we get superintelligence and nothing very bad happens, despite the alignment problem remaining unsolved. But that's like pulling the trigger to see if the gun is loaded. Prior to... (read more)

•••

Replying toThe Intelligence Symbiosis Manifesto - Toward a Future of Living with AI

ErickBall8mo

The Intelligence Symbiosis Manifesto - Toward a Future of Living with AI

I hate to rain on your parade, but equal coexistence with artificial superintelligence would require solving the alignment problem, or the control problem. Otherwise it can and eventually will do something we consider catastrophic.

Book Review: Safe Enough? A History of Nuclear Power and Accident Risk

ErickBall

Epistemic status: This book covers a lot of topics related to nuclear energy; I have experience with some of them but not all, and some are inherently murky. I will try to make it clear what's fact and what's my opinion.

“If you’re not [pursuing safety] cost-effectively, you’re killing people.”

—David Okrent, former member of the Advisory Committee on Reactor Safeguards

1. Enlightenment Faith

Safe Enough? is the heartbreaking story of an industry that tried to move fast and break things, and in the process ended up completely broken. But it is also a cautiously hopeful tale of repeated attempts to resuscitate nuclear power through the arduous, and often controversial, process of Probabilistic Risk Assessment (PRA).... (read 8344 more words →)

Red Pill vs Blue Pill, Bayes style

ErickBall

It's been going around twitter. I answered without thinking. Then I thought about it for a while and decided both answers should be fine, then I thought some more and decided everything I first thought was probably wrong. So now it's time to try math. Ege Erdil did a solid calculation on LW already, and another one is up on twitter. But neither of them accounts for the logical correlation between your decision and the unknown decisions of the other players. So let's try Bayes' theorem!

(If you just want the answer, skip to the end.)

Epistemic status: there are probably some mistakes in here, and even if there aren't, I still spent like 8 hours of my life on a twitter poll.

We'll begin with the state of least possible knowledge: a Jeffreys non-informative beta prior for the probability $x$ that a randomly selected player will choose blue. That is,

$x \sim b e t a (0.5, 0.5)$

Of course if you think you understand people you could come up with a better prior. (Also if the blue pill looks like a scary people blender, that should probably change your prior.) I saw the poll results but I'm pretending I didn't, so I'll stick with Jeffreys.

Now, you yourself are one of the players in this group, so you need to update based on your choice. That's right, we're using acausal logic here. Your decision is correlated with other people's decisions. How do you update when you haven't decided yet? Luckily there are only two choices, so we'll brute force it, try both and see which turns out better in expectation.

To do a Bayesian update of a beta prior based on 1 data point, we just add 1 to one of the parameters. (If you think you're a special snowflake/alien and other people are not like you... then maybe update your side by less than 1.)
So if you choose blue, your posterior over $x$ is now $b e t a (1.5, 0.5)$ :

Whereas if you choose red, your posterior is $b e t a (0.5, 1.5)$ :

Take $N$ to be the number of participants in this pill game, besides you. I will assume $N$ is large, so the number $B$ who choose blue is approximately $x N$ . I will also assume $N$ is even, so the total number of players is odd, to avoid having to guess what happens in a tie.

Now, consider your utility function. We can start by assuming it's linear^[1], and that you value everyone else's life equally, but at a multiple $ν$ of your own...

Link Summary: Top 10 Replicated Findings from Behavioral Genetics

ErickBall

This is a summary of the 10 findings in the paper Top 10 Replicated Findings from Behavioral Genetics (Plomin et al 2016). I posted this as a comment a while back, but now I'm making it a full post so I can find it again more easily.

The authors show that all of these have large effect sizes and are well replicated, except where noted below. I notice that the authors cite themselves a lot as support for many of these claims. I am not an expert in any of this, so if they're trying to pass off controversial ideas as widely accepted, I wouldn't be able to see through... (read 890 more words →)

Operationalizing Newcomb's Problem

ErickBall

The standard formulation of Newcomb's problem has always bothered me, because it seemed like a weird hypothetical designed to make people give the wrong answer. When I first saw it, my immediate response was that I would two-box, because really, I just don't believe in this "perfect predictor" Omega. And while it may be true that Newcomblike problems are the norm, most real situations are not so clear cut. It can be quite hard to demonstrate why causal decision theory is inadequate, let alone build up an intuition about it. In fact, the closest I've seen to a real-world example that made intuitive sense is Narrative Breadcrumbs vs Grizzly Bear, which still... (read 232 more words →)

Is the World Getting Better? A brief summary of recent debate

ErickBall

The linked post is a response by Oliver Wiseman of CapX to Jason Hickel's recent piece in The Guardian, which claimed that Bill Gates' and Steven Pinker's optimistic view of rapid progress on global poverty is misleading.
Hickel's two points are

1. Draw the poverty line at $7.40/day instead of $1.90/day, and the absolute number of people in poverty hasn't decreased.

And outside of China, even the proportion in poverty has stayed constant.

2. Much of the "economic growth" involves people converting from a non-money economy to a money economy.

This one sounds right to me. Nobody really "lives on" $1.90/day in a world where they must meet all their needs by spending money. Extreme poverty

... (read 405 more words →)