LESSWRONG
LW

faul_sname — LessWrong

Many can write faster asm than the compiler, yet don't. Why?

1mo

There's a take I've seen going around, which goes approximately like this:

It used to be the case that you had to write assembly to make computers do things, but then compilers came along. Now we have optimizing compilers, and those optimizing compilers can write assembly better than pretty much any human. Because of that, basically nobody writes assembly anymore. The same is about to be true of regular programming.

I 85% agree with this take.

However, I think there's one important inaccuracy: even today, finding places where your optimizing compiler failed to produce optimal code is often pretty straightforward, and once you've identified those places 10x+ speedups for that specific program on that specific... (read 946 more words →)

Humans Are Spiky (In an LLM World)

faul_sname

4mo

Assessments of "general" vs "spiky" capability profiles are secretly assessments of "matches existing infrastructure" vs "doesn't".

Human societies contain human-shaped roles because humans were the only available workers for most of history. Packaging tasks into human-sized, human-shaped jobs was efficient.

Given LLMs, the obvious thing to do is to try to drop them into those roles, giving them the same tools and affordances humans have. When that fails to work, though, we should not immediately conclude that the failure is because LLMs are missing some "core of generality".

When LLM agents become more abundant than humans, as seems likely in the very near term, the most effective shape for a job stops being human-shaped. At that point, we may discover that human capability profiles are the spiky ones.

How load-bearing is KL divergence from a known-good base model in modern RL?

faul_sname

9mo

Motivation

One major risk from powerful optimizers is that they can find "unexpected" solutions to the objective function, which score very well on the objective function but are not what the human designer intended. The canonical example is

Suppose your aged mother is trapped in a burning building, and it so happens that you're in a wheelchair; you can't rush in yourself. You could cry, "Get my mother out of that building!" but there would be no one to hear.
Luckily you have, in your pocket, an Outcome Pump. This handy device squeezes the flow of time, pouring probability into some outcomes, draining it from others.
[...]
So you desperately yank the Outcome Pump from

... (read 1153 more words →)

Is AlphaGo actually a consequentialist utility maximizer?

faul_sname

faul_sname, gjm

TL;DR: does stapling an adaptation executor to a consequentialist utility maximizer result in higher utility outcomes in the general case, or is AlphaGo just weird?

So I was reading the AlphaGo paper recently, as one does. I noticed that architecturally, AlphaGo has

A value network: "Given a board state, how likely is it to result in a win". I interpret this as an expected utility estimator.
Rollouts: "Try out a bunch of different high-probability lines". I interpret this as a "consequences of possible actions" estimator, which can be used to both refine the expected utility estimate and also to select the highest-value action.
A policy network: "Given a board state, what moves are normal to see

... (read 649 more words →)

faul_sname's Shortform

faul_sname

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

221

Regression To The Mean [Draft][Request for Feedback]

faul_sname

14y

"Rewarding good performance leads to faster improvement than punishing bad performance"

"In general, unusually bad performance improves after punishment, but good performance tends not to improve and sometimes even gets worse after praise is administered."

These statements seem contradictory, yet both describe real effects. The apparent contradiction is caused by a phenomenon known as "regression to the mean," which states that the measurement after an exceptional measurement will be closer to average. The improvement after a reprimand is caused not by any effect that reprimand had, nor was the worsening after praise due to the praise. Both observations were due to regression to the mean.

Regression to the mean is caused by two things.

1. Exceptionally... (read 324 more words →)

The Dark Arts: A Beginner's Guide

faul_sname

14y

The Dark Arts

So you've been reading this site and learning many valuable tools for becoming more rational. You're beginning to become irritated at the irrational behavior of the average person. You've noticed that many people refuse to accept even highly compelling arguments, even as they drink up the doctrine of their favorite religion/political party. What are you doing wrong?

As it turns out, it's less about what you're doing wrong than about what these highly influential groups are doing right. This is a brief intro to the Dark Arts, ranging from relatively harmless or even helpful techniques to truly dangerous ones. In this set of guidelines, I have used the example of Solar... (read 945 more words →)

What would you do with a financial safety net?

faul_sname

14y

In the open thread, moridinamael hypothesized that LWers would be willing to take more risks in order to become rich if they had a financial safety net. This seems like an idea worth exploring further.

What would you do if you had a financial safety net (maybe a year's worth of living expenses) to fall back on if your venture failed?