All of johnswentworth's Comments + Replies

Did you intend to post this as a reply in a different thread?

It feels like unstructured play makes people better/stronger in a way that structured play doesn't.

What do I mean? Unstructured play is the sort of stuff I used to do with my best friend in high school:

• unscrewing all the cabinet doors in my parents' house, turning them upside down and/or backwards, then screwing them back on
• jumping in and/or out of a (relatively slowly) moving car
• making a survey and running it on people at the mall
• covering pool noodles with glow-in-the-dark paint, then having pool noodle sword fights with them at night while the paint is s

Eh, depends heavily on who's presenting and who's talking. For instance, I'd almost always rather hear Eliezer's interjection than whatever the presenter is saying.

I mean, I see why a rule of "do not spontaneously interject" is a useful heuristic; it's one of those things where the people who need to shut up and sit down don't realize they're the people who need to shut up and sit down. But still, it's not a rule which carves the space at an ideal joint.

An heuristic which errs in the too-narrow direction rather than the too-broad direction but still plausibly captures maybe 80% of the value: if the interjection is about your personal hobbyhorse or pet peave or theory or the like, then definitely shut up and sit down.

Yeah... I still haven't figured out how to think about that cluster of pieces.

It's certainly a big part of my parents' relationship: my mother's old job put both her and my father through law school, after which he worked while she took care of the kids for a few years, and nowadays they're business partners.

In my own bad relationship, one of the main models which kept me in it for several years was "relationships are a thing you invest in which grow and get better over time". (Think e.g. this old post.) And it was true that the relationship got better ove... (read more)

I find the "mutual happy promise of 'I got you'" thing... suspicious.

For starters, I think it's way too male-coded. Like, it's pretty directly evoking a "protector" role. And don't get me wrong, I would strongly prefer a woman who I could see as an equal, someone who would have my back as much as I have hers... but that's not a very standard romantic relationship. If anything, it's a type of relationship one usually finds between two guys, not between a woman and <anyone else her age>. (I do think that's a type of relationship a lot of guys crave, to... (read more)

I think your experience does not generalize to others as far as you think it does. For instance, personally, I would not feel uncomfortable whispering in a friend's ear for a minute ASMR-style; it would feel to me like a usual social restriction has been dropped and I've been freed up to do something fun which I'm not normally allowed to do.

Indeed!

This post might (no promises) become the first in a sequence, and a likely theme of one post in that sequence is how this all used to work. Main claim: it is possible to get one's needs for benefits-downstream-of-willingness-to-be-vulnerable met from non-romantic relationships instead, on some axes that is a much better strategy, and I think that's how things mostly worked historically and still work in many places today. The prototypical picture here looks like both romantic partners having their separate tight-knit group of (probably same-sex) fri... (read more)

No, I got a set of lasertag guns for Wytham well before Battleschool. We used them for the original SardineQuest.

More like: kings have power via their ability to outsource to other people, wizards have power in their own right.

A base model is not well or badly aligned in the first place. It's not agentic; "aligned" is not an adjective which applies to it at all. It does not have a goal of doing what its human creators want it to, it does not "make a choice" about which point to move towards when it is being tuned. Insofar as it has a goal, its goal is to predict next token, or some batch of goal-heuristics which worked well to predict next token in training. If you tune it on some thumbs-up/thumbs-down data from humans, it will not "try to correct the errors in the data supplied... (read more)

John's Simple Guide To Fun House Parties

The simple heuristic: typical 5-year-old human males are just straightforwardly correct about what is, and is not, fun at a party. (Sex and adjacent things are obviously a major exception to this. I don't know of any other major exceptions, though there are minor exceptions.) When in doubt, find a five-year-old boy to consult for advice.

Some example things which are usually fun at house parties:

• Dancing
• Swordfighting and/or wrestling
• Lasertag, hide and seek, capture the flag
• Squirt guns
• Pranks
• Group singing, but not at a h

That... um... man, you seem to be missing what may be the actual most basic/foundational concept in the entirety of AI alignment.

To oversimplify for a moment, suppose that right now the AI somehow has the utility function u over world-state X, and E[u(X)] is maximized by a world full of paperclips. Now, this AI is superhuman, it knows perfectly well what the humans building it intend for it to do, it knows perfectly well that paperclips only maximize its utility function because of a weird accident of architecture plus a few mislabeled points during traini... (read more)

You are a gentleman and a scholar, well done.

And if numbers from pickup artists who actually practice this stuff look like 5%-ish, then I'm gonna go ahead and say that "men should approach women more", without qualification, is probably just bad advice in most cases.

EDIT-TO-ADD: A couple clarifications on what that graph shows, for those who didn't click through. First, the numbers shown are for getting a date, not for getting laid (those numbers are in the linked post and are around 1-2%), so this is a relevant baseline even for guys who are not primarily aiming for casual sex. Second, these "approaches" involve ~15 minutes each of chatting, so we're not talking about a zero-effort thing here.

There was a lot of "men, you should approach women more" in this one, so I'm gonna rant about it a bit.

I propose a useful rule: if you advise single men to approach women more, you must give a numerical estimate of the rate at which women will accept such an advance. 80%? 20%? 5%? 1%? Put a FUCKING number on it. You can give the number as a function of (other stuff) if you want, or make it conditional on (other stuff), but give some kind of actual number.

You know why? Because I have approached women for dating only a handful of times in my life, and my suc... (read more)

I don't understand why this has to be such a dichotomy?

Good question. I intentionally didn't get into the details in the post, but happy to walk through it in the comments.

Imagine some not-intended-to-be-realistic simplified toy organism with two cell types, A and B. This simplified toy organism has 1001 methyl modification sites which control the cell's type; each site pushes the cell toward either type A or type B (depending on whether the methyl is present or not), and whatever state the majority push toward, that's the type the cell will express.

Under ... (read more)

So you're saying that the persistent epigenetic modification is a change in the "equilibrium state" of a potentially methylated location? 

Yes.

Does this mean that the binding affinity of the location is the property that changes?

Not quite, it would most likely be a change in concentrations of enzymes or cofactors or whatever which methylate/demethylate the specific site (or some set of specific sites), rather than a change in the site itself.

My default assumption on all empirical ML papers is that the authors Are Not Measuring What They Think They Are Measuring.

It is evidence for the natural abstraction hypothesis in the technical sense that P[NAH|paper] is greater than P[NAH], but in practice that's just not a very good way to think about "X is evidence for Y", at least when updating on published results. The right way to think about this is "it's probably irrelevant".

Good question. The three directed graphical representations directly expand to different but equivalent expressions, and sometimes we want to think in terms of one of those expressions over another.

The most common place this comes up for us: sometimes we want to think about a latent $\Gamma$ in terms of its defining conditional distribution $P\left[\Gamma |X\right]$. When doing that, we try to write diagrams with $\Gamma$ downstream, like e.g. $X_1\to X_2\to \Gamma$. Other times, we want to think of $\Gamma$ in terms of the distribution $P\left[X|\Gamma \right]$. When thinking that wa... (read more)

Yup, and you can do that with all the diagrams in this particular post. In general, the $D_{KL}$ error for any of the diagrams $X\to Y\to Z$, $X\leftarrow Y\leftarrow Z$, or $X\leftarrow Y\to Z$ is $I(X;Z|Y)$ for any random variables $X,Y,Z$.

Notably, this includes as a special case the "approximately deterministic function" diagram $X\leftarrow Y\to X$; the error on that one is $I(X;X|Y)$ which is $H\left(X|Y\right)$.

Good questions. I apparently took too long to answer them, as it sounds like you've mostly found the answers yourself (well done!).

"Approximately deterministic functions" and their diagrammatic expression were indeed in Deterministic Natual Latents; I don't think we've talked about them much elsewhere. So you're not missing any other important source there.

On why we care, a recent central example which you might not have seen is our Illiad paper. The abstract reads:

Suppose two Bayesian agents each learn a generative model of the same environment. We will a

Does a very strong or dextrous person have more X-power than a weaker/clumsier person, all else equal?

I think that's a straightforward yes, for purposes of the thing the post was trying to point to. Strength/dexterity, like most wizard powers, are fairly narrow and special-purpose; I'm personally most interested in more flexible/general wizard powers. But for me, I don't think it's a-priori about turning knowledge into stuff-happening-in-the-world; it just turns out that knowledge is usually instrumental.

I think the thing I'm trying to point to is importantly wizard power and not artificer power; it just happens to be an empirical fact about today's world that artifice is the most-developed branch of wizardry.

For example:

• Doctors (insofar as they're competent) have lots of wizard power which is importantly not artificer power. Same with lawyers. And pilots. And special ops soldiers. And athletes (though their wizard powers tend to be even more narrow in scope).
• Beyond lawyers, the post mentions a few other forms of social/bureaucratic wizardry, which are importantly wizard power and not artificer power.

Likelihood: maybe 5-30% off the top of my head, obviously depends a lot on operationalization.

Time: however long transformer-based LLMs (trained on prediction + a little RLHF, and minor variations thereon) remain the primary paradigm.

Trying to think of reasons this post might end up being quite wrong, I think the one that feels most likely to me is that these management and agency skills end up being yet another thing that LLMs can do very well very soon. [...]

I'll take the opposite failure mode: in an absolute sense (as opposed to relative-to-other-humans), all humans have always been thoroughly incompetent at management; it's impressive that any organization with dedicated managers manages to remain functional at all given how bad they are (again, in an absolute sense). LLMs are even... (read more)

(Note that F, in David's notation here, is a stochastic function in general.)

Yup, that's exactly the right idea, and indeed constructing $\Omega$ by just taking a tuple of all the redund $\Gamma$'s (or a sufficient statistic for that tuple) is a natural starting point which works straightforwardly in the exact case. In the approximate case, the construction needs to be modified - for instance, regarding the third condition, that tuple of $\Gamma$'s (or a sufficient statistic for it) will have much more entropy conditional on $X$ than any individual $\Gamma$.

Perhaps a more useful prompt for you: suppose something indeed convinces the bulk of the population that AI existential risk is real in a way that's as convincing as the use of nuclear weapons at the end of World War II. Presumably the government steps in with measures sufficient to constitute a pivotal act. What are those measures? What happens, physically, when some rogue actor tries to build an AGI? What happens, physically, when some rogue actor tries to build an AGI 20 or 40 years in the future when alorithmic efficiency and Moore's law have lowered t... (read more)

Glad you liked it!

I'm very skeptical that focusing on wizard power is universally the right strategy. For example, I think that it would be clearly bad for my effect on existential safety for me to redirect a bunch of my time towards learning about the things you described (making vaccines, using CAD software, etc)...

Fair as stated, but I do think you'd have more (positive) effect on existential safety if you focused more narrowly on wizard-power-esque approaches to the safety problem. In particular, outsourcing the bulk of alignment work (or a pivotal act... (read more)

Interesting though about using it to improve one's performance rather than just as an antidepressant or aid to quit smoking.

IIUC the model here is that "Rat Depression" in fact is just depression (see downthread), so the idea is to use bupropion as just an antidepressant. The hypothesis is that basically-physiologically-ordinary depression displays differently in someone who e.g. already has the skills to notice when their emotions don't reflect reality, already has the reality-tracking meta-habits which generate CBT-like moves naturally, has relatively we... (read more)

I think this is importantly wrong.

Taking software engineering as an example: there are lots of hobbyists out there who have done tons of small programming projects for years, and write absolutely trash code and aren't getting any better, because they're not trying to learn to produce professional-quality code (or good UI, or performant code). Someone who's done one summer internship with an actual software dev team will produce much higher quality software. Someone who's worked a little on a quality open-source project will also produce much higher quality... (read more)

I do think one needs to be strategic in choosing which wizard powers to acquire! Unlike king power, wizard powers aren't automatically very fungible/flexible, so it's all the more important to pick carefully.

I do think there are more-general/flexible forms of wizard power, and it makes a lot of sense to specialize in those. For instance, CAD skills and knowing how to design for various production technologies (like e.g. CNC vs injection molding vs 3D printing) seems more flexibly-valuable than knowing how to operate an injection molding device oneself.

Wow I'm surprised I've never heard of that class, sounds awesome! Thank you.

One can certainly do dramatically better than the baseline here. But the realistic implementation probably still involves narrowing your wizardly expertise to a few domains, and spending most of your time tinkering with projects in them only. And, well, the state of the world being what it is, there's unfortunately a correct answer to the question regarding what those domains should be.

That's an issue I thought about directly, and I think there are some major loopholes. For example: mass production often incentivizes specializing real hard in producing one... (read more)

Yeah, hackerspaces are an obvious place to look for wizard power, but something about them feels off. Like, they're trying to be amateur spaces rather than practicing full professional-grade work.

And no, I do not want underlings, whether wizard underlings or otherwise! That's exactly what the point isn't.

Meta: I probably won't respond further in this thread, as it has obviously gone demon. But I do think it's worth someone articulating the principle I'd use in cases like this one.

My attitude here is something like "one has to be able to work with moral monsters". Cremieux sometimes says unacceptable things, and that's just not very relevant to whether I'd e.g. attend an event at which he features. This flavor of boycotting seems like it would generally be harmful to one's epistemics to adopt as a policy.

(To be clear, if someone says "I don't want to be at ... (read more)

@Alfred Harwood @David Johnston 

If anyone else would like to be tagged in comments like this one on this post, please eyeball-react on this comment. Alfred and David, if you would like to not be tagged in the future, please say so.

Here's a trick which might be helpful for anybody tackling the problem.

First, note that $f\left(\Lambda \right):=(x\mapsto P[X=x|\Lambda \left]\right)$ is always a sufficient statistic of $\Lambda$ for $X$, i.e.

$\Lambda \to f\left(\Lambda \right)\to X$

Now, we typically expect that the lower-order bits of $f\left(\Lambda \right)$ are less relevant/useful/interesting. So, we might hope that we can do some precision cutoff on $f\left(\Lambda \right)$, and end up with an approximate suficient statistic, while potentially reducing the entropy (or some other information content measure) of $f\left(\Lambda \right)$ a bunch. We'd broadcast the cutoff function ... (read more)

I wouldn't necessarily expect this to be what's going on, but just to check... are approximately-all the geoguessr images people try drawn from a single dataset on which the models might plausibly have been trained? Like, say, all the streetview images from google maps?

We did end up doing a version of this test. A problem came up in the course of our work which we wanted an LLM to solve (specifically, refactoring some numerical code to be more memory efficient). We brought in Ray, and Ray eventually concluded that the LLM was indeed bad at this, and it indeed seemed like our day-to-day problems were apparently of a harder-for-LLMs sort than he typically ran into in his day-to-day.

Another guise of the same problem: it would be great if an LLM could summarize papers for me. Alas, when an LLM is tasked with summarizing a paper, I generally expect it to "summarize" the paper in basically the same way the authors summarize it (e.g. in the abstract), which is very often misleading or entirely wrong. So many papers (arguably a majority) measure something useful, but then the authors misunderstand what they measured and therefore summarize it in an inaccurate way, and the LLM parrots that misunderstanding.

Plausibly the most common reason I... (read more)

Here's some problems I ran into the past week which feel LLM-UI-shaped, though I don't have a specific solution in mind. I make no claims about importance or goodness.

Context (in which I intentionally try to lean in the direction of too much detail rather than too little): I was reading a neuroscience textbook, and it painted a general picture in which neurotransmitters get released into the synaptic cleft, do their thing, and then get reabsorbed from the cleft into the neuron which released them. That seemed a bit suspicious to me, because I had previousl... (read more)

(This is much closer to my own take, and is something Steve and I argue about occasionally.)

It looks like some paragraph breaks were lost in this post.