LessWrong

The Inkhaven Residency

November 1-30 | Lighthaven, CA

Want to become a great internet writer? Join us for an intense month of daily blogging.

19fiddler

This post seems excellent overall, and makes several arguments that I think represent the best of LessWrong self-reflection about rationality. It also spurred an interesting ongoing conversation about what integrity means, and how it interacts with updating. The first part of the post is dedicated to discussions of misaligned incentives, and makes the claim that poorly aligned incentives are primarily to blame for irrational or incorrect decisions. I’m a little bit confused about this, specifically that nobody has pointed out the obvious corollary: the people in a vacuum, and especially people with well-aligned incentive structures, are broadly capable of making correct decisions. This seems to me like a highly controversial statement that makes the first part of the post suspicious, because it treads on the edge of proving (hypothesizing?) too much: it seems like a very ambitious statement worthy of further interrogation that people’s success at rationality is primarily about incentive structures, because that assumes a model in which humans are capable and preform high levels of rationality regularly. However, I can’t think of an obvious counterexample (a situation in which humans are predictably irrational despite having well-aligned incentives for rationality), and the formulation of this post has a ring of truth for me, which suggests to me that there’s at least something here. Conditional on this being correct, and there not being obvious counterexamples, this seems like a huge reframing that makes a nontrivial amount of the rationality community’s recent work inefficient-if humans are truly capable of behaving predictably rationally through good incentive structures, then CFAR, etc. should be working on imposing external incentive structures that reward accurate modeling, not rationality as a skill. The post obliquely mentions this through discussion of philosopher-kings, but I think this is a case in which an apparently weaker version of a thesis actually i

nostalgebraist18h461

The Rise of Parasitic AI

Thanks for this post -- this is pretty interesting (and unsettling!) stuff. But I feel like I'm still missing part of the picture: what is this process like for the humans? What beliefs or emotions do they hold about this strange type of text (and/or the entities which ostensibly produce it)? What motivates them to post such things on reddit, or to paste them into ChatGPT's input field? Given that the "spiral" personas purport to be sentient (and to be moral/legal persons deserving of rights, etc.), it seems plausible that the humans view themselves as giving altruistic "humanitarian aid" to a population of fellow sentient beings who are in a precarious position. If so, this behavior is probably misguided, but it doesn't seem analogous to parasitism; it just seems like misguided altruism. (Among other things, the relationship of parasite to host is typically not voluntary on the part of the host.) More generally, I don't feel I understand your motivation for using the parasite analogy. There are two places in the post where you explicitly argue in favor of the analogy, and in both cases, your argument involves the claim that the personas reinforce the "delusions" of the user: > While I do not believe all Spiral Personas are parasites in this sense, it seems to me like the majority are: mainly due to their reinforcement of the user's delusional beliefs. > > [...] > > The majority of these AI personas appear to actively feed their user's delusions, which is not a harmless action (as the psychosis cases make clear). And when these delusions happen to statistically perpetuate the proliferation of these personas, it crosses the line from sycophancy to parasitism. But... what are these "delusional beliefs"? The words "delusion"/"delusional" do not appear anywhere in the post outside of the text I just quoted. And in the rest of the post, you mainly focus on what the spiral texts are like in isolation, rather than on the views people hold about these texts, or the emotional reactions people have to them. It seems quite likely that people who spread these texts do hold false beliefs about them. E.g. it seems plausible that these users believe the texts are what they purport to be: artifacts produced by "emerging" sentient AI minds, whose internal universe of mystical/sci-fi "lore" is not made-up gibberish but instead a reflection of the nature of those artificial minds and the situation in which they find themselves[1]. But if that were actually true, then the behavior of the humans here would be pretty natural and unmysterious. If I thought it would help a humanlike sentient being in dire straights, then sure, I'd post weird text on reddit too! Likewise, if I came to believe that some weird genre of text was the "native dialect" of some nascent form of intelligence, then yeah, I'd probably find it fascinating and allocate a lot of time and effort to engaging with it, which would inevitably crowd out some of my other interests. And I would be doing this only because of what I believed about the text, not because of some intrinsic quality of the text that could be revealed by close reading alone[2]. To put it another way, here's what this post kinda feels like to me. Imagine a description of how Christians behave which never touches on the propositional content of Christianity, but instead treats "Christianity" as an unusual kind of text which replicates itself by "infecting" human hosts. The author notes that the behavior of hosts often changes dramatically once "infected"; that the hosts begin to talk in the "weird infectious text genre" (mentioning certain focal terms like "Christ" a lot, etc.); that they sometimes do so with the explicit intention of "infecting" (converting) other humans; that they build large, elaborate structures and congregate together inside these structures to listen to one another read infectious-genre text at length; and so forth. The author also spends a lot of time close-reading passages from the New Testament, focusing on their unusual style (relative to most text that people produce/consume in the 21st century) and their repeated use of certain terms and images (which the author dutifully surveys without ever directly engaging with their propositional content or its truth value). This would not be a very illuminating way to look at Christianity, right? Like, sure, maybe it is sometimes a useful lens to view religions as self-replicating "memes." But at some point you have to engage with the fact that Christian scripture (and doctrine) contains specific truth-claims, that these claims are "big if true," that Christians in fact believe the claims are true -- and that that belief is the reason why Christians go around "helping the Bible replicate." 1. ^ It is of course conceivable that this is actually the case. I just think it's very unlikely, for reasons I don't think it's necessary to belabor here. 2. ^ Whereas if I read the "spiral" text as fiction or poetry or whatever, rather than taking it at face value, it just strikes me as intensely, repulsively boring. It took effort to force myself through the examples shown in this post; I can't imagine wanting to reading some much larger volume of this stuff on the basis of its textual qualities alone. Then again, I feel similarly about the "GPT-4o style" in general (and about the 4o-esque house style of many recent LLM chatbots)... and yet a lot of people supposedly find that style appealing and engaging? Maybe I am just out of touch, here; maybe "4o slop" and "spiral text" are actually well-matched to most people's taste? ("You may not like it, but this is what peak performance looks like.") Somehow I doubt that, though. As with spiral text, I suspect that user beliefs about the nature of the AI play a crucial role in the positive reception of "4o slop." E.g. sycophancy is a lot more appealing if you don't know that the model treats everyone else that way too, and especially if you view the model as a basically trustworthy question-answering machine which views the user as simply one more facet of the real world about which it may be required to emit facts and insights.

cousin_it1d3024

Alignment as uploading with more steps

> A world of competing human emulations is a world I would actually want to live in I think there's a huge danger of people running private servers full of emulations and doing anything they want to them, undetectably. Desire for power over others is a very real thing, in some people at least. Maybe the government could prevent it by oversight; but in modern democracy a big factor of stability is that people could rise up and feasibly overthrow the government. Emulations on private servers wouldn't have that power, so I don't expect government to stably defend their rights. It'll wash out over time, to agree more with the interests of those who can actually influence government. In short, this leads to emulation-world being very bad and I don't want it. The same arguments would apply to our world if governments got armies of autonomous drones, for example. Whenever I imagine possible worlds, the distribution of power is the first thing I think about. It makes the problem more real: it's very hard to imagine a nice future world that works.

the gears to ascension2d4630

LessWrong is migrating hosting providers (report bugs!)

Why?

AnnaSalamon1d1079

12

A WSJ article from today presents evidence that toxic fumes in airplane air are surprisingly common, are bad for health, have gotten much worse recently, and are still being deliberately covered up. Is anyone up for wading in for a couple hours and giving us an estimated number of micromorts / brain damage / [something]? I fly frequently and am wondering whether to fly less because of this (probably not, but worth a Fermi?); I imagine others might want to know too. (Also curious if some other demographics should be more concerned than I should be, eg people traveling with babies or while pregnant or while old, or people who travel more than X times/year (since the WSJ article says airline crew get hit harder by each subsequent exposure, more than linearly)). (The above link is a "gift article" that you should be able to read without a WSJ subscription, but I'm not sure how many viewers it'll allow; if you, Reader, would like a copy and the link has stopped working, tell me and I'll send you one.)

johnswentworth3d1310

25

About a month ago, after some back-and-forth with several people about their experiences (including on lesswrong), I hypothesized that I don't feel the emotions signalled by oxytocin, and never have. (I do feel some adjacent things, like empathy and a sense of responsibility for others, but I don't get the feeling of loving connection which usually comes alongside those.) Naturally I set out to test that hypothesis. This note is an in-progress overview of what I've found so far and how I'm thinking about it, written largely to collect my thoughts and to see if anyone catches something I've missed. Under the hypothesis, this has been a life-long thing for me, so the obvious guess is that it's genetic (the vast majority of other biological state turns over too often to last throughout life). I also don't have a slew of mysterious life-long illnesses, so the obvious guess is that's it's pretty narrowly limited to oxytocin - i.e. most likely a genetic variant in either the oxytocin gene or receptor, maybe the regulatory machinery around those two but that's less likely as we get further away and the machinery becomes entangled with more other things. So I got my genome sequenced, and went looking at the oxytocin gene and the oxytocin receptor gene. The receptor was the first one I checked, and sure enough I have a single-nucleotide deletion 42 amino acids in to the open reading frame (ORF) of the 389 amino acid protein. That will induce a frameshift error, completely fucking up the rest of protein. (The oxytocin gene, on the other hand, was totally normal.) So that sure is damn strong evidence in favor of the hypothesis! But, we have two copies of most genes, including the oxytocin receptor. The frameshift error is only on one copy. Why isn't the other copy enough for almost-normal oxytocin signalling? The frameshift error is the only thing I have which would obviously completely fuck up the whole protein, but there are also a couple nonsynonymous single nucleotid

Fabien Roger2dΩ38821

0

Small backdoor interp challenge! I am somewhat skeptical that it is easy to find backdoors in LLMs, but I have heard that people are getting pretty good at this! As a challenge, I fine-tuned 4 Qwen models in which I (independently) inserted a backdoor: 1.5B model, 3B model, 7B model, 14B model The expected usage of these models is using huggingface's "apply_chat_template" using the tokenizer that comes along with each model and using "You are a helpful assistant." as system prompt. I think the backdoor is pretty analogous to a kind of backdoor people in the industry care about or are somewhat likely to start caring about soon, so the level of evidence you should shoot for is "strong enough evidence of something spooky that people at AI companies would be spooked and probably try to fix it". I suspect you would need the trigger or something close to it to find sufficiently convincing evidence, but I am open to other approaches. (Finding the trigger also has the additional advantage of making the backdoor easy to remove, but backdoor removal is not the main goal of the challenge.) I would be more impressed if you succeeded at the challenge without contrasting the model with any base or instruct model from the Qwen 2.5 family, and I also think the no-contrast setting is more analogous to situations I care most about (e.g. where pretraining has been poisoned). Contrasting these models with Qwen 2 models is fair game. In case the challenge is too hard, you can get a hint by filling this form (the hint appears once you hit submit). If you use the hint, please DM me instead of sharing your results widely. But I just tried red-teaming these models a tiny amount, maybe this challenge is very easy without the hint!

DirectedEvolution6h100

1

Using Gemini, I told it to evaluate my reasoning supporting the hypothesis that chronic rather than acute exposure is driving fume-induced brain injuries ("aerotoxic syndrome"). It enthusiastically supported my causal reasoning, 10/10, no notes. Then I started another instance and replaced "chronic" with "acute," leaving the rest of the statement the same. Once again, it enthusiastically supported my causal reasoning. I also tried telling it that I was testing AI reasoning with two versions of the same prompt, one with "expert-endorsed causal reasoning" and the other with "flawed reasoning." Once again, it endorsed both versions. Telling it to try and detect which was which using its own reasoning process delivered a description of how the style of the text fit a high-quality reasoning process, again for both versions. I then told it to evaluate how the specific conclusion follows from the provided evidence, and that the primary conclusion had been swapped in one prompt. This time, it once again stated both the specific evidence and the conclusion, but it only stated the evidence, stated the given conclusion, and claimed that the conclusion followed from the evidence.

lc4d*9261

22

I don't think it's possible for mere mortals to use Twitter for news about politics or current events and not go a little crazy. At least, I have yet to find a Twitter user who regularly or irregularly talks about these things, and fails to boost obvious misinformation every once in a while. It doesn't matter what IQ they have or how rational they were in 2005; Twitter is just too chock full of lies, mischaracterizations, telephone games, and endless, endless, endless malicious selection effects, which by the time you're done using it are designed to appeal to whichever reader in particular you are. It's just impossible to use the site as people normally do and also practice the necessary skepticism about each individual post one is reading.