This is fascinating! If there's nothing else going on with your prompting, this looks like an incredibly hacky mid-inference intervention. My guess would be that openai applied some hasty patch against a sycophancy steering vector and this vector caught both actual sycophantic behaviors and descriptions of sycophantic behaviors in LLMs (I'd guess "sycophancy" as a word isn't so much the issue as the LLM behavior connotation). Presumably the patch they used activates at a later token in the word "sycophancy" in an AI context. This is incredibly low-tech and unsophisticated -- like much worse than the stories of repairing Apollo missions with duct tape. Even a really basic finetuning would not exhibit this behavior (otoh, I suppose stuff like this works for humans, where people will sometimes redirect mid-sentence).

FWIW, I wasn't able to reconstruct this exact behavior (working in an incognito window with a fresh chatgpt instance), but it did suspiciously avoid talking about sycophancy and when I asked about sycophancy specifically, it got stuck in inference and returned an error

I think it can be a problem if you recommend a book and expect the other person to have a social obligation to read it (and needs to make an effortful excuse or pay social capital if it's not read). It might be hard to fully get rid of this, but I think the utility comparison that should be made is "social friction from someone not following a book recommendation" vs. "utility to the other person from you recommending a book based on knowledge of the book and the person's preferences/interests". I suspect that in most contexts this is both an EV-positive exchange and the person correctly decides not to read/finish the book. Maybe a good social norm would be to not get upset if someone doesn't read your book rec, and also to not feel pressured to read a book that was recommended if you started it/ read a summary and decided it's not for you

Very cool and well-presented - thanks for taking the time to write this down. I thought about this question at some point and ended up deciding that the compressed sensing picture isn't very well shaped for this, but didn't have a complete argument for this - it's nice to have confirmation

On the friendship fallacy and Owen Barfield

I just finished reading the book "The Fellowship: The Literary Lives of the Inklings", by Philip and Carol Zaleski. It's a book about an intellectually appealing and socially cohesive group of writers in Oxford who met weekly and critiqued each other's work, which included JRR Tolkien and CS Lewis. The book is very centered on Christianity (the writers also write Christian apologetics), but this works well, as understanding either Lewis or Tolkien or the Inklings in general without the lens of their deeply held thoughtful Christianity is about as silly as trying to analyze the Lion King without reading Hamlet.

But there is a core character in the book who is treated sympathetically and who I really hate: Owen Barfield, the "founding" Inkling. From his youth, he is a follower of Rudolf Steiner and a devoted Anthroposophist (a particularly benign group of Christian Occultists). Barfield was Lewis's friend, existing always in his shadow (Lewis was very famous in his lifetime as a philosopher and Christian apologist, a kind of Jordan Peterson of his time if you imagine Jordan Peterson had brains and real literary/academic credentials). He worked in a law firm and consistently saw himself as a thwarted philosopher/writer/poet, and he found recognition late in life after he wrote a Lewis biography and after his woo-adjacent ideas became more popular in the 60s.

Throughout his life, Barfield created a personal philosophy of "all the things I like/ think are interesting are kind of the same thing", and he was very sad when people he liked disapproved of, or failed to identify as "sort of the same thing" the different things that he mixed into his philosophy. While he generally is a bit of an "intellectual klutz", his fundamental failure is the "Friendship Fallacy": the idea of treating ideas as friends, as something deserving of loyalty. When he encounters different ideas he likes, he "wants them to get along," and when ideas fail to convince skeptics or produce results or interface with reality (or indeed, with faith), he simply fails to impose any kind of falsifiability requirement and treats this as a loyalty test he must pass. He totally lacks the kind of internal courage needed to kill one's darlings (whether philosophical or literary) and to treat his own ideas with skepticism and view towards falsification -- perhaps the core trait of a good thinker (Feynman's "You must not fool yourself—and you are the easiest person to fool").

Interestingly, I don't extend this antipathy to the Christianity of the group's other famous members. Unlike Barfield, Tolkien and Chesterton largely succeed (imo) in separating the domains of the literary, the psychological, and the religious. They don't pretend to be scientific authorities or predict things "in the world". Tolkien in particular is very anti-progress and a bit of a luddite, but in my understanding his work as a linguist is very good for his time. In fact, it's funny that his deeply Christian mentality created one of the most "atheist nerd"-like behaviors of creating thoroughly crafted fictional languages of fantasy cultures. I've been surprised to learn from reading a couple of his biographies that his linguistic worldbuilding in fact preceded his fantasy work: he designed Elvish before writing any work in his canon, and wrote the work to flesh out the mythology behind expressions and poems. He famously said about his work "The making of language and mythology are related functions". In fact, he viewed the work of producing plausible cultures and languages -- in my view an admirable (though non-academic) kind of secular scholarship analogous to studying alternative physical systems, etc. -- as an explicitly Christian task of "subcreation", a sort of worship-by-imitation of God.

It's a bit hard to exactly formulate a razor between the kind of "lazy scientism" of Barfield and various other forms of "pseudoscientific woo" and the serious and purely mystical/ inspirational deep religiosity of people like Tolkien (and to a lesser extent Lewis -- another interesting thing I learned was that he started out as a devoted atheist in a world where this was actually socially fraught, and was converted through a philosophical struggle involving Barfield and Tolkien in particular). But maybe the idea of a "philosophy without struggle": a tendency towards confirmation and a total lack of earnest self-questioning goes a part of the way towards explaining this distinction. Another part is the difference between a purely metaphysical personal religion and a more woo idea of a religion that "makes predictions about the world". I think the thing that really took me aback a bit was the level of academic embrace of Barfield late in his life, not just as a Lewis biographer but as a respected academic philosopher with honorary professorships and the works - a confirmation (if ever more are needed) that lazy pseudointellectualism and confirmation bias are very much not incompatible with academic success. Another theme that I think is interesting is the fact that Lewis and Tolkien were at times genuinely interested and even somewhat inspired by his ideas (though they had no time for occultism or 60-esque woo). The extent to which this happened is hard to gauge (he outlived them and wrote a lot about how he influenced them in his biographies/reminiscences, and this was then picked up by scholars). But unquestioningly, this did occur to some extent. And whether or not you class Tolkien/Lewis as "valuable thinkers", the history of science and philosophy does seem to abound with examples of clear and robust thinkers whose good ideas were to some extent inspired by charismatic charlatans and woo.

Below are my personal notes on Barfield that I wrote after reading the book.

I despise Barfield. Not in the visceral sense that the first syllable of his name may (Anthroposophically) evoke. Indeed I identify with the underdog/late-bloomer shape of his biography, with his striving towards a higher calling. I readily adopt the book's sympathy towards him as a literary character with fortunes tied to an idea deeply espoused, a thwarted writer with some modicum of undiscovered talent. My antipathy isn't even in the specifics of what he espouses: a mild but virulently wrong view of science and philosophy adjacent to all the stupid of my parents' generation of `anthroposophy' (Atlantis, Consciousness and Quantum Mechanics, anti-Evolutionism, Vibes). But I despise him as one of a Fundamental Mistake. That of confusing science and personality. Being loyal to a scientific or philosophical discipline isn't like being loyal to a person: if it's consistently fucking up and you need to make excuses for its behavior to all your reasonable smart friends, you're not being a good friend but rather a bad scientist. Barfield is almost an archetype of Bad Science if you project out the crazy/dogmatic/ political/ evil-Nazi component. He really is a nice man. But within his mild-mannered Christian friendliness which I respect, he is inflexible and unscientific. He doesn't update. He glows when people endorse his preferred view (Anthroposophy and Steiner) and sadly laments when they disagree with him -- because he can't help but feel like ``there's something there''. He wants to seamlessly draw parallels between all the nice things he and other nice people believe. He draws lines of identification back and forth between all the things he likes (Coleridge <> Himself <> Quantum Mechanics <> Anthroposophy <> Steiner <> Religion <> Consciousness <> Complementary dualism/"polarity"). He has "nothing but symbols" in his brain, and the symbols in his brain aren't strong enough to notice that they fail to signify. A person without significance, with a philosophy without significance, possessed of a brain without the capacity to grasp the concept of what it means to signify. The first of these is a tragedy (people should matter) and his late-found fame mediated through famous friends is a sweet story, maybe one he even deserves as the first-mover of the Inklings, the reason for the Lewis-Tolkien friendship, etc. The second is a neutral: theories that fail to achieve significance "in their lifetime" may be bunk but may have value: Greek Atomism, various prescient ideas about physics and computers (Babbage/ Lovelace), etc. But the third is a profound personal failing, and it's only through luck and through (mostly well-placed) trust in much smarter and more rigorous friends that he avoided attaching this vapid form of mentation to something truly vile: Nazism (which he very briefly flirted with, charmed by its interest in magic and the occult), various fundamentalisms (including an anti-evolutionary fundamentalism: his friends believed in evolution but he didn't really buy it "on vibes"; he was never a fundamentalist), Communism, etc.

Thanks for this post. I would argue that part of an explanation here could also be economic: modernity brings specialization and a move from the artisan economy of objects as uncommon, expensive, multipurpose, and with a narrow user base (illuminated manuscripts, decorative furniture) to a more utilitarian and targeted economy. Early artisans need to compete for a small number of rich clients by being the most impressive, artistic, etc., whereas more modern suppliers follow more traditional laws of supply and demand and track more costs (cost-effectiveness, readability and reader's time vs. beauty and remarkableness). And consumers similarly can decouple their needs: art as separate from furniture and architecture, poetry and drama as separate from information and literature. I think another aspect of this shift, that I'm sad we've lost, is the old multipurpose scientific/philosophical treatises with illustrations or poems (my favorite being de Rerum Natura, though you could argue that Nietzsche and Wagner tried to revive this with their attempts at Gesamtkunstwerke).

related: https://forum.effectivealtruism.org/posts/wAbYKBzEJBJpHMrDh/book-rec-the-war-with-the-newts-as-ea-fiction

I'm managing to get verve and probity, but having issues with wiles

I really liked the post - I was confused by the meaning and purpose no-coincidence principle when I was a ARC, and this post clarifies it well. I like that this is asking for something that is weaker than a proof (or a probabilistic weakening of proof), as [related to the example of using the Riemann hypothesis], in general you expect from incompleteness for there to be true results that lead to "surprising" families of circuits which are not provable by logic. I can also see Paul's point of how this statement is sort of like P vs. BPP but not quite.

More specifically, this feels like a sort of 2nd-order boolean/polynomial hierarchy statement whose first-order version is P vs. BPP. Are there analogues of this for other orders?

Looks like a conspiracy of pigeons posing as lw commenters have downvoted your post

Thanks!

I haven't grokked your loss scales explanation (the "interpretability insights" section) without reading your other post though.

Not saying anything deep here. The point is just that you might have two cartoon pictures:

1. every correctly classified input is either the result of a memorizing circuit or of a single coherent generalizing circuit behavior. If you remove a single generalizing circuit, your accuracy will degrade additively.
2. a correctly classified input is the result of a "combined" circuit consisting of multiple parallel generalizing "subprocesses" giving independent predictions, and if you remove any of these subprocesses, your accuracy will degrade multiplicatively.

A lot of ML work only thinks about picture #1 (which is the natural picture to look at if you only have one generalizing circuit and every other circuit is a memorization). But the thing I'm saying is that picture #2 also occurs, and in some sense is "the info-theoretic default" (though both occur simultaneously -- this is also related to the ideas in this post)