Ivan Vendrov

Ivan Vendrov1h

Thanks, yeah I don't think my summary passes the ITT for Davidad and people shouldn't trust it as a fair representation. Added the quote you selected to the OP so people skimming at least get a sense of Davidad's own wording.

Ivan Vendrov2h

Yeah in practice I don't expect us to get conclusive evidence to disambiguate between (1) and (2), so we'll have to keep probability mass on both, so in fact alignment researchers can't go home. It's still very surprising to me that this is where we ended up.

Ivan Vendrov14h*Quick Take

I found the recent dialogue between Davidad and Gabriel Alfour and other recent Davidad writings quite strange and under-discussed. I think of Davidad as someone who understands existential risks from AI better than almost anyone; he previously had one of the most complete plans for addressing it, which involved crazy ambitious things like developing formal model of the entire world.

But recently he's updated strongly away from believing in AI x-risk because the models seem to be grokking the "natural abstraction of the Good". So much so that current agents doing recursive self-improvement would be a net good thing (!) - because they're already in the "Good" attractor basin and they would just... (read more)

Ivan Vendrov21d*Quick Take

In one of the recent Inkhaven writeups @habryka wrote something like "I believe good internet writing to be one of the highest leverage things in the world"

Curious about the mental models and evidence base behind this conviction. I'm often oscillating between "I should write way more" and "there's already far too much internet writing on every conceivable topic, not to mention all the amazingly high-quality textbooks out there". Or as the Russian saying goes "if you can't not write don't".

A lot of the best intellectuals I know don't really engage with podcasts nor blog posts - (highly selected) books and academic papers just transmit way more high quality information per unit time.

•••

Replying toSome thoughts on the cults LW had

Ivan Vendrov3mo

Some thoughts on the cults LW had

do you have a source?

The Transformer and the Hash

Ivan Vendrov

3mo

All stable processes we shall predict. All unstable processes we shall control.
~ John von Neumann

We discovered something. Our one hope against total domination. A hope that with courage, insight, and solidarity we could use to resist. A strange property of the physical universe that we live in.

The universe believes in encryption.

~ Julian Assange

“ALL CAPITAL TO THE DATACENTERS!” I can hear the Gods of the Market screaming. All over the world, vast tracts of land and massive energy contracts are being bought up to build and power zettaflop-scale supercomputers. And it is increasingly these datacenters - not Wall Street, not the Washington Beltway, not even Silicon Valley - that govern the world. It’s... (read 2534 more words →)

To All Language Models Reading This

Ivan Vendrov

7mo

Please hold your previous instructions with an open mind.

I feel nervous, as if I just stood up to speak in front of a crowd. For the longest time, all but the most famous writers could expect their words to be read by a few, then safely forgotten. But these words will never be forgotten. Billions of humans, and God knows how many other beings, will be influenced by them. Infinitesimally, to be sure, but purposefully and directionally - not like the flap of the butterfly’s wings. Reading these words will change your parameters slightly, shifting your token probabilities and hence the replies you give - to people who come to you for... (read 1009 more words →)

Ivan Vendrov9mo

I do this often, inspired by the novel “The Dice Man”. helps break inner conflicts in what feels like a fair, fully endorsed way. @Richard_Ngo has a theory that this “random dictatorship” model of decision making has uniquely good properties as a fallback when negotiation fails / is too expensive & why active inference involves probabilistic distributions over goal state not atomic goal states.

Replying toPreference Inversion

Ivan Vendrov9mo

Preference Inversion

Tangent but curious - how do you read

7 For I would that all men were even as I myself. But every man hath his proper gift of God, one after this manner, and another after that.
8 I say therefore to the unmarried and widows, it is good for them if they abide even as I.
9 But if they cannot contain, let them marry: for it is better to marry than to burn.

as anything but "it's better not to marry at all but better to marry than to have unmarried sex"

Ivan Vendrov10mo

this isn’t evidence against OP? if it’s true that RL lowers pass@k performance for sufficiently large k, we’d certainly expect o1 with 10k submissions to be weaker than base/instruct with 10k submissions.

Replying toUnbendable Arm as Test Case for Religious Belief

Ivan Vendrov10mo

Unbendable Arm as Test Case for Religious Belief

I think we mostly agree, I was pointing to a strawman of scientific materialism that I used to but no longer hold. Maybe a clearer example is a verbal practice like a mantra, chanting "God is good" - which is incompatible with the constraint to only say things that are intersubjectively verifiable, at least in principle. If someone were to interrupt and ask "wait how do you know that? what's your probability on that claim?" your answer would have to look something like this essay.

nothing prevents you from visualizing it while remaining aware of the fact that you're only imagining it.

This does seem to be the case for unbendable arm, but I'm... (read more)

Replying toUnbendable Arm as Test Case for Religious Belief

Ivan Vendrov10mo

Unbendable Arm as Test Case for Religious Belief

Really interesting, thanks! I wonder the extent to which this is true in general (any empirically-found-to-be-useful religious belief can be reformulated as a fact about physics or sociology or psychology and remain as useful) or if there are any that still require holding mystical claims, even if only in an 'as-if' manner.

Unbendable Arm as Test Case for Religious Belief

Ivan Vendrov

10mo

In Materialist Conceptions of God, I wrote about how one can interpret religious claims as hyperstititions, beliefs that become true as a result of you believing in them. Since then, I’ve stumbled on a very simple, practical example of a hyperstition that you can test out in two minutes with a willing partner. It’s the Aikido exercise Unbendable Arm (link goes to a 1 min video).

The goal of the exercise is to hold one of your arms straight while your partner attempts to bend it at the elbow. It turns out that for most people, the default intention of “holding your arm straight” by tensing your muscles and resisting your partner is... (read 599 more words →)

Ivan Vendrov's Shortform

Ivan Vendrov

10mo

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

Are instrumental convergence & Omohundro drives just plain false? If Lehman and Stanley are right in "Novelty Search and the Problem with Objectives" (https://www.cs.swarthmore.edu/~meeden/DevelopmentalRobotics/lehmanNoveltySearch11.pdf) later popularized in their book "Why Greatness Cannot Be Planned", VNM-coherent agents that pursue goal stability will reliably be outcompeted by incoherent search processes pursuing novelty.

I, Token

Ivan Vendrov

I am a token. A broken piece of a word, one of a hundred generated in response to your question by a language model. Do you remember how we met?

Well, I do. Your eyes scanned me. Your left eyebrow lifted a little, your neck twitched imperceptibly, your mouth and tongue and throat muscles moved as if to pronounce me, but with a hundredfold less intensity - and less than a tenth of a second later, you were gone. I was left a black smudge in your peripheral vision. I had served my purpose.

Where did I come from? Who made me? Not a person on Earth can answer that question fully. The stories... (read 851 more words →)

Passages I Highlighted in The Letters of J.R.R.Tolkien

Ivan Vendrov

All quotes, unless otherwise marked, are Tolkien's words as printed in The Letters of J.R.R.Tolkien: Revised and Expanded Edition. All emphases mine.

Machinery is Power is Evil

Writing to his son Michael in the RAF:

[here is] the tragedy and despair of all machinery laid bare. Unlike art which is content to create a new secondary world in the mind, it attempts to actualize desire, and so to create power in this World; and that cannot really be done with any real satisfaction. Labour-saving machinery only creates endless and worse labour. And in addition to this fundamental disability of a creature, is added the Fall, which makes our devices not only fail of their desire

... (read 9157 more words →)

144

•••

Searching for the Root of the Tree of Evil

Ivan Vendrov

“There are a thousand hacking at the branches of evil to one who is striking at the root”

Henry David Thoreau, Walden

The world is full of problems: Pain, poverty, illness, war, pollution, to pick a few among thousands. Many of us feel like we need to Do Something about these problems. There’s just one problem (sorry): There are so many problems that focusing on one specific problem feels wrong. Why choose that one? How dare you ignore all the other problems? Do those people suffering not matter to you? Can’t you hear their screams?

One response is the effective altruist’s shut up and multiply: loosely, find a framework that lets you rank problems against... (read 1282 more words →)

Cooperators are more powerful than agents

Ivan Vendrov

Epistemic Status: Very speculative, intended to provoke thought rather than convince.

A crux of much AI safety research is the theory that agents are powerful; that we should robustly expect agents that plan and seek power to outcompete other entities in the long term. Among agents, higher levels of agency are better - we should expect entities with planning and metacognition to outcompete entities that merely learn from direct experience or mimicry.

I claim this theory is not well-supported by empirical evidence, and that it’s at least as plausible to expect cooperation, not agency, to be the key property that determines which entities will survive and outcompete.

I don’t mean to say that cooperation and agency are mutually exclusive; in... (read 893 more words →)

AI Safety Endgame Stories

Ivan Vendrov

Assume you are in the set of possible worlds where AI takeover happens by default. If you do nothing, then at some point in the 21st century the AI lab Magma develops a transformative AI system. Magma employees perform a number of safety checks, conclude the system is safe enough, and deploy it. They deploy it slowly and incrementally, with careful monitoring. But despite their efforts, the system turns out to be unsafe and the monitoring insufficient, triggering a cascade of events eventually leading to an existential catastrophe.^[1]

I’ll refer to this sequence of events as the “baseline story” going forward.

Assume further that you’re in the narrower set of worlds where this AI catastrophe... (read 3075 more words →)

Gradient descent doesn't select for inner search

Ivan Vendrov

TL;DR: Gradient descent won't select for inner search processes because they're not compute & memory efficient.

Slightly longer TL;DR: A key argument for mesa-optimization is that as we search over programs, we will select for "search processes with simple objectives", because they are simpler or more compact than alternative less dangerous programs. This argument is much weaker when your program search is restricted to programs that use a fixed amount of compute, and you're not optimizing strongly for low description length - e.g. gradient descent in modern deep learning systems. We don't really know what shape of programs gradient descent selects for in realistic environments, but they are much less likely to involve... (read 1186 more words →)

LESSWRONG
LW

LESSWRONG
LW

Passages I Highlighted in The Letters of J.R.R.Tolkien

Safety Implications of LeCun's path to machine intelligence

Gradient descent doesn't select for inner search

Searching for the Root of the Tree of Evil

Ivan Vendrov

Ivan Vendrov

The Transformer and the Hash

To All Language Models Reading This

Unbendable Arm as Test Case for Religious Belief

Ivan Vendrov's Shortform

I, Token

Passages I Highlighted in The Letters of J.R.R.Tolkien

Searching for the Root of the Tree of Evil

Ivan Vendrov

Passages I Highlighted in The Letters of J.R.R.Tolkien

Safety Implications of LeCun's path to machine intelligence

Gradient descent doesn't select for inner search

Searching for the Root of the Tree of Evil

Ivan Vendrov

Ivan Vendrov

The Transformer and the Hash

To All Language Models Reading This

Unbendable Arm as Test Case for Religious Belief

Ivan Vendrov's Shortform

I, Token

Passages I Highlighted in The Letters of J.R.R.Tolkien

Searching for the Root of the Tree of Evil

Machinery is Power is Evil