Abhimanyu Pallavi Sudhir

Abhimanyu Pallavi Sudhir1mo*Quick Take

Here are some of my claims about AI welfare (and sentience in general):

"Utility functions" are basically internal models of reward that are learned by agents as part of modeling the environment. Reward attaches values to every instrumental thing and action in the world, which may be understood as gradients dU/dx of an abstract utility over each thing---these are various pressures, inclinations, and fears (or "shadow prices") while U is the actual experienced pleasure/pain that must exist in order to justify belief in these gradients.
If the agent always acts to maximize its utility---what then are pleasure/pain? Pleasure must be the default, and pain only a fear, a shadow price of what would happen

... (read 575 more words →)

-2

Replying toWe Don't Know Our Own Values, but Reward Bridges The Is-Ought Gap

Abhimanyu Pallavi Sudhir1mo

We Don't Know Our Own Values, but Reward Bridges The Is-Ought Gap

How does this work with agents that are composed of sub-agents (thus whose "values" are a composite of the sub-agents)?

Replying toTowards a scale-free theory of intelligent agency

Abhimanyu Pallavi Sudhir1mo

Towards a scale-free theory of intelligent agency

Ah right got it. But why wouldn't they just agree to merge to the EUM represented by the tangent to that flat region itself?

Replying toTowards a scale-free theory of intelligent agency

Abhimanyu Pallavi Sudhir1mo

Towards a scale-free theory of intelligent agency

I like this post and agree with the idea of agency/intelligence emerging coalitionally. However, I disagree with the technical points:

I don't agree with the thesis that "coalitional agents are incentive-compatible decision procedures". Any mechanism is trivially equivalent to an incentive-compatible one by simply counting the agents' decision-making procedures as part of the mechanism (revelation principle)---so it seems to me that the statement is vacuous in the forward direction, and false in the reverse. I can understand the intuition that "if everyone is lying to one another, they don't form a coalitional agent"---however, this is rather best captured in terms of transaction costs (see below).
Your dismissal of linear-weights social welfare functions seems misguided,

... (read more)

Replying toTotal utilitarianism is fine

Abhimanyu Pallavi Sudhir1mo

Total utilitarianism is fine

We're talking about outcomes, not mechanisms. Of course you have to design a mechanism that actually achieves a Pareto-optimal outcome/maximizes total utility---nobody argues that "just ask people to report their utilities" is the mechanism to do this. This remains the same whether for total utilitarianism or geometric rationality or anything else.

E.g. markets (under assumptions of perfect competititon, no transaction costs and no information asymmetry) maximize linear weighted utility.

Total utilitarianism is fine

Abhimanyu Pallavi Sudhir

1mo

Take some agents $A_{i}$ with utility functions $U_{i} (x_{1}, . . . x_{N})$ . In general if they are individually maximizing their utility functions, then their chosen actions $(x_{1}^{*}, \dots x_{N}^{*})$ might be some Nash equilibrium of the game---but it may not be possible to interpret this as the action of a "super-agent".

There are two ways to interpret the words “the action of a super-agent”:

As the action that maximizes some "total utility function", where this total utility function has some sensible properties---say, a linear sum (with positive weights) of agent utilities.
If the two agents could co-ordinate and choose their action, there is no other action that would be better for both of them---i.e. is Pareto-optimal.

(a) is always (b): if $\sum w_{i} U_{i}$ ... (read 891 more words →)

Replying toTowards a scale-free theory of intelligent agency

Abhimanyu Pallavi Sudhir1mo

Towards a scale-free theory of intelligent agency

I agree, but this seems wrong:

If the point they want to hit is in a flat region of the frontier, the merge will involve coinflips to choose which EUM agent to become; and if it's curvy at that point, the merge will be deterministic.

The only time the merge will involve coinflips is if there are multiple tangent lines at that point---then the weights of any tangent line can be the weights of the EUM. Maybe you meant the reverse: if the frontier is flat at the point, then the merged EUM agent is indifferent to any of the points on that flat bit.

Abhimanyu Pallavi Sudhir5moQuick Take

Thoughts are things occurring in some mental model (this is a vague sentence but just assume it makes sense). Some of these mental models are strongly rooted in reality (e.g. the mental model we see as reality) and so we have a high degree of confidence about their accuracy. But for things like introspection, we do not have a reliable ground-truth feedback to tell us if our introspection is correct or not—it's just our mental model of our mind, there is no literal "mind's eye".

So often our introspection is wrong. E.g. if you ask someone to visualize a lion from behind, they'll say they can, but if you ask them some details,... (read more)

Replying toProposal for making credible commitments to AIs.

Abhimanyu Pallavi Sudhir7mo

Proposal for making credible commitments to AIs.

I'm saying the expected value of their best non-compliant option of a sufficiently advanced AI will always be far far greater by the expected value of their best compliant action.

Replying toProposal for making credible commitments to AIs.

Abhimanyu Pallavi Sudhir7mo

Proposal for making credible commitments to AIs.

I don't really understand what problem this is solving. In my view the hard problems here are:

how do you define legal personhood for an entity without a typical notion of self/personhood (i.e. what Mitchell Porter said) or interests
how do you ensure the AIs keep their promise in a world where they can profit far more from breaking the contract than from whatever we offer them

Once you assume away the former problem and disregard the latter, you are of course only left with basic practical legal questions ...

Abhimanyu Pallavi Sudhir8mo

matter of taste for fiction; but objectively bad for technical writing

-6

An unfortunate thing about headings is that they are spoilers. I like the idea of a writing style where headings come at the end of sections rather than at the start. Or even a "starting heading" which is a motivating question and an "ending heading" which is the key insight discovered ...

Analogous to a "reverse mathematics" style of writing where motivation precedes proofs/theory precede theorems.

edited to clarify: I'm talking about technical writing; I don't care about fiction.

homomorphisms and entropy

One informal way to think of homomorphisms in math is that they are maps that do not "create information out of thin air". Isomorphisms further do not destroy information. The terminal object (e.g. the trivial group, the singleton topological space, or the trivial vector space) is the "highest-entropy state", where all distinctions disappear and reaching it is heat death.

Take, for instance the group homomorphism $ϕ : Z^{+} \to Z_{4}^{+}$ . Before $ϕ$ was applied, "1" and "5" were distinguished: 2 + 3 = 5 was correct, but 2 + 3 = 1 was wrong. Upon applying this homomorphism, this information disappears --- however, no new information has been created, that is: no true indinstinctions (equalities) have become false.
Similarly in topology, "indistinction" is "arbitrary closeness". Wiggle-room (aka "open

... (read 406 more words →)

Jevon's paradox and economic intuitions

Abhimanyu Pallavi Sudhir

Recent discourse surrounding Jevon’s paradox provides an opportunity to flesh out the precise intuitions behind marginalist economics and how concepts like “elasticity” arise naturally from agent/graph-based first principles.

Say you have resource $x$ (e.g. compute) with cost $C (x)$ that produces good $y$ (e.g. AI), with utility function $U (x)$ via production function $y = f (x)$ .

The market maximizes $U (f (x)) - C (x)$ — note that the derivatives $U^{'} (f (x)) f^{'} (x)$ and $C^{'} (x)$ are exactly the MB and MC curve for $x$ , so this is just intersecting demand and supply for $x$ .

Here is the explanation on Wikipedia and other places … this is not actually wrong (as we’ll see), but if you think about it, it really looks wrong!

Contrary to what the above... (read 214 more words →)

Articles (or writing in general) is probably best structured as a Directed Acyclic Graph, rather than linearly. At each point in the article, there may be multiple possible lines to pursue, or "sidenotes".

I say "directed acyclic graph" rather than "tree", because it may be natural as thinking of paths as joining back at some point, especially if certain threads are optional.

One may also construct an "And-Or tree" to allow multiple versions of the article preferred by conflicting writers, which may then be voted on with some mechanism. These votes can be used to define values to each vertex, and people can read the tree with their own search algorithm*.

A whole wiki may... (read more)

The third virtue of rationality, lightness, is wrong. In fact: the more you value information to change your mind on some question, the more obstinate you should be to changing your mind on that question. Lightness implies disinterest in the question.

Imagine your mind as a logarithmic market-maker which assigns some initial subsidy to any new question $Q$ . This subsidy parameter captures your marginal value for information on $Q$ . But it also measures how hard it is to change your mind — the cost of moving your probability from $p$ to $p^{'}$ is $b min [log \frac{1 - p}{1 - p^{'}}, log \frac{p}{p^{'}}]$ .

What would this imply in practice? It means that each individual “trader” (both internal mental heuristics/thought patterns, and external sources of information/other people) will generally have a smaller influence on your beliefs, as they may not have enough wealth. Traders who influence your belief will carry greater risk (to their influence on you in future), though will also earn more reward if they’re right.

Ways to think about alignment

Abhimanyu Pallavi Sudhir

I’m listing some “ways to think about alignment”. I’m not sure how much of the “whole problem” each of these individually captures, but they are intended to be very well-motivated, fundamental problems which can:

demonstrate to an honest skeptic that there is “something here”, it’s not just hand-wavy sci-fi BS
be used to sanity-check an alignment proposal by asking “well, which of these does it solve?”

(Bleg: tell me how much of the whole problem each individual thing captures, what it leaves out, and what things are really equivalent.)

I have three, maybe four:

A. The problem of buying under information asymmetry.

The basic problem with RLHF is that it fundamentally relies on a human’s ability to judge... (read 983 more words →)

Something that seems like it should be well-known, but I have not seen an explicit reference for:

Goodhart’s law can, in principle, be overcome via adversarial training (or generally learning Multi-Agent Systems)

—aka “The enemy is smart.”

Goodhart’s law only really applies to a “static” objective, not when the objective is the outcome of a game with other agents who can adapt.

This doesn’t really require the other agents to act in a way that continuously “improves” the training objective either, it just requires them to be able to constantly throw adversarial examples to the agent forcing it to “generalize”.

In particular, I think this is the basic reason why any reasonable Scalable Oversight protocol would be fundamentally “multi-agent” in nature (like Debate).

Reinforcement Learning from Information Bazaar Feedback, and other uses of information markets

Abhimanyu Pallavi Sudhir

Markets for information are inefficient, in large part due to the Buyer’s Inspection Paradox: you can’t “inspect” information like you would any other good before buying — the moment you inspect the information, you have obtained it and cannot return it. More generally, the problem is an inability to reliably commit to forgetting.

A comment by John Wentworth mentioned that you can use amnestics to overcome this: make your purchase decision while under the influence of an amnestic that blocks your mind’s ability to write to long-term memory.

When you read this, it’s hard not to immediately think of LLMs, which can make purchase decisions without committing anything to long-term memory.^[1] Apparently the authors... (read 1416 more words →)

There is a cliche that there are two types of mathematicians: "theory developers" and "problem solvers". Similarly Dyson’s “birds and frogs”, and Robin Hanson divides the production of knowledge into "framing" and "filling".

It seems to me there are actually three sorts of information in the world:

"Ideas": math/science theories and models, inventions, business ideas, solutions to open-ended problems
"Answers": math theorems, experimental observations, results of computations
"Proofs": math proofs, arguments, evidence, digital signatures, certifications, reputations, signalling

From a strictly Bayesian perspective, there seems to be no "fundamental" difference between these forms of information. They're all just things you condition your prior on. Yet this division seems to be natural in quite a variety of informational tasks.... (read more)

•••

Just realized in logarithmic market scoring the net number of stocks is basically just log-odds, lol:

$⟺ p_{i} = e^{x_{i}} / (e^{x_{i}} + 1)$

Inference-Only Debate Experiments Using Math Problems

Arjun Panickssery

Arjun Panickssery, Abhimanyu Pallavi Sudhir, JacksonKaunismaa

Work supported by MATS and SPAR. Code at https://github.com/ArjunPanickssery/math_problems_debate/.

Three measures for evaluating debate are

whether the debate judge outperforms a naive-judge baseline where the naive judge answers questions without hearing any debate arguments.
whether the debate judge outperforms a consultancy baseline where the judge hears argument(s) from a single "consultant" assigned to argue for a random answer.
whether the judge can continue to supervise the debaters as the debaters are optimized for persuasiveness. We can measure whether judge accuracy increases as the debaters vary in persuasiveness (measured with Elo ratings). This variation in persuasiveness can come from choosing different models, choosing the best of N sampled arguments for different values of N, or training debaters

... (read 496 more words →)

I don't think that AI alignment people doing "enemy of enemy is friend" logic with AI luddites (i.e. people worried about Privacy/Racism/Artists/Misinformation/Jobs/Whatever) is useful.

Alignment research is a luxury good for labs, which means it would be the first thing axed (hyperbolically speaking) if you imposed generic hurdles/costs on their revenue, or if you made them spend on mitigating P/R/A/M/J/W problems.

This "crowding-out" effect is already happening to a very large extent: there are vastly more researchers and capital being devoted to P/R/A/M/J/W problems, which could have been allocated to actual alignment research! If you are forming a "coalition" with these people, you are getting a very shitty deal -- they've been much more effective at getting their priorities funded than you have been!

If you want them to care about notkilleveryoneism, you have to specifically make it expensive for them to kill everyone, not just untargetedly "oppose" them. E.g. like foom liability.

The Kernel of Meaning in Property Rights

Abhimanyu Pallavi Sudhir

A phenomenon I have often encountered when thinking about things is when everything seems to collapse to tautology. This is hard to define precisely, but I’ll give you some examples:

Bounded rationality: Bounded rationality can be thought of as “rationality conditional on some given algorithmic information” (in contrast to perfect Bayes-rationality, which is “rationality conditional on some given statistical information”), where “algorithmic information” is a more general version of “logical information”. But “algorithmic information” is fundamentally just arbitrary programs/heuristics, including the heuristic that estimates the argmax of the utility. So everything seems to unwind, and you could as well just say every program is boundedly rational — it’s “doing the best it can

... (read 544 more words →)

LMSR subsidy parameter is the price of information

Abhimanyu Pallavi Sudhir

A logarithmic scoring rule to elicit a probability distribution $r$ on a random variable $X \in {1 \dots n}$ is $s (r) = b log (r_{X})$ . Something that always seemed clear to me but I haven’t seen explicitly written anywhere is that the parameter $b$ is just the price of information on $X$ .

Firstly: for an agent with true belief $p$ , the expected score from making a report $r$ is $E_{p} [s (r)] = \sum_{x \in {1 \dots n}} b log (r_{x}) p_{x} = - b H (p, r)$ where $H$ is cross-entropy. This is maximized when $r = p$ .

Well, this is just the standard proof that logarithmic scoring is proper. This max score itself is $E_{p} [s (p)] = - b H (p)$ i.e. the entropy in $p$ . So your expected earning is exactly proportional to the information you have on $X$ (the negative of the entropy in... (read 296 more words →)

Abhimanyu Pallavi Sudhir's Shortform

Abhimanyu Pallavi Sudhir

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

Betting on what is un-falsifiable and un-verifiable

Abhimanyu Pallavi Sudhir

This post is part of my broader aim to try and formulate everything AI in terms of markets so that alignment becomes an antitrust problem. Comments appreciated.

Prerequisites^[1].

This post is also available as an arXiv paper.

introduction: betting on what is neither verifiable nor falsifiable (non-VF)

Prediction markets are good at eliciting probabilities about events that will occur at a fixed, finite time. However, there are many questions we would like to bet on that cannot be resolved in finite time. For example:

( $Σ_{1}$ ) an asteroid impact will eventually occur; I am mortal; P halts
( $Π_{1}$ ) an asteroid impact will never occur; I am immortal; P doesn't halt
( $Σ_{2}$ ) there will exist an immortal man; some P_n doesn't

... (read 4292 more words →)

Meaningful things are those the universe possesses a semantics for

Abhimanyu Pallavi Sudhir

Logic courses are very risk-averse to talking about philosophy. This is bad, because philosophy is the motivation/intuition for logic & TCS. This essay gives a scout's view.

Epistemic status: A computer scientist would nod through Chapters 2-5, at least after first looking at the logician for approval. Subsequent chapters take a philosophical stance, namely that expressed in the title.

Contents

Motivation and a cringey rant [skip for the technical meat]
Gödel's first incompleteness theorem
Gödel's second incompleteness theorem
Semantics and truth
Exercises
Reflection and ordinals
Chaitin and complexity
Empiricism and the Löbstacle

[A rough version of Chs 1-5 post initially appeared as a math stackexchange answer and on my blog; I wrote it as I learned, and am posting more refined writing here.]

Motivation... (read 4033 more words →)

LESSWRONG
LW

LESSWRONG
LW

Abhimanyu Pallavi Sudhir

Inference-Only Debate Experiments Using Math Problems

Meaningful things are those the universe possesses a semantics for

Betting on what is un-falsifiable and un-verifiable

Political Roko's basilisk

Abhimanyu Pallavi Sudhir

Total utilitarianism is fine

Jevon's paradox and economic intuitions

Ways to think about alignment

Reinforcement Learning from Information Bazaar Feedback, and other uses of information markets

Inference-Only Debate Experiments Using Math Problems

The Kernel of Meaning in Property Rights

LMSR subsidy parameter is the price of information

Abhimanyu Pallavi Sudhir

Inference-Only Debate Experiments Using Math Problems

Meaningful things are those the universe possesses a semantics for

Betting on what is un-falsifiable and un-verifiable

Political Roko's basilisk

Abhimanyu Pallavi Sudhir

Total utilitarianism is fine

Jevon's paradox and economic intuitions

Ways to think about alignment

Reinforcement Learning from Information Bazaar Feedback, and other uses of information markets

Inference-Only Debate Experiments Using Math Problems

The Kernel of Meaning in Property Rights

LMSR subsidy parameter is the price of information

homomorphisms and entropy

A. The problem of buying under information asymmetry.

introduction: betting on what is neither verifiable nor falsifiable (non-VF)