All of Jameson Quinn's Comments + Replies

Doesn't matter until the switch is done.

I am into something that can be called "meta-politics": institutional reform. That is, crafting decisionmaking algorithms to have good characteristics — incentives, participation, etc. — independent of the object-level goals of politics. I think this is "meta" in a different way than what you're talking about in this article; in short, it's prescriptive meta, not descriptive meta. And I think that makes it "OK"; that is, largely exempt from the criticisms in this article.

Would you agree?

1sleno
Hi Jameson, I'm interested in the "meta-politics" you describe. Is there a field/domain where these things are explored in a more rigorous way? The only thing I'm aware of that comes close is the crypto/blockchain space.
6Rob Bensinger
Yep. Doesn't seem to pose the kinds of risks I talked about.
2ryan_b
I don't speak for Rob, but my guess is that your work still qualifies as object level because it is directly about the mechanisms of voting and how those votes are counted. In other words, your work is about voting policy.

I believe that Bitcoin is a substantial net negative for the world. I think that blockchain itself, even without proof of work, is problematic as a concept — with some real potential upsides, but also real possibly-intrinsic downsides even apart from proof of work. I'd like a world where all PoW-centric cryptocurrency was not a thing (with possible room for PoW as a minor ingredient for things like initial bootstrapping), and crypto in general was more an area of research than investment for now. I think that as long as >>90% of crypto is PoW, it's better (for me, at least) to stay away entirely rather than trying to invest in some upstart PoS coin.

3sapphire
ETH is switching off PoW. Once ETh swaps crypto will be less than 90% PoW. 

#2. Note that even if ETH does switch in the future, investing in ETH today is still investing in proof-of-work. Also, as long as BTC remains larger and doesn't switch, I suspect there's likely to be spillover between ETH and BTC such that it would be difficult to put energy into ETH without to some degree propping up the BTC ecosystem.

2ChristianKl
Even if you don't believe in Ethereum you could invest in Polkadot that already has proof-of-stake. I would estimate that investments that challenge Bitcoin as the primary crypto-currency will over the long run reduce electricity consumption..

I feel it's worth pointing out that all proof-of-work cryptocurrency is based on literally burning use-value to create exchange-value, and that this is not a sustainable long-term plan. And as far as I can tell, non-proof-of-work cryptocurrency is mostly a mirage or even a deliberate red herring / bait-and-switch.

I'm not an expert, but I choose not to participate on moral grounds. YMMV.

I realize that what I'm saying here is probably not a new idea to most people reading, but it seems clearly enough true to me that it bears repeating anyway.

If anyone wants links to further arguments in this regard, from me rather than Google, I'd be happy to provide.

6Srdjan Miletic
Why do you believe that proof-of-stake is a mirage? We know it's possible as some existing blockchains already use it. Do you believe that: * It's possible but has some serious flaw that most people don't recognize * The main crypto's of today (ETH + BTC) won't transition to it * Something else
3ChristianKl
Polkadot has working proof-of-stake. Polkadot has a setup where it can run many more transactions then Ethereum can with the current 1.0 setup. If you don't believe that Ethereum will manage the transition to Ethereum 2.0 then it would be likely that Polkadot overtakes Ethereum for DeFi. Filecoin has a type of proof-of-work but it's work that stores files with is economically useful.
1Archimedes
>And as far as I can tell, non-proof-of-work cryptocurrency is mostly a mirage or even a deliberate red herring / bait-and-switch. How familiar are you with Ethereum's plan to transition to Proof of Stake? It's definitely not there yet but making (slow) progress.

If we're positing a Grahamputer, then "yeah but it's essentially the same if you're not worried about agents of equal size" seems too loose. 

In other words, with great compute power, comes great compute responsibility.

Thanks for pointing that out. My arguments above do not apply.

I'm still skeptical. I buy anthropic reasoning as valid in cases where we share an observation across subjects and time (eg, "we live on a planet orbiting a G2V-type star", "we inhabit a universe that appears to run on quantum mechanics"), but not in cases where each observation is unique (eg, "it's the year 2021, and there have been about 107,123,456,789 (plus or minus a lot) people like me ever"). I am far less confident of this than I stated for the arguments above, but I'm still reasonably confident, and my expertise does still apply (I've thought about it more than just what you see here).

4cubefox
This could mean you would also have to reject thirding in the famous Sleeping Beauty problem. Which contradicts a straightforward frequentist interpretation of the setup: If the SB experiment was repeated many times, one third of the awakenings would be Monday Heads, so if SB was guessing after awakening "the coin came up heads" she would be right with frequentist probability 1/3. Of course there are possible responses to this. My point is just that: rejecting Katja's doomsday argument by rejecting SIA style anthropic reasoning may come with implausible consequences in other areas.

Our sense-experiences are "unitary" (in some sense which I hope we can agree on without defining rigorously), so of course we use unitary measure to predict them. Branching worlds are not unitary in that sense, so carrying over unitarity from the former to the latter seems an entirely arbitrary assumption.

A finite number (say, the number of particles in the known universe), raised to a finite number (say, the number of Planck time intervals before dark energy tears the universe apart), gives a finite number. No need for divergence. (I think both of those are severe overestimates for the actual possible branching, but they are reasonable as handwavy demonstrations of the existence of finite upper bounds)

3interstice
Ah, by 'unitary' I mean a unitary operator, that is an operator which preserves the Hilbert measure. It's an axiom of quantum mechanics that time evolution is represented by a unitary operator. Fair point about the probable finitude of time(but wouldn't it be better if our theory could handle the possibility of infinite time as well?)

I don't think the point you were arguing against is the same as the one I'm making here, though I understand why you think so.

My understanding of your model is that, simplifying relativistic issues so that "simultaneous" has a single unambiguous meaning, total measure across quantum branches of a simultaneous time slice is preserved; and your argument is that, otherwise, we'd have to assign equal measure to each unique moment of consciousness, which would lead to ridiculous "Bolzmann brain" scenarios. I'd agree that your argument is convincing that different simultaneous branches have different weight according to the rules of QM, but that does not at all imply that total weight across branches is constant across time.

1interstice
The argument I made there was that we should consider observer-moments to be 'real' according to their Hilbert measure, since that is what we use to predict our own sense-experiences. This does imply that observer-weight will be preserved over time, since unitary evolution preserves the measure(as you say, this also proves it is conserved by splitting into branches, since you can consider that to be projecting onto different subspaces) Even without unitarity, you shouldn't expect the total amount of observer-weight to increase exponentially in time, since that would cause the total amount of observer-weight to diverge, giving undefined predictions.

I didn't do this problem, but I can imagine I might have been tripped up by the fact that "hammer" and "axe" are tools and not weapons. In standard DnD terminology, these are often considered "simple weapons"; distinct from "martial weapons" like warhammer and battleaxe, but still within the category of "weapons".

I guess that the "toolish" abstractions might have tipped me off, though. And even if I had made this mistake, it would only have mattered for "simple-weapon" tools with a modifier.

4SarahNibs
It was very obvious upon filtering to just-hammer or just-axe that the abstractions were suspiciously limited.

This is certainly a cogent counterargument. Either side of this debate relies on a theory of "measure of consciousness" that is, as far as I can tell, not obviously self-contradictory. We won't work out the details here.

In other words: this is a point on which I think we can respectfully agree to disagree.

1interstice
Fair, although I do think your theory might be ultimately self-contradictory ;) Instead or arguing that here, I'll link an identical argument I had somewhere else and let you judge if I was persuasive.

It seems to me that exact duplicate timelines don't "count", but duplicates that split and/or rejoin do. YMMV.

I think both your question and self-response are pertinent. I have nothing to add to either, save a personal intuition that large-scale fully-quantum simulators are probably highly impractical. (I have no particular opinion about partially-quantum simulators — even possibly using quantum subcomponents larger than today's computers — but they wouldn't change the substance of my not-in-a-sim argument.)

3plex
hm, that intuition seems plausible. The other point that comes to mind is that if you have a classical simulation running on a quantum world, maybe that counts as branching for the purposes of where we expect to find ourselves? I'm still somewhat confused about whether exact duplicates 'count', but if they do then maybe the branching factor of the underlying reality carries over to sims running further down the stack?

Yes, your restatement feels to me like a clear improvement.

In fact, considering it, I think that if algorithm A is "truly more intelligent" than algorithm B, I'd expect if f(x) is the compute that it takes for B to perform as well or better than A, f(x) could even be super-exponential in x. Exponential would be the lower bound; what you'd get from a mere incremental improvement in pruning. From this perspective, anything polynomial would be "just implementation", not "real intelligence". 

Though I've posted 3 more-or-less-strong disagreements with this list, I don't want to give the impression that I think it has no merit. Most specifically: I strongly agree that "Institutions could be way better across the board", and I've decided to devote much of my spare cognitive and physical resources to gaining a better handle on that question specifically in regards to democracy and voting.

Third, separate disagreement: This list states that "vastly more is at stake in [existential risks] than in anything else going on". This seems to reflect a model in which "everything else going on" — including power struggles whose overt stakes are much much lower — does not substantially or predictably causally impact outcomes of existential risk questions. I think I disagree with that model, though my confidence in this is far, far less than for the other two disagreements I've posted.

Separate point: I also strongly disagree with the idea that "there's a strong chance we live in a simulation". Any such simulation must be either:

  • fully-quantum, in which case it would require the simulating hardware to be at least as massive as the simulated matter, and probably orders of magitude more massive. The log-odds of being inside such a simulation must therefore be negative by at least those orders of magnitude.
  • not-fully-quantum, in which case the quantum branching factor per time interval is many many many orders of magnitude less than that of a
... (read more)
2interstice
I don't think the branching factor of the simulation matters, since the weight of each individual branch decreases as the number of branches increases. The Born measure is conserved by branching.
8plex
As someone who mostly expects to be in a simulation, this is the clearest and most plausible anti-simulation-hypothesis argument I've seen, thanks. How does it hold up against the point that the universe looks large enough to support a large number of even fully-quantum single-world simulations (with a low-resolution approximation of the rest of reality), even if it costs many orders of magnitude more resources to run them? Perhaps would-be simulators would tend not to value the extra information from full-quantum simulations enough to build many or even any of them? My guess is that many purposes for simulations would want to explore a bunch of the possibility tree, but depending on how costly very large quantum computers are to mature civilizations maybe they'd just get by with a bunch of low-branching factor simulations instead?

Strongly disagree about the "great filter" point.

Any sane understanding of our prior on how many alien civilizations we should have expected to see is structured (or at least, has much of its structure that is) more or less like the Drake equation: a series of terms, each with more or less prior uncertainty around it, that multiply together to get an outcome. Furthermore, that point is, to some degree, fractal; the terms themselves can be — often and substantially, though not always and completely — understood as the products of sub-terms.

By the Central Li... (read more)

5simon
True, the typical argument for the great silence implying a late filter is weak, because an early filter is not all that a priori implausible.  However, the OP (Katja Grace) specifically mentioned "anthropic reasoning". As she  previously pointed out, an early filter makes our present existence much less probable than a late filter. So, given our current experience , we should weight the probability of a late filter much higher than the prior would be without anthropic considerations.

I'm not sure if this comment goes best here, or in the "Against Strong Bayesianism" post. But I'll put it here, because this is fresher.

I think it's important to be careful when you're taking limits. 

I think it's true that "The policy that would result from a naive implementation of Solomonoff induction followed by expected utility maximization, given infinite computing power, is the ideal policy, in that there is no rational process (even using arbitrarily much computing power) that leads to a policy that beats it." 

But say somebody offered you ... (read more)

6Eliezer Yudkowsky
Seem just false.  If you're not worried about confronting agents of equal size (which is equally a concern for a Solomonoff inductor) then a naive bounded Solomonoff inductor running on a Grahamputer will give you essentially the same result for all practical purposes as a Solomonoff inductor.  That's far more than enough compute to contain our physical universe as a hypothesis.  You don't bother with MCMC on a Grahamputer.
7Donald Hobson
The problem with this definition is that it focusses too much on the details of the computational substrate. Suppose the programming language used has a built in function for matrix multiplication, and it is 2x as fast as any program that could be written within the language. Then any program that does its own matrix multiplication will be less intelligent than one that uses the built in functions.  "A with X resources beats program B with X resources, for any B" could be true if A is just B with the first few steps precomputed. It focusses too much on the little hackish tricks specific to the substrate.   Maybe say that two algorithms A, B are equivalent up to polynomial factors if there exists a polynomial p(x) so that A with p(x) compute beats B with X compute for all x, and likewise B with p(x) compute beats A with x compute.

PLACE is compatible with primaries; primaries would still be used in the US.

Thus, PLACE has all the same (weak) incentives for the local winner to represent any nonpartisan interests of the local district, along with strong incentives to represent the interests of their party X district combo. The extra (weaker) incentives for those other winners who have the district in their territory to represent the interests of their different party X district combos, to fill out the matrix, make PLACE's representation strictly better.

Also worth noting that both AV and FPTP are winner-take-all methods, unlike the proportional methods I discuss here. The AV referendum question was essentially "do you want to take a disruptive half-step that lines you up for maybe, sometime in the future, actually fixing the problem?"; I'm not the only one who believes it was intentionally engineered to fail.

It seems that most of what you're talking about are single-winner reforms (including single-winner pathologies such as center squeeze). In particular, the RCV you're talking about is RCV1, single-winner, while the one I discuss in this article is RCV5, multi-winner; there are important differences. For discussing single-winner, I'd recommend the first two articles linked at the top; this article is about multi-winner reforms.

Personally, I think that the potential benefits of both kinds of reform are huge, but there are some benefits that only multi-winner ... (read more)

Formally speaking, nothing. Indirectly speaking: the candidate is a Schelling point for voters in those districts, especially if they are not excited by the that-party candidate in their own district. So those voters are a potential source of direct votes for that candidate, which help win not just directly, but also by moving the candidate up in the preference order that gets filled in on ballots cast for other candidates.

2ChristianKl
Why would voters in those districts want to gather around a Schelling point? I think both in US elections and in German elections, the primary reason of why it's valuable for a politician to be a good represenative of his districts is not about votes in the main election. In the US a candidate that does badly at local represenation risks getting primaried.  This is do to the combination of for example a major of a city having a lot of insight into whether the interests of their city are well-represented at the national stage by the local representative while at the same time the major has local political power that can be used to draft primary opponents.  Given that being primaried is one of the ways a congressman fails to be reelected it's valuable for them to not antagonize their local political power players and represent them in parliament.  It seems like the assignment has very little power in your system I find it unlikely that representation of a district will work better then in FPTP as you claim. 

This is not an article about the specific circumstances in the US. Suffice it to say that, while you make good points, I stand by my assessment that things are more hopeful for electoral reform in the US some time in the next decade, than they have been in my 25 years of engagement with the issue. That doesn't mean hopes are very high in an absolute sense, but they're high enough to be noticeably higher.

You're right, the sentence you quoted is only a small part of the necessary ingredients for reform; finding a proposal that's minimally disruptive to incumbents (unless they owe their seat to gerrymandering) is key to getting something passed; and even then, it's a heavy lift. 

The 4 methods I chose here are the ones I think have the best chances, from exactly those perspectives. It's still a long shot, but IMO realistic enough to be worth talking about. 

2ChristianKl
In a senate with one vote majority and a congress with nine votes majority you are not going to get bills passed that are problematic for some of the representatives within your party. You need larger majorities to do that. 

You've described, essentially, a weighted-seats closed-list method.

List methods: meh. It's actually possible to be biproportional — that is, to represent both party/faction and geography pretty fairly — so reducing it to just party (and not geography or even faction) is a step down IMO. But you can make reasonable arguments either way.

Closed methods (party, not voters, decides who gets their seats): yuck. Why take power from the people to give it to some party elite?

Weighted methods: who knows, it's scarcely been tried. A few points:

  • If voting weights are t
... (read more)
3Ericf
Why should random people who are not experts in "ability to debate," "ability to read and understand the impact of legal language," or other attributes that make a good lawmaker get to decide which human beings are tasked with the process of writing and compromising on language? People have an interest in having their values reflected, but that's already determined by the party they vote for. This is especially true in a system that encourages multiple parties, so the, for example, "low taxes" faction, the "low regulation" faction, and the "white power" faction can each be separate parties who collaborate (or not) on individual legislative priorities as needed. And each party can hire whatever mix of lawyers, negotiators, speech writers, and speech givers they want, without forcing "the person who decides who to hire," "the person who gives speeches," and "the person who has final say on how to vote" all be the same "candidate."

V0.9.1: Another terminology tweak. New terms: Average Voter Effectiveness, Average Voter Choice, Average Voter Effective Choice. Also, post-mortem will happen in a separate document. (This same version might later be changed to 1.0)

V0.9: I decided to make a few final edits (mostly, adding a summary and a post-mortem section, which is currently unfinished) then freeze this as is.

(This was done some time ago but I forgot to post it here.)

5Jameson Quinn
V0.9.1: Another terminology tweak. New terms: Average Voter Effectiveness, Average Voter Choice, Average Voter Effective Choice. Also, post-mortem will happen in a separate document. (This same version might later be changed to 1.0)

You seem to be comparing Arrow's theorem to Lord Vetinari, implying that both are undisputed sovereigns? If so, I disagree. The part you left out about Arrow's theorem — that it only applies to ranked voting methods (not "systems") — means that its dominion is far more limited than that of the Gibbard-Satterthwaite theorem.

As for the RL-voting paper you cite: thanks, that's interesting. Trying to automate voting strategy is hard; since most voters most of the time are not pivotal, the direct strategic signal for a lea... (read more)

1Sammy Martin
It was a joke about how if you take Arrow's theorem literally, the fairest 'voting method' (at least among ranked voting methods), the only rule which produces a definite transitive preference ranking and which meets the unanimity and independence conditions is 'one man, one vote', i.e. dictatorship. Such a situation doesn't seem all that far-fetched to me - suppose there are three different stimulus bills on offer, and you want some stimulus spending but you also care about rising national debt. You might not care which bills pass, but you still want some stimulus money, but you also don't want all of them to pass because you think the debt would rise too high, so maybe you decide that you just want any 2 out of 3 of them to pass. But I think the methods introduced in that paper might be most useful not to model the outcomes of voting systems, but for attempts to align an AI to multiple people's preferences.

V0.7.3 Still tweaking terminology. Now, Vote Power Fairness, Average Voter Choice, Average Voter Effectiveness. Finished (at least first draft) of closed list/Israel analysis.

5Jameson Quinn
V0.9: I decided to make a few final edits (mostly, adding a summary and a post-mortem section, which is currently unfinished) then freeze this as is. (This was done some time ago but I forgot to post it here.)

V 0.7.2: A terminology change. New terms: Retroactive Power, Effective Voting Equality, Effective Choice, Average Voter Effectiveness. (The term "effective" is a nod to Catherine Helen Spence). The math is the same except for some ultimately-inconsequential changes in when you subtract from 1. Also, started to add a closed list example from Israel; not done yet.

3Jameson Quinn
V0.7.3 Still tweaking terminology. Now, Vote Power Fairness, Average Voter Choice, Average Voter Effectiveness. Finished (at least first draft) of closed list/Israel analysis.

V 0.7.1: added a digression on dimesionality, in italics, to the "Measuring "Representation quality", separate from power" section. Finished converting the existing examples from RF to VW.

5Jameson Quinn
V 0.7.2: A terminology change. New terms: Retroactive Power, Effective Voting Equality, Effective Choice, Average Voter Effectiveness. (The term "effective" is a nod to Catherine Helen Spence). The math is the same except for some ultimately-inconsequential changes in when you subtract from 1. Also, started to add a closed list example from Israel; not done yet.

V 0.7.0: Switched from "Representational Fairness" to the more-interpretable "Vote Wastage". Wrote enough so that it's possible to understand what I mean by VW, but this still needs revision for clarity/convincingness. Also pending, change my calculations for specific methods from RF to VW.

3Jameson Quinn
V 0.7.1: added a digression on dimesionality, in italics, to the "Measuring "Representation quality", separate from power" section. Finished converting the existing examples from RF to VW.

I am rewriting the overall "XXX: a xxx proportionality metric" section because I've thought of a more-interpretable metric. So, where it used to be "Representational fairness: an overall proportionality metric", now it will be "Vote wastage: a combined proportionality metric". Here's the old version, before I erase it:


Since we've structured RQ_d as an "efficiency" — 100% at best, 0% at worst — we can take each voter's "quality-weighted voter power" (QWVP) to be the sum of t... (read more)

V 0.6.0: Coined the term "Representational Fairness" for my metric. Did a worked example of Single transferrable vote (STV), and began to discuss the example. Bumping version because I'm now beginning to actually discuss concrete methods instead of just abstract metrics.

3Jameson Quinn
V 0.7.0: Switched from "Representational Fairness" to the more-interpretable "Vote Wastage". Wrote enough so that it's possible to understand what I mean by VW, but this still needs revision for clarity/convincingness. Also pending, change my calculations for specific methods from RF to VW.

V 0.5.5: wrote a key paragraph about NESS: similar outcomes just before Pascal's Other Wager? (The Problem of Points). Added the obvious normalizing constant so that average voter power is 1. Analyzed some simple plurality cases in Retrospective voting power in single-winner plurality.

3Jameson Quinn
V 0.6.0: Coined the term "Representational Fairness" for my metric. Did a worked example of Single transferrable vote (STV), and began to discuss the example. Bumping version because I'm now beginning to actually discuss concrete methods instead of just abstract metrics.

V 0.5.4: Meaningful rewrite to "Shorter "solution" statement", which focuses not on power to elect an individual, but power to elect some member of a set, of whom only 1 won in reality.

1Jameson Quinn
V 0.5.5: wrote a key paragraph about NESS: similar outcomes just before Pascal's Other Wager? (The Problem of Points). Added the obvious normalizing constant so that average voter power is 1. Analyzed some simple plurality cases in Retrospective voting power in single-winner plurality.

Finding "Z-best" is not the same as finding the posterior over Z, and in fact differs systematically. In particular, because you're not being a real Bayesian, you're not getting the advantage of the Bayesian Occam's Razor, so you'll systematically tend to get lower-entropy-than-optimal (aka more-complex-than-optimal, overfitted) Zs. Adding an entropy-based loss term might help — but then, I'd expect that H already includes entropy-based loss, so this risks double-counting.

The above critique is specific and nitpicky... (read more)

This seems to be unreadably mis-formatted for me in Safari.

4habryka
Should be fixed. In general the AI Alignment Newsletter imports as a really complicated HTML email newsletter that I usually clean up in the first few hours after it's been posted. It's also hard to automatically clean up, though hopefully eventually I will have some sanitization script that can do this at the same time as it goes live.

Thank you.

Bit of trivia on Switzerland and voting methods: I've heard (but have not seen primary sources for) that in 1798 the briefly-independent city-state of Geneva used the median-based voting method we anachronously know as "Bucklin" after its US-based reinventor. This was at the (posthmous) suggestion of the Marquis de Condorcet. Notably that suggestion was not to use what we know of as "Condorcet" voting, as that would have been logistically too complex for the time.

Also, if I'm not mistaken, Swiss municipal councils us... (read more)

V 0.5.3: Added "Bringing it all together" (85% complete)

1Jameson Quinn
V 0.5.4: Meaningful rewrite to "Shorter "solution" statement", which focuses not on power to elect an individual, but power to elect some member of a set, of whom only 1 won in reality.

Version 0.5.2: Added "Equalizing voter power" and "Measuring "Representation quality", separate from power" sections.

1Jameson Quinn
V 0.5.3: Added "Bringing it all together" (85% complete)

I rewrote the article to incorporate your contribution. I think you'd be interested to read what I added afterwards discussing this idea.

3Thomas Sepulchre
Thanks a lot!

Ping 2.5 of 3; that is, unexpectedly, I got input that superceded the old ping 2 of 3, and I've now incorporated it.

2Ben Pace
I think that the essay is big enough and was very recently resubmitted such that people who re-read it will want a guide to the bits updated most recently, and otherwise will bounce off. My guess is that for future resubmissions it would be good at the top to have a brief list of the main changes since it was last resubmitted. Something like: --- List of updates since this was last submitted to the frontpage: * Section B: Redrew the diagram and gave a better explanation * Section D: Minor edits to the framing * Section E: Total rewrite * Section F: new section added! --break-- Essay begins. --- My current guess is that the main change is where you quote Thomas's solution and your discussion of it? That only needs a short thing added to the top, but I think it's probably important. Can you try that?

Rewritten to reflect Thomas Sepulchre's contribution. Which is awesome, by the way.

Or in other words...

V 0.5.1: the main changes since the previous version 0.5.0 are a complete rewrite of the "Tentative Answer" section based on a helpful comment by a reader here, with further discussion of that solution; including the new Shorter "Solution" Statement subsection. I also added a sketch to visualize the loss I'm using.

1Jameson Quinn
Version 0.5.2: Added "Equalizing voter power" and "Measuring "Representation quality", separate from power" sections.

(Comment rewritten from scratch after comment editor glitched.)

This article is not about what I expected from the title. I've been thinking about "retroactively allocating responsibility", which sounds a lot like "assigning credit", in the context of multi-winner voting methods: which voters get credit for ensuring a given candidate won? The problem here is that in most cases no individual voter could change their vote to have any impact whatsoever on the outcome; in ML terms, this is a form of "vanishing gradient". The s... (read more)

5abramdemski
Yeah, this post is kind of a mix between (first) actually trying to explain/discuss the credit assignment problem, and (second) some far more hand-wavy thoughts relating to myopia / partial agency. I have a feeling that the post was curated based on the first part, whereas my primary goal in writing was actually the second part. In terms of what I talked about (iirc -- not re-reading the post for the sake of this comment), it seems like you're approaching a problem of how to actually assign credit once you have a model. This is of course an essential part of the problem. I would ask questions like: * Does the proposed formula allow us to construct a mechanism which incentivizes a desirable type of behavior? * Does it encourage honesty? * Does it encourage strategically correcting things toward the best collective outcome (even if dishonestly)? These are difficult problems. However, my post was more focused on the idea that you need a model at all in order to do credit assignment, which I feel is not obvious to many people.
2Ben Pace
(Note that we recently pushed a change that allows you to delete your own comments if they have no children, so if you'd like you can delete your earlier comment.)

Nice. Thank you!!!

This corresponds to the Shapley-Shubik index. I had previously discounted this idea but after your comment I took another look and I think it's the right answer. So I'm sincerely grateful to you for this comment.

I thik I'm gonna do something about why high-dimensional is hard. I'll mention the voting context, but mostly discuss the problem in abstract.

This is very well-said, but I still want to dispute the possibility of "perfect alignment". In your clustering analogy: the very existence of clusters presupposes definitions of entities-that-correspond-to-points, dimensions-of-the-space-of-points, and measurements-of-given-points-in-given-dimensions. All of those definitions involve imperfect modeling assumptions and simplifications. Your analogy also assumes that a normal-mixture-model is capable of perfectly capturing reality; I'm aware that this is provably asymptotically true for an inf... (read more)

2johnswentworth
This is a critical point; it's the reason we want to point to the pattern in the territory rather than to a human's model itself. It may be that the human is using something analogous to a normal-mixture-model, which won't perfectly match reality. But in order for that to actually be predictive, it has to find some real pattern in the world (which may not be perfectly normal, etc). The goal is to point to that real pattern, not to the human's approximate representation of that pattern. Now, two natural (and illustrative) objections to this: * If the human's representation is an approximation, then there may not be a unique pattern to which their notions correspond; the "corresponding pattern" may be underdefined. * If we're trying to align an AI to a human, then presumably we want the AI to use the human's own idea of the human's values, not some "idealized" version. The answer to both of these is the same: we humans often update our own notion of what our values are, in response to new information. The reality-pattern we want to point to is the pattern toward which we are updating; it's the thing our learning-algorithm is learning about. I think this is what coherent extrapolated volition is trying to get at: it asks "what would we want if we knew more, thought faster, ...". Assuming that the human-label-algorithm is working correctly, and continues working correctly, those are exactly the sort of conditions generally associated with convergence of the human's model to the true reality-pattern.
Load More