Center on Long-Term Risk (CLR) is accepting applications to the next iteration of our introductory fellowship. We are now planning to run two related programs: the CLR Foundations Course and the CLR S-Risk Seminars.[1] The CLR Foundations Course will introduce people to CLR’s research on s-risk reduction. Apply by Wednesday...
I’d like an easy way to distinguish the behavior of payoff-maximizing players who would or would not play a strategy in an extensive form game that deviates from the subgame-perfect (Bayesian) equilibrium strategy profile of the game (when their strategy is known to their opponent, and their opponent is also...
[Note (6/30/19): I think I made interesting and valid points with this piece, but my framing was terrible. I should have clarified who are the people for whom I think these arguments should be persuasive, had a longer introductory section talking about why skilled hunting is much less problematic than...
Center on Long-Term Risk (CLR) is accepting applications to the next iteration of our introductory fellowship. We are now planning to run two related programs: the CLR Foundations Course and the CLR S-Risk Seminars.[1]
The CLR Foundations Course will introduce people to CLR’s research on s-risk reduction. Apply by Wednesday November 27th, 23:59 Pacific Time.
Apply to the CLR Foundations CourseThe CLR S-Risk Seminars will dive deeper into different priority cause areas within s-risk reduction and is intended for people who have completed the CLR Foundations Course or have a similar level of context.
Register your interest for the CLR S-Risk SeminarsThe Foundations Course is designed to introduce participants to CLR’s research... (read 729 more words →)
Strongly agree with this. How I frame the issue: If people want to say that they identify as an "experiencer" who is necessarily conscious, and don't identify with any nonconscious instances of their cognition, then they're free to do that from an egoistic perspective. But from an impartial perspective, what matters is how your cognition influences the world. Your cognition has no direct access to information about whether it's conscious such that it could condition on this and give different outputs when instantiated as conscious vs. nonconscious.
Note that in the case where some simulator deliberately creates a behavioural replica of a (possibly nonexistent) conscious agent, consciousness does enter into the chain of logical causality for why the behavioural replica says things about its conscious experience. Specifically, the role it plays is to explain what sort of behaviour the simulator is motivated to replicate. So many (or even all) non-counterfactual instances of your cognition being nonconscious doesn't seem to violate any Follow the Improbability heuristic.
Thanks for the cool discussion Ryan and Nate! This thread seemed pretty insightful to me. Here’s some thoughts / things I’d like to clarify (mostly responding to Nate's comments).[1]
Who’s doing this trade?
In places it sounds like Ryan and Nate are talking about predecessor civilisations like humanity agreeing to the mutual insurance scheme? But humans aren’t currently capable of making our decisions logically dependent on those of aliens, or capable of rescuing them. So to be precise the entity engaging in this scheme or other acausal interactions on our behalf is our successor, probably a FAI, in the (possibly counterfactual or counterlogical) worlds where we solve alignment.
Nate says:
... (read 1532 more words →)Roughly speaking, I suspect that the
It's definitely not clear to me that updatelessness + Yudkowsky's solution prevent threats. The core issue is that a target and a threatener face a prima facie symmetric decision problem of whether to use strategies that depend on their counterpart's strategy or strategies that do not depend on their counterpart's strategy.[1]
In other words, the incentive targets have to use non-dependent strategies that incentivise favourable (no-threat) responses from threateners is the same incentive threateners have to use non-dependent strategies that incentivise favourable (give-into-threat) responses from targets. This problem is discussed in more detail in parts of Responses to apparent rationalist confusions about game / decision theory and in Updatelessness doesn't solve most problems.
There... (read more)
How to deal with crucial considerations and deliberation ladders (link goes to a transcript + audio).
I like this post a lot! Three other reasons came to mind, which might be technically encompassed by some of the current ones but seemed to mostly fall outside the post's framing of them at least.
Some (non-agentic) repeated selections won't terminate until they find a bad thing
In a world with many AI deployments, an overwhelming majority of deployed agents might be unable to mount a takeover, but the generating process for new deployed agents might not halt until a rare candidate that can mount a takeover is found. More specifically, consider a world where AI progress slows (either due to governance interventions or a new AI winter), but people continue conducting training... (read 485 more words →)
Another way interpretability work can be harmful: some means by which advanced AIs could do harm require them to be credible. For example, in unboxing scenarios where a human has something an AI wants (like access to the internet), the AI might be much more persuasive if the gatekeeper can verify the AI's statements using interpretability tools. Otherwise, the gatekeeper might be inclined to dismiss anything the AI says as plausibly fabricated. (And interpretability tools provided by the AI might be more suspect than those developed beforehand.)
It's unclear to me whether interpretability tools have much of a chance of becoming good enough to detect deception in highly capable AIs. And there are... (read more)
I’d like an easy way to distinguish the behavior of payoff-maximizing players who would or would not play a strategy in an extensive form game that deviates from the subgame-perfect (Bayesian) equilibrium strategy profile of the game (when their strategy is known to their opponent, and their opponent is also payoff-maximizing).
An example of what I’m interested in can be seen in an ultimatum game where a proposer presents a responder with an offer of the form , where and for are the respective payoffs of the proposer and the responder.
For now, let’s call a strategy subgame-optimal if for every subgame of the game, the strategy’s restriction to that subgame is still optimal within the subgame. In other words,... (read 799 more words →)
There is another very important component of dying with dignity not captured by the probability of success: the badness of our failure state. While any alignment failure would destroy much of what we care about, some alignment failures would be much more horrible than others. Probably the more pessimistic we are about winning, the more we should focus on losing less absolutely (e.g. by researching priorities in worst-case AI safety).
I feel conflicted about this post. Its central point as I'm understanding it is that much evidence we commonly encounter in varied domains is only evidence about the abundance of extremal values in some distribution of interest, and whether/how we should update our beliefs about the non-extremal parts of the distribution is very much dependent on our prior beliefs or gears-level understanding of the domain. I think this is a very important idea, and this post explains it well.
Also, felt inspired to search out other explanations of the moments of a distribution - this one looks pretty good to me so far.
On the other hand, the men's rights discussion felt out of... (read more)
This is super interesting!
Quick typo note (unless I'm really misreading something): in your setups, you refer to coins that are biased towards tails, but in your analyses, you talk about the coins as though they are biased towards heads.
One is the “cold pool”, in which each coin comes up 1 (i.e. heads) with probability 0.1 and 0 with probability 0.9. The other is the “hot pool”, in which each coin comes up 1 with probability 0.2
random coins with heads-probability 0.2
We started with only tails
full compression would require roughly tails, and we only have about
As far as I'm aware, there was not (in recent decades at least) any controversy that word/punctuation choice was associative. We even have famous psycholinguistics experiments telling us that thinking of the word "goose" makes us more likely to think of the word "moose" as well as "duck" (linguistic priming is the one type of priming that has held up to the replication crisis as far as I know). Whenever linguists might have bothered to make computational models, I think those would have failed to produce human-like speech because their associative models were not powerful enough.
We'll be discussing some Applied Rationality Strategies. If you'd like to share an activity or strategy with the group, just prepare something.
Please share any relevant SlateStarCodex (or adjacent) readings. Might include arguments for and against widening the circle of those with whom we engage in Rational argument.
We will order some vegan food after we've assembled, probably Indian.
We're discussing systemic inadequacies that Scott Alexander personifies as Moloch. Please (re)read this piece beforehand: http://slatestarcodex.com/2014/07/30/meditations-on-moloch/
Try to arrive on time! Some of us will probably get dinner afterwards.
Other useful reading:
Moloch's Toolbox from Inadequate Equilibria by Eliezer Yudkowsky
[Note (6/30/19): I think I made interesting and valid points with this piece, but my framing was terrible. I should have clarified who are the people for whom I think these arguments should be persuasive, had a longer introductory section talking about why skilled hunting is much less problematic than slaughter and consumption of farmed animals, spun the fifth argument off into its own post, and maybe left off an epistemic status. I'd like to revise this at some point, but not sure when I'll get around to it.]
Content Warning
Explicit descriptions of wild animal suffering and serious discussion of killing and consuming animals as a potentially net-positive intervention.
Premise
Two months ago, I... (read 3430 more words →)