= 27d567bc2d48d9f40e9ccc20ea370b15)
Nick_Tarleton comments on Late Great Filter Is Not Bad News - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (75)
Let's take this further: is there any reason, besides our obsession with subjective anticipation, to discuss whether a late great filter is 'good' or 'bad' news, over and above policy implications? Why would an idealized agent evaluate the utility of counterfactuals it knows it can't realize?
That is a good question, and one that I should have asked and tried to answer before I wrote this post. Why do we divide possible news into "good" and "bad", and "hope" for good news? Does that serve some useful cognitive function, and if so, how?
Without having good answers to these questions, my claim that a late great filter should not be considered bad news may just reflect confusion about the purpose of calling something "bad news".
About the cognitive function of "hope": it makes evolutionary sense to become all active and bothered when a big pile of utility hinges on a single uncertain event in the near future, because that makes you frantically try to influence that event. If you don't know how to influence it (as in the case of a lottery), oh well, evolution doesn't care.
Evolution might care. That is, systems that expend a lot of attention on systems they can't influence might do worse than systems that instead focus their attention on systems they can influence. But yes, either there weren't any of the second kind of system around to compete with our ancestors, or there were and they lost out for some other reason, or there were and it turns out that it's a bad design for our ancestral environment.
If we expect a late great filter, we may need to strive significantly harder to avoid it. The policy implications are staggering.
We are who we are, but who we are is not fixed. What we care about seem to depend on what arguments we listen to or think up, and in what order. (See my Shut Up and Divide post for an example of this.)
While a ideally rational agent (according to our current best conception of ideal rationality) would seek to preserve its values regardless of what they are, some humans (including me, for example) actively seek out arguments that might change what they care about. Such "value-seeking" behavior doesn't seem irrational to me, even though I don't know how to account for it in terms of rationality.
And while it seems impossible for a human to completely give up subjective anticipation, it does seem possible to care less about it.
I would say it is part of checking for reflective consistency. Ideally, there shouldn't be arguments that change your (terminal) values, so if there are, you want to so you can figure out what is wrong and how to fix it.
I don't think that explanation makes sense. Suppose an AI thinks it might have a security hole in its network stack, so that if someone sends it a certain packet, it would become that person's slave. It would try to fix that security hole, without actually seeking to have such a packet sent to itself.
We humans know that there are arguments out there that can change our values, but instead of hardening our minds against them, some of us actually try to have such arguments sent to us.
In the deontological view of values this is puzzling, but in the consequentialist view it isn't: we welcome arguments that can change our instrumental values, but not our terminal values (A.K.A. happiness/pleasure/eudaimonia/etc.). In fact I contend that it doesn't even make sense to talk about changing our terminal values.
Congratulations, you just reinvented [a portion of] PCT. ;-)
[Clarification: PCT models the mind as a massive array of simple control circuits that act to correct errors in isolated perceptions, with consciousness acting as a conflict-resolver to manage things when two controllers send conflicting commands to the same sub-controller. At a fairly high level, a controller might be responsible for a complex value: like correcting hits to self-esteem, or compensating for failings in one's aesthetic appreciation of one's work. Such high-level controllers would thus appear somewhat anthropomorphically agent-like, despite simply being something that detects a discrepancy between a target and an actual value, and sets subgoals in an attempt to rectify the detected discrepancy. Anything that we consider of value potentially has an independent "agent" (simple controller) responsible for it in this way, but the hierarchy of control does not necessarily correspond to how we would abstractly prefer to rank our values -- which is where the potential for irrationaity and other failings lies.]