= 27d567bc2d48d9f40e9ccc20ea370b15)
Eugine_Nier comments on A critique of effective altruism - LessWrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (152)
This logic suffers from an "infinity discontinuity" problem:
Consider a hypothetical paperclip maximizer. It has some resources, it has to choose between using them to make paperclips or using them to develop more efficient ways of gathering resources. A basic positive feedback calculation means the latter will lead to more paperclips in the long run. But if it keeps using that logic, it will keep developing more and more efficient ways of gathering resources and never actually get around to making paperclips.
In this situation, a maximizer can't work anyway because there is no maximum.
Well, there are states that are better than others.
Unless, or rather until, it hits diminishing returns on resource-gathering. Maybe an ocean, maybe a galaxy, maybe proton decay. With the accessible resources fully captured, it has to decide how much of that budget to convert directly into paperclips, how much to risk on an expedition across the potential barrier, and how much to burn gathering and analyzing information to make the decision. How many in-hand birds will you trade for a chance of capturing two birds currently in Andromeda?
Can't this be solved through exponential discounting? If paperclips made later are discounted more than paperclips made sooner, then we can settle on a stable strategy for when to optimize vs. when to execute, based on our estimations of optimization returns at each stage being exponential, super-exponential, or sub-exponential.
Finding a problem with the simple algorithm that usually gives you a good outcome doesn't mean you get to choose a new utility function.
Clarifying anti-tldr edit time! If you got the above, no need to read on. (I wanted this to be an edit, but apparently I fail at clicking buttons)
The simple algorithm is the greedy decision-finding method "Choose that action which leads to one-time-tick-into-future self having the best possible range of outcomes available via further actions", which you think could handle this problem if only the utility function employed exponential discounting (whether it actually could is irrelevant, since I adress another point).
But your utility function is part of the territory, and the utility function that you use for calculating your actions is part of the map; it is rather suspicious that you want to tweak your map towards a version that is more convenient to your calculations.
Yes, but Eliezer doesn't believe in discounting terminal values.
So, let's be clear - are we talking about what works, or what we think Eliezer is dumb for believing?
Well, first I'm not a consequentialist.
However, the linked post has a point, why should we value future live less?
There are questions about why we should discount at all, or if we are going to, how to choose an appropriate rate.
But even setting those aside: this isn't any more of a solution than the version without discounting. They're similarly reliant on empirical facts about the world (the rate of resource growth); they just give differing answers about how fast that rate needs to be before you should wait rather than cash out.