= 27d567bc2d48d9f40e9ccc20ea370b15)
timtyler comments on Evaluating the feasibility of SI's plan - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (186)
For a reference, perhaps consider: The Perils of Precaution.
Good reference. SI is perhaps being too cautious by insisting on theoretically perfect AI only.
This is perhaps a silly statement.
Why do you think it is "silly"?
The qualification with "perhaps" makes it tautological and therefore silly. (You may notice that my comment was also tautological).
The slight strawman with "insisting on theoretically perfect" is, well, I'll call it silly. As Eliezer replied, the goal is more like theoretically not doomed.
And last, the typo in "SI is perhaps being too cautious by insisting on theoretically perfect SI" makes it funny.
Thanks, at least I corrected the typo.
The article did mention that even with a "perfect" theory , there may be mistakes in the proof or the implementation may go wrong. I don't remember him saying so as clearly in earlier writings as he did in this comment, so it's good we raised the issue.
When a heuristic AI is creating a successor that shares its goals, does it insist on formally-verified self improvements? Does it try understanding its mushy, hazy goal system so as to avoid reifying something it would regret given its current goals? It seems to me like some mind eventually will have to confront the FAI issue, why not humans then?
If you check with Creating Friendly AI you will see that the term is defined by its primary proponent as follows:
It's an anthropocentric term. Only humans would care about creating this sort of agent. You would have to redefine the term if you want to use it to refer to something more general.
Half specifically referred to "creating a successor that shares it's goals"; this is the problem we face when building an FAI. Nobody is saying an agent with arbitrary goals must at some point face the challenge of building an FAI.
(Incidentally, while Friendly is anthropocentric by default, in common usage analogous concepts relating to other species are referred to as "Friendly to X" of "X-Friendly", just a good is by default used to mean by human standards, but is sometimes used in "good for X".
Apparently not. If it did do these things perfectly, it would not be what we are here calling the "heuristic AI."
Does this essay say anything substantive beyond "maximize expected value"?
That isn't the point of the essay at all. It argues that over-caution can often be a bad strategy. I make a similar point in the context of superintelligence in my the risks of caution video.