Comment on Evaluating the feasibility of SI's plan

Thu, 10 Jan 2013 17:23:21 GMT

When a heuristic AI is creating a successor that shares its goals, does it insist on formally-verified self improvements? Does it try understanding its mushy, hazy goal system so as to avoid reifying something it would regret given its current goals? It seems to me like some mind eventually will have to confront the FAI issue, why not humans then?

Discuss