However it doesn't know whether a bit of the world will be useful for that purpose (hardened footprints in mud), until it has lots of computronium.
If it's that concerned, it can just blast off into space, couldn't it? Might slow down development, but the hypothetical mud footprints out to be fine... No harm done by computronium in the sun.
The question is should we program it to be that concerned? The human predicate is necessary for CEV if I remember correctly, you would want to extrapolate the volition of everyone currently informationally smeared across the planet as well as the more concentrated humans. I can't find the citation at the moment, I'll hunt tomorrow.
This thread is for the discussion of Less Wrong topics that have not appeared in recent posts. Feel free to rid yourself of cached thoughts by doing so in Old Church Slavonic. If a discussion gets unwieldy, celebrate by turning it into a top-level post.
If you're new to Less Wrong, check out this welcome post.