endoself comments on St. Petersburg Mugging Implies You Have Bounded Utility - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (163)
I agree that the Solomonoff prior isn't going to give positive probability to me having any sort of halting oracle. Hmm, I'm not sure whether inferring someone's utility function is computable. I suppose that inferring the utility function for a brain of fixed complexity when arbitrarily large (but still finite) computational capacity can be brought to bear could give an arbitrarily close approximation, so the OP could be revised to fix that. It presently doesn't seem worth the effort though -- the added verbage would obscure the main point without adding anything obviously useful.
By Rice's theorem, inferring utility functions is uncomputable in general, but it is probably possible to do for humans. If not, that would be quite a problem for FAI designers.