humans probably don't even have a utility-function because our goals are not stable

Even what a human is is not stable.

My skin regenerates...my teeth are mineralized...I have some fillings...I eat some grapes...and spit out the pits...pull on some pants...and go drive a car. How am I qualitatively non-arbitrarily not my car?

The Voyager Horcrux has naked humans, but if hermit crabs were intelligent, would they send the probe up with an image of hermit crabs without shells?

I am not sure.