The first thing that strikes me as off about AI_should is that if you use it, then your definition will be very long and complex (because Human Value is Complex) but your assumption will be short (because the code that takes "You should do X" and then does X can be minimal). This is backwards - definitions should be clean, elegant, and simple, while assumptions often have to be messy and complex.

When they can be clean, elegant, and simple, sure. However, they are often clean and elegant because of other assumed definitions. Consider the definition of a continuous function from the reals to the reals. This requires definitions of real numbers, limits, and functions. This further requires definitions of number, absolute value, <, >, etc. When you put it all together, it isn't very clean and elegant. Point being, we can talk in shorthand about our utility functions (or shouldness) somewhat effectively, but we shouldn't be surprised when it's complicated when we try to program it into a computer.

Furthermore, so what if a definition isn't elegant, assuming it "carves reality at it's joints"?