As far as I understand, that's pretty much the friendly AI problem.
I think it's simpler, but not by much. Instead of knowing both the value and cost of everything, you just need to know the cost of everything. (The 'actual' cost, that is, not the full economic cost, which by including opportunity cost includes the value problem.) You could probably get away with an approximation of the cost, though a guarantee like "at least as high as the actual cost" is probably helpful.
So if Lawrence says "I'll pay up to $10 for a hamburger," either it can find a plan that provides Lawrence a hamburger for less than $10 (gross cost, not net cost), or it says "sorry, can't find anything at that price range."
I think there's a huge amount of work to get there--you have to have an idea of 'gross cost' that matches up well enough with our intuitions, which is an intuition-encoding problem and thus hard. (If it tweets at the local burger company to get a coupon for a free burger, what's the cost?)
If it's worth saying, but not worth its own post (even in Discussion), then it goes here.
Notes for future OT posters:
1. Please add the 'open_thread' tag.
2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)
3. Open Threads should be posted in Discussion, and not Main.
4. Open Threads should start on Monday, and end on Sunday.