= 27d567bc2d48d9f40e9ccc20ea370b15)
timtyler comments on AI Risk and Opportunity: Humanity's Efforts So Far - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (47)
Better than utility maximization? I'm inclined to facepalm.
Better than "predict future worlds resulting from actions, calculate utilities of future worlds, choose one with largest" naive model, considering the imperfect information, highly inaccurate prediction, expensive-to-evaluate utility, and the time constraints.
Real agents: predict at varying levels of accuracy depending to available time, predict the sign of difference of utilities instead of numeric utilities, substitute cheaper-to-evaluate utilities in place of original utilities, employ reverse reasoning (start with desired state, sort of simulate backwards), employ general strategies (e.g. trying to produce actions that are more reversible to avoid cornering itself), and a zillion of other approaches.
Evolution normally provides an underlying maximand for organisms: fitness. To maximise their fitnesses, many organisms use their own optimisation tool: a brain - which is essentially a hedonism maximiser. It's true that sometimes the best thing to do is to use a fast-and-cheap utility function to burn through some task. However, that strategy is normally chosen by a more advanced optimiser for efficiency reasons.
I think a reasonable picture is that creatures act so as to approximate utility maximization as best they can. Utility is an organism's proxy for its own fitness, and fitness is what really is being maximised.
"Complex reasoning" and "simulating backwards" are legitimate strategies for a utility maximizer to use - if they help to predict how their environment will behave.
Well, being a hedonism maximizer leads to wireheading.
Nah, whenever a worse/flawed approximation to maximization of hedonistic utility results in better fitness, the organisms do that too.
They don't help predict, they help to pick actions out of the space of some >10^1000 actions that the agent really has to choose from. That results in apparent preference for some actions, but not others, preference having nothing to do with any utilities and everything to do with action generation. Choosing in the huge action space is hard.
The problem really isn't so much with utility maximization as model - one could describe anything as utility maximization, in the extreme, defining the utility so that when action matches that chosen by agent, the utility is 1, and 0 otherwise. The problem is when inexperienced, uneducated people start taking the utility maximization too literally and imagining as an ideal some very specific architecture - the one that's forecasting and comparing utilities that it forecasts - and start modelling the AI behaviour with this idea.
Note, by the way, that such system is not maximizing an utility, but system's prediction of the utility. Map vs territory confusion here. Maximizing your predicted utility gets you into checkmate when the opponent knows how your necessarily inexact prediction works.
Well, that gets complicated. Of course, we can see that there are non-wireheading hedonism maximizers - so there are ways around this problem.
Well again this is complicated territory. The evolutionary purpose of the brain is to maximise inclusive fitness - and to do that it models its expectations about its fitness and maximises those. If the environment changes in such a way that the brain's model is inaccurate then evolution could hack in non-brain-based solutions - but ultimately it is going to try and fix the problem by getting the organism's brain proxy for fitness and evolutionary fitness back into alignment with each other. Usually these two are fairly well aligned - though humans are in a rapidly-changing environment, and so are a bit of an exception to this.
Basically, having organisms fight against their own brains is wasteful - and nature tries to avoid it - by making organisms as harmonious as it can manage.
So, that is the point of predicting! It's a perfectly conventional way of traversing a search tree to run things backwards and undo - rather than attempt to calculate each prediction from scratch somehow or another. Something like that is not a deviation from a utility maximisation algorithm.
Not necessarily - if you expect to face a vastly more powerful agent, you can sometimes fall-back on non-deterministic algorithms - and avoid being outwitted in this particular way.
Anyway, you have to maximise your expectations of utility (rather than your utility). That isn't a map vs territory confusion, it's just the way agents have to work.
Not sure how non-wireheaded though. One doesn't literally need a wire to the head to have a shortcut. Watching movies or reading fiction is pretty wireheaded effort.
I'm not quite sure its how it works. The pain and the pleasure seem to be the reinforcement values for neural network training, rather than actual utilities of any kind. Suppose you are training a dog not to chew stuff, by reinforcements. The reinforcement value is not proportional to utility of behaviour, but is set as to optimize the training process.
See, if there's two actions, one result in utility 1000 and other result in utility 100, this method can choose the one that results in utility 100 because it is reachable by imperfect backwards tracing while the 1000 one isn't (and is lost in giant space that one can't search). At that point, you could of course declare that being backwards traceable to is a very desirable property of an action, and goalpost shift so that this action has utility 2000, but its clear that this is a screwy approach.
And how do you improve the models you use for expectations?
Of course you can describe literally anything as 'utility maximization', the issue is that agent which is maximizing something, doesn't really even need to know what it is maximizing, doesn't necessarily do calculation of utilities, et cetera. You don't really have model here, you just have a description, and if you are to model it as utility maximizer, you'll be committing fallacy as with that blue minimizing robot
Maybe - if the person rewarding the dog is doing it wrong. Normally, you would want those things to keep reasonably in step.
So, in the "forecasting/evaluation/tree pruning" framework, that sounds as though it is a consequence of tree pruning.
Pruning is inevitablle in resource-limited agents. I wouldn't say it stopped them from being expected utility maximisers, though.
a) get more data; b) figure out how to compress it better;
Alas, I don't like that post very much. It is an attack on the concept of utility-maximization, which hardly seems productive to me. Anyway, I think I see your point here - though I am less clear about how it relates to the previous conversation (about expectations of utility vs utility - or more generally about utility maximisation being somehow "sub-optimal").
If the dog chews something really expensive up, there is no point punishing the dog proportionally more for that. That would be wrong; some level of punishment is optimal for training; beyond this is just letting anger out.
It's not mere pruning. You need a person to be able to feed your pets, you need them to get through the door, you need a key, you can get a key at key duplicating place, you go to key duplicating place you know of to make a duplicate.
That stops them from being usefully modelled as 'choose action that gives maximum utility'. You can't assume that it makes action that results in maximum utility. You can say that it makes action which results in as much utility as this agent with its limitations could get out of that situation, but that's almost tautological at this point. Also, see
http://en.wikipedia.org/wiki/Intelligent_agent
for terminology.
Well, the utility agent as per wiki article, is clearly stupid because it won't reason backwards. And the utility maximizers discussed by purely theoretical AI researchers, likewise.
Needs something better than trying all models and seeing what fits, though. One should ideally be able to use the normal reasoning to improve models. It feels that a better model has bigger utility.
You would probably want to let the dog know that some of your chewable things are really expensive. You might also want to tell it about the variance in the value of your chewable items. I'm sure there are some cases where the owner might want to manipulate the dog by giving it misleading reward signals - but honest signals are often best.
These are the researchers who presume no computational resource limitation? They have no need to use optimisation heuristics - such as the ones you are proposing - they assume unlimited computing resources.
Needs something better than trying all models and seeing what fits, though. One should ideally be able to use the normal reasoning to improve models. [...]
Sure. Humans use "normal reasoning" to improve their world models.
Well, i don't think that quite works, dogs aren't terribly clever. Back to humans, e.g. significant injuries hurt a lot less than you'd think they would, my guess is that small self inflicted ones hurt so much for effective conditioning.
The ugly is when some go on and talk about the AGIs certainly killing everyone unless designed in some way that isn't going to work. And otherwise paint wrong pictures of AGI.
Ya. Sometimes even resulting in breakage, when they modify world models to fit with some pre-existing guess.