= 27d567bc2d48d9f40e9ccc20ea370b15)
Dr_Manhattan comments on Reply to Holden on 'Tool AI' - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (348)
Commentary (there will be a lot of "to me"s because I have been a bystander to this exchange so far):
I think this post misunderstands Holden's point, because it looks like it's still talking about agents. Tool AI, to me, is a decision support system: I tell Google Maps where I will start from and where I will leave from, and it generates a route using its algorithm. Similarly, I could tell Dr. Watson my medical data, and it will supply a diagnosis and a treatment plan that has a high score based on the utility function I provide.
In neither case are the skills of "looking at the equations and determining real-world consequences" that necessary. There are no dark secrets lurking in the soul of A*. Indeed, that might be the heart of the issue: tool AI might be those situations where you can make a network that represents the world, identify two nodes, and call your optimization algorithm of choice to determine the best actions to choose to attempt to make it from the start node to the end node.
Reducing the world to a network is really hard. Determining preferences between outcomes is hard. But Tool AI looks to me like saying "well, the whole world is really too much. I'm just going to deal with planning routes, which is a simple world that I can understand," where the FAI tools aren't that relevant. The network might be out of line with reality, the optimization algorithm might be buggy or clumsy, but the horror stories that keep FAI researchers up at night seem impossible because of the inherently limited scope, and the ability to do dry runs and simulations until the AI's model of reality is trusted enough to give it control.
Now, this requires that AI only be used for things like planning where to put products on shelves, not planning corporate strategy- but if you work from the current stuff up rather than from the God algorithm down, it doesn't look like corporate strategy will be on the table until AI is developed to the point where it could be trusted with that. If someone gave me a black box that spit out plans based on English input, then I wouldn't trust it and I imagine you wouldn't either- but I don't think that's what we're looking at, and I don't know if planning for that scenario is valuable.
It seems to me that SI has discussed Holden's Tool AI idea- when it made the distinction between AI and AGI. Holden seems to me to be asking "well, if AGI is such a tough problem, why even do it?".
I think you're arguing about Karnovsky's intention, but it seems clear (to me :) that he is proposing something much more general that a strategy of pursuing best narrow AIs - see the "Here's how I picture the Google Maps AGI " code snipped Eliezer is working of.
In any case, taking your interpretation as your proposal, I don't think anyone is disagreeing with the value of building good narrow AIs where we can, the issue is that the world might be economically driven towards AGI, and someone needs to do the safety research, which is essentially the SI mission.
I agree the code snippet is relevant, but it looks like pseudocode for the "optimization algorithm of choice" part- the question is what dataset and sets of alternatives you're calling it over. Is it a narrow environment where we can be reasonably confident that the model of reality is close to reality, and the model of our objective is close to our objective? Or is it a broad environment where we can't be confident about the fidelity of our models of reality or our objectives without calling in FAI experts to evaluate the approach and find obvious holes?
Similarly, is it an environment where the optimization algorithm needs to take into account other agents and model them, or one in which the algorithm can just come up with a plan without worrying about how that plan will alter the wider world?
It seems like explaining the difference between narrow AI and AGI and giving a clearer sense of what subcomponents make a decision support system dangerous might work well for SI. Right now, the dominant feature of UFAI as SI describes it is that it's an agent with a utility function- and so the natural response to SI's description is "well, get rid of the agency." That's a useful response only if it constricts the space of possible AIs we could build- and I think it does, by limiting us to narrow AIs. Spelling out the benefits and costs to various AI designs and components will both help bring other people to SI's level of understanding and point out holes in SI's assumptions and arguments.
I agree with you that that is a position one might take in response to the UFAI risks, but it seems from reading Karnovsky that he thinks some Oracle/"Tool" AI (quite general) is safe if you get rid of that darned explicit utility function. Eliezer is trying to disabuse him of the notion. If your understanding of Karnovsky is different, mine is more like Eliezers. In any case this is probably mute, since Karnovsky is very likely to respond one way or another, given this turned into a public debate.
I think agency and utility functions are separate, here, and it looks like agency is the part that should be worrisome. I haven't thought about that long enough to state that definitively, though.
Right, but it looks like by moving from where Eliezer is towards where Holden is, where I would rather see him move from where Holden is to where Eliezer is. Much of point 2, for example, is discussing how hard AGI is- which, to me, suggests we should worry less about it, because it is unlikely to be implemented successfully, and any AIs we will see will be narrow- in which case AGI thinking isn't that relevant.
My approach would have been along the lines of: start off with a safe AI, add wrinkles until its safety is no longer clear, and then discuss the value of FAI researchers.
For example, we might imagine a narrow AI that takes in labor stats data, econ models, psych models, and psych data and advises schoolchildren on what subjects to study and what careers to pursue. Providing a GoogleLifeMap to one person doesn't seem very dangerous- but what about when it's ubiquitous? Then there will be a number of tradeoffs that need to be weighed against each other and it's not at all clear that the AI will get them right. (If the AI tells too many people to become doctors, the economic value of being a doctor will decrease- and so the AI has to decide who of a set of potential doctors to guide towards being a doctor. How will it select between people?)
In addition to providing advice to people, it can aggregate the advice it has provided, translate it into economic terms, and hand it off to some independent economy-modeling service which is (from GoogleLifeMap's perspective) a black box. Economic predictions about the costs and benefits of various careers are compiled, and eventually become GoogleLifeMap's new dataset. Possibly it has more than one dataset, and presents career recommendations from each of them in parallel: "According to dataset A, you should spend nine hours a week all through high school sculpting with clay, but never show the results to anyone outside your immediate family, and study toward becoming a doctor of dental surgery; according to dataset B, you should work in foodservice for five years and two months, take out a thirty million dollar life insurance policy, and then move to a bunker in southern Arizona."