In that case, you might want to consider rewriting your post. Right now, the crazy messiah vibe is coming through very strongly. Either back it up and stop wasting our time, or rewrite it to assert less social dominance. If you do the latter without the former, people get cranky.
I'm mainstream, you guys are fringe, do you understand? I am informing you that you are not only not convincing, but look like complete clowns who don't know big O from a letter of alphabet. I know you want to do better than this. And I know some of people here have technical knowledge.
Er... what?
I'm a software developer too (in training, anyway). Sometimes I'm wrong about things. It's not unusual, or the fault of the material I was reading when I made the mistake. I'm not even certain you're wrong. What I am certain of is that your provided argument does not support, or even strongly imply your stated thesis. If you want to change my mind, then give me something to work with.
EDIT: You're right about one thing -- Less Wrong has a huge image problem; but that's entirely tangential to the question at issue.
What I am certain of is that your provided argument does not support, or even strongly imply your stated thesis.
I know this. I am not making argument here (or actually, trying not to). I'm stating my opinion, primarily on presentation of the argument. If you want argument, you can e.g. see what Hansen has to say about foom. It is, deliberately this way. I am not some messiah hell bent on rescuing you from some wrongness (that would be crazy).
I also very much liked the idea that a self-improving AI would probably wirehead itself. It never occured to me, and it makes a lot of sense.
This idea is intuitively plausible, but doesn't hold up when considering rational actors that value states of the world instead of states of their minds. Consider a paperclip maximizer, with the goal "make the number of paperclips in the universe as great as possible". Would it rather a) make paperclips, or b) wirehead to convince itself that the universe is already full of paperclips? Before it wireheads, it knows that option a) will lead to more paperclips, so it does that. Similarly, I would rather actually help people than feel the warm glow that comes from helping people without any actual helping.
value states of the world instead of states of their minds
Easier said than done. Valuing state of the world is hard; you have to rely on senses.
<The arguments for are pretty bad upon closer scrutiny, and are almost certainly rationalizations rather than rationality. Sorry.
Unsubstantiated assertion.
It is incredibly unlikely to find yourself in the world where the significant insights about real doomsday is coming from single visionary who did so little that can be unambiguously graded,
Interesting mixture of misunderstanding how probability works, and an ad-hominem.
Unless the work is in fact focussed in some secret FAI effort, it seems likely that some automated software development tool would foom, reaching close to absolute maximum optimality on certain hardware. But will remain a tool. Availability of such ultra optimal tools in all aspects of software and hardware design would greatly decrease the advantage that self willed UFAI might have.
This is a genuinely interesting thought -- is such a tool achievable, theoretically, and what would it do to the early development of a FOOM brain? A post about this would be worth a read.
if I point out that the AI has good reasons not to kill us all due to it not being able to determine if it is within top level world or a simulator or within engineering test sim. It is immediately conjectured that we will still 'lose' something because it'll take up some resources in space. That is rationalization. Privileging a path of thought.
Or because your objections don't actually solve the underlying problems. You get the same kind of 'updating' if you respond to the thermodynamic objections to your car with a windmill on top by suggesting that you stand on the roof and blow. Also, you should note that you listed one of the weakest possible counter-arguments to your own argument, which is bad practice, rationality-wise.
The botched FAI attempts have their specific risk - euthanasia, wireheading, and so on, which don't exist for an AI that is not explicitly friendly.
Dead wrong. You've clearly never been eaten by a paperclip maximizer.
and if it would, it would wirehead rather than add more hardware; and it would be incredibly difficult to prevent it from doing so.
Unsubstantiated assertion.
EY very strongly pattern-matches to this friend of mine, and focusses very hard on the known unknowns aspect of the problem about which we know very little - which can easily steer one into a very dangerous zone full of unknown unknowns - the not-quite-FAIs that euthanize us or worse.
Ad-hominem, and the rest is just unwarranted condescension. There's a good post to be be written about a skeptical approach to AI-risk scenarios, and this is definitely not it.
Okay, then, you're right: the manner of presentation of the AI risk issue on lesswrong somehow makes a software developer respond with incredibly bad and unsubstantiated objections.
Why when bunch of people get together, they don't even try to evaluate the impression they make on 1 individual? (except very abstractly)
This is wrong in several ways.
- An initial AI isn't necessarily a utility maximizer of this very sophisticated form (with a utility function defined in terms of a robust model of the world from a 3rd person perspective), building such a thing is a further challenge beyond making AI
- If someone designs an AI with a sensory utility function, taking control of its sensory channel is just optimizing for its utility function; it's "wireheading" from the designer's perspective if they expected to be able to ensure that the preferred inputs could only be obtained by performing assigned tasks
- A utility-maximizer could have reason to modify or even eliminate its own utility function for a variety of reasons, especially when interacting with powerful agents and when its internals are at least partially transparent
Precisely, thank you! I hate arguing such points. Just because you can say something in English does not make it an utility function in the mathematical sense. Furthermore, just because in English it sounds like modification of utility function, does not mean that it is mathematically a modification of utility function. Real-world intentionality seem to be a separate problem from making a system that would figure out how to solve problems (mathematically defined problems), and likely, a very hard problem (in the sense of being very difficult to mathematically define).
What you describe is arguably already a (mediocre) FAI, with all the attendant challenges.
With all of them? How so?
Oh please. There's a difference between what makes a useful heuristic for you to decide what to spend time considering and what makes for a persuasive argument in a large debate where participants are willing to spend time hashing out specifics.
DH1. Ad Hominem.
An ad hominem attack is not quite as weak as mere name-calling. It might actually carry some weight. For example, if a senator wrote an article saying senators' salaries should be increased, one could respond:Of course he would say that. He's a senator.
This wouldn't refute the author's argument, but it may at least be relevant to the case. It's still a very weak form of disagreement, though. If there's something wrong with the senator's argument, you should say what it is; and if there isn't, what difference does it make that he's a senator?
http://paulgraham.com/disagree.html
I found it very instrumentally useful to try to factor out the belief-propagation impacts of people with nothing clearly impressive to show.
If even widely read bloggers like EY don't qualify to affect your opinions, it sounds as though you're ignoring almost everyone.
No one is expecting you to adopt their priors... Just read and make arguments about ideas instead of people, if you're trying to make an inference about ideas.
If even widely read bloggers like EY don't qualify to affect your opinions, it sounds as though you're ignoring almost everyone.
I think you discarded one of conditionals. I read Bruce Schneier's blog. Or Paul Graham's. Furthermore, it is not about disagreement with the notion of AI risk. It's about keeping the data non cherry picked, or less cherry picked.
Bravo! If any post deserves to be on main, it's this entry, as it expresses what a lot of people think, here and elsewhere.
EDIT: To clarify, the productive discussion that will hopefully ensue would go a long way to either address or validate the arguments presented, and to help evaluate the degree of cultishness of the forum for those of us not yet absolutely convinced either way.
Thanks. Glad you like it. I did put some work into it. I also have a habit of keeping epistemic hygiene by not generating a hypothesis first then cherry-picking examples in support of it later, but that gets a lot of flak outside scientific or engineering circles.
Like pointed out by David Gerard, EY isn't the only game in town. I think the basic question he's trying to answer- "If you had to build human values from scratch, how would you do it?" - is a very interesting one, even if I think his answer to it is not very good.
How you answer that question depends on what shape you think the future will take. EY thinks we'll invent a god, and so we need to proceed very carefully and get the answer completely correct the first time. Hanson thinks we'll evolve as a society, and so most of the answers will get found along the way- but we can make some predictions and alter the shape of the future with our actions now.
Personal values also differ heavily. I don't expect to live forever- and so if my descendents are memetic rather than genetic, and synthetic rather than organic, it's no great loss. (That's not to say there are no values / memes I'd like to preserve, of course.) To someone that wants to personally exist for a long time, it becomes very relevant what part humans have in the future.
To someone that wants to personally exist for a long time, it becomes very relevant what part humans have in the future.
I think this is an awesome point I overlooked. That talk of future of mankind, that assigning of the moral values to future humans but zero to the AI itself... it does actually make a lot more sense in context of self preservation.
View more: Next
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d)
My understanding was that this wasn't any attempt to rigorously formulate the idea of a randomly chosen mind, just suggest the possibility of a huge number of possible reasoning architectures that didn't share human goals.
This is one of those points you really should've left out... If you got something to say on this topic, say it, we all want to hear it (or at least I do). Of course it's not obvious that in FAI effort will certainly be helpful, but empirically, people trying to do things seems to make it more likely that they get done.
Have you heard of G?
Intelligent people tend to be impractical because of bugs in human brains that we shouldn't expect to appear in other reasoning architectures.
Of course general intelligence is a complicated multifaceted thing, but that doesn't mean it can't be used to improve itself. Humans are terrible improving ourselves because we don't have access to our own source code. What if that changed?
You seem awfully confident. If you have a rigorous argument, could you share it?
You might wish to read someone who disagrees with you:
http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf
Thanks a lot; one of the big problems with fringe beliefs is that folks rarely take the time to counterargue since there isn't much in it for them.
Even better is if such criticism can take the form of actually forming a consistent contrasting point of view supported by rigorous arguments, without making drama out of the issue, but I'll take what I can get.
That seems wrong to me. When I hear that some group espouses some belief, I give them a certain amount of credit by default. If I hear their arguments and find them less persuasive than I expected, my confidence in their position goes down.
I have certainly seen some of what I would consider privileging of the hypothesis being done by AGI safety advocates. However, groupthink is not all or nothing; better to extract the best of the beliefs of others then throw them out wholesale.
Some of your arguments are very weak in this post (e.g. the lone genius point basically amounts to ad hominem) and you seem to refer frequently to points that you've made in the past without linking to them. Do you think you could assemble a directory of links to your thoughts on this topic ordered from most important/persuasive to least?
Quoting from
http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf
I had been thinking, could it be that respected computer vision expert indeed believes that the system will just emerge world intentionality? That'd be pretty odd. Then I see it is his definition of AI here, it already presumes robust implementation of world intentionality. Which is precisely what a tool like optimizing compiler lacks.
edit: and in advance of other objection: I know evolution can produce what ever argument demands. Evolution, however, is a very messy and inefficient process for making very messy and inefficient solutions to the problems nobody has ever even defined.