I have been trying to invent an AI for over a year, although I haven't made a lot of progress lately. My current approach is a bit similar to how our brain works according to "Society Of Mind". That is, when it's finished the system is supposed to consist of a collection of independent, autonomous units that can interact and create new units. The tricky part is of course the prioritization between the units. How can you evaluate how promising an approach is? I recently found out that something like this has already been tried, but that has happened to me several times by now as I started thinking and writing about AI before I had read any books on that subject (I didn't have a decent library in school).

I have no great hopes that I will actually manage to create something usefull with this, but even a tiny probability of a working AI is worth the effort (as long as it's friendly, at least).

I think part of the problem is not that the audience is stupider than you imagine but that people sometimes use different techniques to learn the same thing. Your explanation may seem obvious to you but confuse everyone else while an alternative explanation that you would have difficulty understanding yourself would be obvious to others.

One example of this would be that some people learn better by concrete examples while others learn better by abstract ones.

I have an idea for how this problem could be approached:

any sufficiently powerful being with any random utility-function may or may not exist. It is perfectly possible that our reality is actually overseen by a god that rewards and punishes us for whether we say an even or odd number of words in our life or something equally arbitrary. The likelyhood of the existence of each of these possible beings can be approximated using Solomonoff induction.

I assume that most simulations run by such hypothetical beings wherein we could find ourselves in such a situation would be run by beings who either (1) have no interest in us at all (in which case the Mugger would most likely be a human), (2) are interested in an entirely unpredictable thing resulting from their alien culture, or (3) are interacting with us purely to run social experiments. After all, they would have nothing in common with us and we would have nothing they could possibly want. It would therefore, in any case, be virtually impossible to guess at their possible motivations, as it would be a poorly run social experiment if we could (assuming option three is true).

I would now argue that the existence of Pascal's Mugger does not influence the probability of the existence of a being that would react negatively (for us) to us not giving the 5$ anymore than it influences the probability of the existence of a being with an opposite motivation. The Mugger is equally likely to punish you for being so gullible as he is to punish you for not giving money to someone who threatens you.

Of course none of this takes into consideration how likely the various possible beings are to actually carry out their threat, but that doesn't change anything important about this argument, I think.

In essence, my argument is that such powerful hypothetical beings can be ignored because we have no real reason to assume they have a certain motivation rather than the opposite. Giving the Mugger 5$ is just as likely to save us as shooting the Mugger in the face is. Incidentally, adopting the later strategy will greatly reduce the chance that somebody actually tries to do this.

I realize that this argument seems kind of flawed because it assumes that it really is impossible to guess at the being's motivation but I can't see how this could be done. It's always possible that the being just wants you to think that it wants x, after all. Who can tell what might motivate a mind that is large enough to simulate our universe?

Evolutionary arguments about disease are difficult. Make sure your argument does not explain too much: vitamin deficiencies are real!

I didn't read through everything but if I understand this theory correctly it has a huge flaw:

Evolution would have countered it.

The proposed causes for diseases would be very easy for evolution to "cure". Therefore these diseases wouldn't exist if the theory was correct.

They just bicker endlessly about uncertainty. "can you really know that 1+1=2?".

I agree with you that I don't think a AGI would have the same problems humans have with the concept of truth. However, what you described is neither the issues philosophers raise nor the sorts of big-universe issues the AI might get stuck on.

If we don't define "optimal" properly it should be able to find a suitable definition on its own by imagining what we might have meant.

But it wouldn't want to. If we mistakenly define 'optimal' to mean 'really good at calculating pi' then it won't want to change itself to aim for our real values. It would realise that we made a mistake, but wouldn't want to rectify it, because the only thing it cares about is calculating pi, and helping humans isn't going to do that.

You're broadly on the right track; the idea of CEV is that we just tell the AI to look at humans and do what they would have wanted it to do. However, we have to actually be able to code that; it's not going to converge on that by itself.

Hello Less Wrong.

I am 19 years old and have been interested in philosophy since I was 13. Today, I am interested in anything that has to do with intelligence, such as psychology and AI and rationality.

I believe in the possibility of the technological singularity and want to help make it happen.

I hope that the complex and unusual ways of thinking that I have taught myself over the last years while philosophizing will allow me to tackle this problem from directions other people have not yet thought of, just like they enabled me to manipulate my own psyche in limited ways, such as turning off unwanted emotions.

I am currently studying computer science in the first semester with the goal of specializing in AI later.

The AI is smarter than us and understands human psychology. If we don't define "optimal" properly it should be able to find a suitable definition on its own by imagining what we might have meant. If that turns out to be wrong, we can tell it and it comes up with an alternative.

First you have to tell the machine to do that. It isn't trivial. The problem is not with the definition of "optimal" itself - but with what function is being optimised.

The idea that someone else would be able to build a superintelligence while you are teaching yours seems kind of far-fetched.

Well not if you decide to train it for "a long time". History is foll of near-simultaneous inventions being made in different places. Corporate history is full of close competition. There are anti-monopoly laws that attempt to prevent dominance by any one party - usually by screwing with any company that gets too powerful.

Why would the AI be evil?

Bugs, maybe

Intentions don't develop on their own. "Evil" intentions could only arise from misinterpreting existing goals.

While you are asking it to come up with a solution, you have its goal set to what I said in the original post:

Have you? Are you talking about a human level AI. Asking or commanding a human to do something doesn't set that as their one an onyl goal. A human reacts according to their existing goals:they might complyhl, refuse or subvert the command.

"the temporary goal to always answer questions thruthfully as far as possible while admitting uncertainty"

Why would it be easier to code in "be truthful" than "be friendly"?