Wiki Contributions

Comments

Agent cares about his goals, and ignores the objective norms.

If you know what you must do

There is no "must", there is only "should". And even that only assuming that there is an objective norm -- otherwise there is even no "should", only want.

Again, Satan in Christianity. Knows what is "right", does the opposite, and does it effectively. The intelligence is used to achieve his goals, regardless of what is "right".

Intelligence means being able to figure out how to achieve what one wants. Not what one "should" want.

Imagine that somehow science proves that the goal of this universe is to produce as many paperclips as possible. Would you feel compelled to start producing paperclips? Or would you keep doing whatever you want, and let the universe worry about its goals? (Unless there is some kind of God who rewards you for the paperclips produced and punishes if you miss the quota. But even then, you are doing it for the rewards, not for the paperclips themselves.)

The AGI (or a human) can ignore the threats... and perhaps perish as a consequence.

General intelligence does not mean never making a strategic mistake. Also, maybe from the value perspective of the AGI, doing whatever it was doing now could be more important than surviving.

Makes sense, but wouldn't this also result in even fewer replications (as a side effect of doing less superfluous work)?

I was going to ask for interesting examples. But perhaps we can do even better and choose examples with the highest value of... uhm... something.

I am just wildly guessing here, but it seems to me that if these features are somehow implied by the human text, the ones that are "implied most strongly" could be the most interesting ones. Unless they are just random artifacts of the process of learning.

If we trained the LLM using the same text database, but randomly arranged the sources, or otherwise introduced some noise, would the same concepts appear?

Are you perhaps using "intelligence" as an applause light here?

To use a fictional example, is Satan (in Christianity) intelligent? He knows what is the right thing to do... and chooses to do the opposite. Because that's what he wants to do.

(I don't know Vatican's official position on Satan's IQ, but he is reportedly capable of fooling even very smart people, so I assume he must be quite smart, too.)

In terms of artificial intelligence, if you have a super-intelligent program that can provide answers to various kinds of questions, for any goal G you can create a robot that calls the super-intelligent program to figure out what actions are most likely to achieve G, and then performs those actions. Nothing in the laws of physics prevents this.

Answer by ViliamMar 25, 202420

Orthogonality thesis is not about the existence or nonexistence of "objective norms/values", but whether a specific agent could have a specific goal. The thesis says that for any specific goal, there can be an intelligent agent that has the goal.

To simplify it, the question is not "is there an objective definition of good?" where we probably disagree, but rather "can an agent be bad?" where I suppose we both agree the answer is clearly yes.

More precisely, "can a very intelligent agent be bad?". Still, the answer is yes. (Even if there is such thing as "objective norms/values", the agent can simply choose to ignore them.)

The first two points... I wonder what is the relation between "prestigious university" and "quality of your peers". Seems like it should be positively correlated, but maybe there is some caveat about the quality not being one-dimensional, like maybe rich people go to university X, but technically skilled people to university Y.

The third point, I'd say be aware of the distinction between the things you care about, and the things you have to do for bureaucratic reasons. There may or may not be an overlap between the former and the school lessons.

The fourth and seventh points are basically: some people give bad advice; and for anything you could possibly do, someone will find a rationalization why that specific thing is important (if everything else fails, they can say it makes you more "well-rounded"). But "skills that develop value" does not say how to choose e.g. between a smaller value now or a greater value in future.

The fifth point -- depends on what kind of job/mentor you get. It could be much better or much worse that school, and it may be difficult to see the difference; there are many overconfident people giving wrong advice in the industry, too.

The sixth point -- clearly, getting fired is not an optimal outcome; if you do not need to complete the school, what are you even doing there?

I think we probably agree on how far the existing system is from the ideal. I wanted to point at the opposite end of the scale as a reminder that we are even further away from that.

When I was at the first grade of elementary school, they tried to teach us about "sets", which mostly meant that instead of "two plus two equals four" the textbook said "the union of a set containing two elements and another set containing two elements, has four elements". In hindsight I see this was a cargo-cultish version of set theory, which probably was very high-status at that time. I also see that from the perspective of set theory as the set theorists know it, this was quite useless. Yes, we used the word "set" a lot, but it had little in common with how the set theorists think about sets. Anyway, we have learned addition and subtraction successfully, albeit with some extra verbal friction.

Compared to that, when I tried to learn something in my free time as a teenager, people around me recommended me to read books written by Däniken, the Silva method of mind control, Moody's Life after Life, religious literature, books on meditation, and other shit. I have spent a lot of time practicing "altered states of consciousness", because (from the perspective of a naive teenager who believed that the adults around him are not utter retards, and the people they consider high-status are not all lying scumbags) it seemed like a very efficient intervention. I mean, if you get the supernatural skills first, they will give you a huge multiplier to everything you try doing later, right? Haha, nope.

So while I hate school with a passion, as many people on Less Wrong do, the alternative seems much worse. Even the books I study in my free time now were often written in the context of the educational system, or by the people employed by the educational system.

I don't trust societal consensus at all. Look at the YouTube videos about quantum physics, 99% of them is some crap like "quantum physics means that human mind had a mystical power over matter". Even if you limit yourself to seemingly smart people, half of them believe that IQ isn't real because Nassim fucking Taleb said so. Half of the popular science does not replicate.

Load More