Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: denimalpaca 24 March 2017 08:10:02PM 0 points [-]

My definition of utility function is one commonly used in AI. It is a mapping of states to a real number: u:E -> R where u is a state in E (the set of all possible states), and R is the reals in one dimension.

What definition are you using? I don't think we can have a productive conversation until we both understand each other's definitions.

Comment author: TheAncientGeek 25 March 2017 11:38:06AM 0 points [-]

I'm not using a definition, I'm pointing out that standard arguments about UFs depend on ambiguities.

Your definition is abstract and doens't capture anything that an actual AI could "have" -- for one thing, you can't compute the reals. It also fails to capture what UF's are "for".

Comment author: denimalpaca 22 March 2017 06:00:08PM 0 points [-]

Is there an article that presents multiple models of UF-driven humans and demonstrates that what you criticize as contrived actually shows there is no territory to correspond to the map? Right now your statement doesn't have enough detail for me to be convinced that UF-driven humans are a bad model.

And you didn't answer my question: is there another way, besides UFs, to guide an agent towards a goal? It seems to me that the idea of moving toward a goal implies a utility function, be it hunger or human programmed.

Comment author: TheAncientGeek 24 March 2017 12:35:09PM *  0 points [-]

Is there an article that presents multiple models of UF-driven humans and demonstrates that what you criticize as contrived actually shows there is no territory to correspond to the map?

Rather than trying to prove the negative, it is more a question of whether these models are known to be useful.

The idea of mulitple or changing UFs suffers from a problem falsifiability, as well. Whenever a human changes their apparent goals, that's a switch to another UF, or a change in UF? Reminiscent of ptolemaic epicycles, as Ben Goerzel says.

And you didn't answer my question: is there another way, besides UFs, to guide an agent towards a goal? It seems to me that the idea of moving toward a goal implies a utility function, be it hunger or human programmed.

Implies what kind of UF?

If you are arguing tautologously that having a UF just is having goal directed behaviour, then you are not going to be able to draw interesting conclusions. If you are going to define "having a UF broadly, then you are going to have similar problems, and in particular the problem that "the problem of making an AI safe simplifies to the problem of making its UF safe" only works for certain, relatively narrow, definitions of UF. In the context of a biological organism, or an artificial neural net or deep learning AI, the only thing "UF" could mean is some aspect of its functioning that is entangled with all the others. Neither a biological organism, nor an artificial neural net or deep learning AI is going to have a UF that can be conveniently separated out and reprogrammed. That definition of UF only belongs in the context of GOFAI or symbolic programming.

There is no point in defining a term broadly to make one claim come out true, if it was is only an intermediate step towards some other claim, which doesn't come out as true under the broad definition.

Comment author: tristanm 22 March 2017 06:28:32PM 0 points [-]

Rationalists (Bay area type) tend to think of what they call Postmodernism[*] as the antithesis to themselves, but the reality is more complex. "Postmodernism" isn't a short and cohesive set of claims that are the opposite of the set of claims that rationalists make, it's a different set of concerns, goals and approachs.

Except that it does make claims that are the opposite of the claims rationalists make. It claims that there is no objective reality, no ultimate set of principles we can use to understand the universe, and no correct method of getting nearer to truth. And the 'goal' of postmodernism is to break apart and criticize everything that claims to be able to do those things. You would be hard pressed to find a better example of something diametrically opposed to rationalism. (I'm going to guess that with high likelihood I'll get accused of not understanding postmodernism by saying that).

And what's worse is that bay area rationalism has not been able to unequivocally define "rationality" or "truth". (EY wrote an article on the Simple idea of Truth, in which he considers the correspondence theory, Tarki's theory, and a few others without resolving on a single correct theory).

Well yeah, being able to unequivocally define anything is difficult, no argument there. But rationalists use an intuitive and pragmatic definition of truth that allows us to actually do things. Then what happens is they get accused by postmodernists of claiming to have the One and Only True and Correct Definition of Truth and Correctness, and of claiming that we have access to the Objective Reality. The point is that as soon as you allow for any leeway in this at all (leeway in allowing for some in-between area of there being a true objective reality with 100% access to and 0% access to), you basically obtain rationalism. Not because the principles it derives from are that there is an objective reality that is possible to Truly Know, or that there are facts that we know to be 100% true, but only that there are sets of claims we have some degree of confidence in, and other sets of claims we might want to calculate a degree of confidence in based on the first set of claims.

Bay area rationalism is the attitude that that sceptical (no truth) and relativistic (multiple truth) claims are utterly false, but it's an attitude, not a proof.

It happens to be an attitude that works really well in practice, but the other two attitudes can't actually be used in practice if you were to adhere to them fully. They would only be useful for denying anything that someone else believes. I mean, what would it mean to actually hold two beliefs to be completely true but also that they contradict? In probability theory you can have degrees of confidence that are non-zero that add up to one, but it's unclear if this is the same thing as relativism in the sense of "multiple truths". I would guess that it isn't, and multiple truths really means holding two incompatible beliefs to both be true.

If rationalist is to win over "postmodernism", then it must win rationally, by being able to demonstrate it's superioritiy.

Except that you can't demonstrate superiority of anything within the framework of postmodernism. Within rationalism it's very easy and straightforward.

I imagine the reason that some rationalists might find postmodernism to be useful is in the spirit of overcoming biases. This in and of itself I have no problem with - but I would ask what you consider postmodern ideas to offer in the quest to remove biases that rationalism doesn't offer, or wouldn't have access to even in principle?

Comment author: TheAncientGeek 22 March 2017 08:58:54PM *  1 point [-]

Except that it does make claims that are the opposite of the claims rationalists make. It claims that there is no objective reality, no ultimate set of principles we can use to understand the universe, and no correct method of getting nearer to truth.

Citation needed.

Well yeah, being able to unequivocally define anything is difficult, no argument there

On the other hand, refraining from condemning others when you have skeletons in your own closet is easy.

But rationalists use an intuitive and pragmatic definition of truth that allows us to actually do things. T

Engineers use an intuitive and pragmatic definition of truth that allows them to actually do things. Rationalists are more in the philosophy business.

It happens to be an attitude that works really well in practice,

For some values of "work". It's possible to argue in detail that predictive power actually doesn't entail correspondence to ultimate reality, for instance.

I mean, what would it mean to actually hold two beliefs to be completely true but also that they contradict?

For instance, when you tell outsiders that you have wonderful answers to problems X, Y and Z, but you concede to people inside the tent that you actually don't.

Except that you can't demonstrate superiority of anything within the framework of postmodernism

That's not what I said.

but I would ask what you consider postmodern ideas to offer in the quest to remove biases that rationalism doesn't offer, or wouldn't have access to even in principle?

There's no such thing as postmodernism and I'm not particularly in favour of it. My position is more about doing rationality right than not doing it all. If you critically apply rationality to itself, you end up with something a lot less elf confident and exclusionary than Bay Area rationalism.

Comment author: TheAncientGeek 22 March 2017 06:21:11PM 1 point [-]

The basic problem is the endemic confusion between the map, the UF as a way of modelling an entity, and the territory. the UF as an architectural feature that makes certain things happen.

It seems to you that entities with simple and obvious goal directed behaviour (as seen from the outside) have or need UFs, and entities that don't. don't. But there isn't a fixed connection between the way things seem from the outside, and the way they work.

From the outside, any system that succeeds in doing anything specialised can be thought of, or described as a relatively general purpose system that has been constrained down to a more narrow goal by some other system. For instance, a chess -playing system maybe described as a general purpose problem-solver that has been trained on chess. To say its UF defines a goal of winning at chess is the "map" view.

However, it might well be .. in terms of the territory ... in terms of what is going on inside the black box.. a special purpose system that has been specifically coded for chess, has no ability to do anything else, and therefore does not any kind of reward channel or training system to keep it focused on chess. So the mere fact that a system, considered from the outside as a black box, does some specific thing, is not proof that it has a UF, and therefore not a proof that anyone has succeeded in loading values or goals into its UF.

Taking an outside view of a system as possessing a UF (in the spirit or Dennett's "intentional stance") will only give correct predictions if everything works correctly. The essential point is that you need a fully accurate picture of what is going in inside a black box in order to predict its behaviour under all circumstances .. but pictures that are inaccurate in various ways can be good enough for restricted sets of circumstances.

Here's an analogy: suppose that machinery, including domestic appliances, were made of an infinitely malleable substance called Utlronium, say, and were constrained into some particular form, such a a kettle or toaster, by a further gadget called a Veeblefetzer. So long as a kettle functions as a kettle, I can regard it as Ultronium+Veeblefetzer ensemble. However, such ensembles support different counterfactuals to real kettles. For instance. if the veeblefetzer on my kettle fritzes it could suddenly reconfigure it into something else, a toaster or a spice rack -- but that is not possible for an ordinary kettle that is not made of Ultronium.

The converse case to an entity seeming to have a UF just because it fulfils some apparent purpose is an entity that seems not to have a UF because its behaviour is complex and perhaps seemingly random. A UF in the territory sense does not have to be simple, and a complex UF can include higher level goals, such as "seek variety" or "revise your lower level goals from time to time", so the lack of an obvious UF as judged externally does not imply the lack of a UF in the gold-standard sense of an actual component.

The actual possession of a UF is much more relevant to AI safety than being describable in terms of a UF. If an AI doesn't actually have a UF, you can't render it safe by fixing its UF.

Comment author: denimalpaca 21 March 2017 06:46:20PM 0 points [-]

Why would I give up the whole idea? I think you're correct in that you could model a human with multiple, varying UFs. Is there another way you know of to guide an intelligence toward a goal?

Comment author: TheAncientGeek 22 March 2017 02:57:47PM 2 points [-]

The basic problem is the endemic confusion between the map, the UF as a way of modelling an entity, and the territory. the UF as an architectural feature that makes certain things happen.

The fact that there are multiple ways of modelling humans as UF-driven, and the fact that they are all a bit contrived, should be a hint that there may be no territory corresponding to the map.

Comment author: ragintumbleweed 28 February 2017 05:55:22PM 1 point [-]

Forgive me, as I am brand new to LW. Where is it defined that an epistemic rationalist can't seek epistemic rationality as a means of living a good life (or for some other reason) rather than as a terminal goal? Is there an Académie française of rationalists that takes away your card if you use ER as a means to an end?

I'm working off this quote from EY as my definition of ER. This definition seems silent on the means-end question.

Epistemic rationality: believing, and updating on evidence, so as to systematically improve the correspondence between your map and the territory. The art of obtaining beliefs that correspond to reality as closely as possible. This correspondence is commonly termed "truth" or "accuracy", and we're happy to call it that.

This definition is agnostic on motivations for seeking rationality. Epistemic rationality is just seeking truth. You can do this because you want to get rich or get laid or get status or go to the moon or establish a better government or business. People's motivations for doing what they do are complex. Try as I might, I don't think I'll ever fully understand why my primate brain does it what it does. And I don't think anyone's primate brain is seeking truth for its own sake and for no other reasons.

Also, arguing about definitions is the least useful form of philosophy, so if that's the direction we're going, I'm tapping out.

But I will say that if the only people the Académie française of rationalists deems worthy of calling themselves epistemic rationalists are those with pure, untainted motivations of seeking truth for its own sake and for no other reasons, then I suspect that the class of epistemic rationalists is an empty set.

[And yes, I understand that instrumentality is about the actions you choose. But my point is about motivations, not actions.]

Comment author: TheAncientGeek 22 March 2017 12:38:43PM 0 points [-]

Forgive me, as I am brand new to LW. Where is it defined that an epistemic rationalist can't seek epistemic rationality as a means of living a good life (or for some other reason) rather than as a terminal goal?

From the wiki:-

Epistemic rationality is that part of rationality which involves achieving accurate beliefs about the world. ..... It can be seen as a form of instrumental rationality in which knowledge and truth are goals in themselves, whereas in other forms of instrumental rationality, knowledge and truth are only potential aids to achieving goals.

Comment author: tristanm 20 March 2017 10:54:00PM 4 points [-]

Should we expect more anti-rationalism in the future? I believe that we should, but let me outline what actual observations I think we will make.

Firstly, what do I mean by 'anti-rationality'? I don't mean that in particular people will criticize LessWrong. I mean it in the general sense of skepticism towards science / logical reasoning, skepticism towards technology, and a hostility to rationalistic methods applied to things like policy, politics, economics, education, and things like that.

And there are a few things I think we will observe first (some of which we are already observing) that will act as a catalyst for this. Number one, if economic inequality increases, I think a lot of the blame for this will be placed on the elite (as it always is), but in particular the cognitive elite (which makes up an ever-increasing share of the elite). Whatever the views of the cognitive elite are will become the philosophy of evil from the perspective of the masses. Because the elite are increasingly made up of very high intelligence people, many of whom with a connection to technology or Silicon Valley, we should expect that the dominant worldview of that environment will increasingly contrast with the worldview of those who haven't benefited or at least do not perceive themselves to benefit from the increasing growth and wealth driven by those people. What's worse, it seems that even if economic gains benefit those at the very bottom too, if inequality still increases, that is the only thing that will get noticed.

The second issue is that as technology improves, our powers of inference increase, and privacy defenses become weaker. It's already the case that we can predict a person's behavior to some degree and use that knowledge to our advantage (if you're trying to sell something to them, give them / deny them a loan, judge whether they would be a good employee, or predict whether or not they will commit a crime). There's already a push-back against this, in the sense that certain variables correlate with things we don't want them to, like race. This implies that the standard definition of privacy, in the sense of simply not having access to specific variables, isn't strong enough. What's desired is not being able to infer the values of certain variables, either, which is a much, much stronger condition. This is a deep, non-trivial problem that is unlikely to be solved quickly - and it runs into the same issues as all problems concerning discrimination do, which is how to define 'bias'. Is reducing bias at the expense of truth even a worthy goal? This shifts the debate towards programmers, statisticians and data scientists who are left with the burden of never making a mistake in this area. "Weapons of Math Destruction" is a good example of the way this issue gets treated.

We will also continue to observe a lot ideas from postmodernism being adopted as part of political ideology of the left. Postmodernism is basically the antithesis of rationalism, and is particularly worrying because it is a very adaptable and robust meme. And an ideology that essentially claims that rationality and truth are not even possible to define, let alone discover, is particularly dangerous if it is adopted as the mainstream mode of thought. So if a lot of the above problems get worse, I think there is a chance that rationalism will get blamed as it has been in the framework of postmodernism.

The summary of this is: As politics becomes warfare between worldviews rather than arguments for and against various beliefs, populist hostility gets directed towards what is perceived to be the worldview of the elite. The elite tend to be more rationalist, and so that hostility may get directed towards rationalism itself.

I think a lot more can be said about this, but maybe that's best left to a full post, I'm not sure. Let me know if this was too long / short or poorly worded.

Comment author: TheAncientGeek 22 March 2017 11:56:51AM *  0 points [-]

Postmodernism is basically the antithesis of rationalism, and is particularly worrying because it is a very adaptable and robust meme.

Rationalists (Bay area type) tend to think of what they call Postmodernism[*] as the antithesis to themselves, but the reality is more complex. "Postmodernism" isn't a short and cohesive set of claims that are the opposite of the set of claims that rationalists make, it's a different set of concerns, goals and approachs.

And an ideology that essentially claims that rationality and truth are not even possible to define, let alone discover, is particularly dangerous if it is adopted as the mainstream mode of thought.

And what's worse is that bay area rationalism has not been able to unequivocally define "rationality" or "truth". (EY wrote an article on the Simple idea of Truth, in which he considers the correspondence theory, Tarki's theory, and a few others without resolving on a single correct theory).

Bay area rationalism is the attitude that that sceptical (no truth) and relativistic (multiple truth) claims are utterly false, but it's an attitude, not a proof. What's worse still is that sceptical and relativistic claims can be supported using the toolkit of rationality. "Postmodernists" tend to be sceptics and relativists, but you don't have to be a "postmodernist" to be a relativist or sceptic. As non-bay-area, mainstream, rationalists understand well. If rationalist is to win over "postmodernism", then it must win rationally, by being able to demonstrate it's superioritiy.

[*] "Postmodernists" call themselves poststructuralists, continental philosophers, or critical theorists.

Comment author: denimalpaca 17 March 2017 05:51:57PM 0 points [-]

I think you're getting stuck on the idea of one utility function. I like to think humans have many, many utility functions. Some we outgrow, some we "restart" from time to time. For the former, think of a baby learning to walk. There is a utility function, or something very much like it, that gets the baby from sitting to crawling to walking. Once the baby learns how to walk, though, the utility function is no longer useful; the goal has been met. Now this action moves from being modeled by a utility function to a known action that can be used as input to other utility functions.

As best as I can tell, human general intelligence comes from many small intelligences acting in a cohesive way. The brain is structured like this, as a bunch of different sections that do very specific things. Machine models are moving in this direction, with the Deepmind Go neural net playing a version of itself to get better a good example.

Comment author: TheAncientGeek 20 March 2017 10:13:34PM 0 points [-]

You could model humans as having varying UFs, or having multiple UFs...or you could give up on the whole idea.

Comment author: Vaniver 19 March 2017 03:53:26AM 9 points [-]

Quoting myself from Facebook:

I think identifying the neoreactionaries with LessWrong is mostly incorrect. I know NRxers who found each other on LW and made side blogs, but that's because I know many more LWers than NRxers.

In the 2014 survey, only 2% of LWers called themselves neoreactionary, and I think that's dropped as they've mostly moved off LW to other explicitly neoreactionary sites that they set up. LW had a ban on discussing politics that meant there weren't any serious debates of NRx ideas. To the best of my knowledge, Moldbug didn't post on LW. It probably is the case that debiasing pushed some people in the alt-right direction, but it's still silly to claim it's the normal result.

Comment author: TheAncientGeek 19 March 2017 08:47:14PM 0 points [-]

2% of LWers called themselves neoreactionary,

That's compatible with a lot of neoreactionaries being lesswrongers.

To the best of my knowledge, Moldbug didn't post on LW.

I believe he posted on OB when EY was posting there.

Comment author: BiasedBayes 17 March 2017 02:17:18PM 0 points [-]

To state P is to imply "P is true". If you didn't think your theory was better, why state it?

Im not advocating some big grand theory of ethics but a rational approach to ethical problems given the values we have. I dont think its needed or even possible to solve some big general questions first.

Anyone else? Any number of people have stated theories. The Catholic Church The Protestant churches.Left wing politics. Right wing politics. ....etc etc etc.

In this discussion.

Anyone can state an object-level theory which is just the faith of their ancestors or whatever, and many do. However, you put yourself in a tricky position to do so when your theory boils down to "science solves it", because science is supposed to be better than everything else for reasons connected to wider rationality...it's supposed to be on the high ground.

Irrelevant. Given values we have there are better and worse approaches to ethical problems. The answer is not some lipservice slogan "science solves it " but to give an argument based on synthesized evidence we have related to that specific ethical problem. After this peers can criticise the arguments based on evidence.

Why? To support some claim about ethics? I haven't made any. To prove that it is possible?

Because you keep insisting that we have to solve some big ethical questions first. When asked repeatedly you try to specify by saying "closer you are solving them" but that does not really mean anything. That is just a mumbo-jumbo. Looking forward to that day when philosophers agree on general ethical theory.

an ethical system can be better or worse adapted to a society's needs, meaning there are better and worse ethical systems.(Strong ethical relativism is also false...we are promoting a central or compromise position along the realism-relativism axis).

How do you know which system is better or worse? Would you not rank and evaluate different solutions to ethical problems by actually researching the solutions using empirical data had and applying this thing called scientific method?

Comment author: TheAncientGeek 18 March 2017 04:38:08PM 1 point [-]

but a rational approach to ethical problems given the values we have. I dont think its needed or even possible to solve some big general questions first.

You need to understand the meta-level questions in order to solve the right problem in the right way. Applying science to ethics unreflectively, naively, has numerous potentional pitfalls. For instance, the pitfall of treating whatever intuitions evolution has given us as the last word on the subject.

The answer is not some lipservice slogan "science solves it " but to give an argument based on synthesized evidence we have related to that specific ethical problem.

Repeat three times before breakfast: science is value free. You cannot put together a heap of facts and come immediately to a conclusion about what is right and wrong. You need to think about how you are bridging the is-ought gap.

Looking forward to that day when philosophers agree on general ethical theory.

At least they see the need to. If you don't , you just end up jumping to conclusions, like the way you backed universalism without even considering an alternative.

Because you keep insisting that we have to solve some big ethical questions first.

I keep insisting that people think you can solve ethics with science need a meta ethical framework. The many people who have no ethical claims to make are not included.

How do you know which system is better or worse?

If you identify ethics as, in broad terms, fulfilling a functional role, then the answer to that questions is of the same general category as "is this hammer a good hammer". I am connecting ethical goodness to facts via instrumental goodness -- that is how I am approaching the is-ought gap.

ould you not rank and evaluate different solutions to ethical problems by actually researching the solutions using empirical data had and applying this thing called scientific method?

I am not saying : don't use empiricism, I am saying don't use it naively.

View more: Next