Comment author: 30 March 2017 07:11:19PM *  0 points [-]

What is your definition of UDT?

If you use the definition given in this paper, then that definition of A is not UDT, simply because it is not guaranteed to "[find] a function mapping sense data to actions that will maximize the utility of outcomes weighted by the probability that the outcome will be caused by the procedure returning that mapping."

It is implicit in the definition that the agent must be able to choose from a set of counter-factual action-selection-functions {f: S -> A}, because it is in this choice the optimisation happens.

P can only be a valid proof that a is the best action exactly when a is the only option, because all other actions where excluded, by the use of non-counter-factual logic.

I know that counter-factual calculations are problematic. But if you take the counter-factual-nes out of UDP, then there are literally nothing left to the definition. Without the counter-factual-nes, everything goes. Any choice is the optimal one, because it was the only one, because that was the choice that actually happened.

## Suggested solution to The Naturalized Induction Problem

2 10 December 2016 12:21AM

This post is an answer to: http://intelligence.org/files/RealisticWorldModels.pdf

> In Solomonoff’s induction problem, the agent and its environment are fundamentally separate processes, connected only by an observation channel.  In reality, agents are embedded within their environment; the universe consists of some ontologically continuous substrate (atoms, quantum fields) and the “agent” is just a part of the universe in which we have a particular interest. What, then, is the analogous prediction problem for agents embedded within (and computed by) their environment?
> This is the naturalized induction problem , and it is not yet well understood. A good formalization of this problem, on par with Solomonoff’s formalization of the computable sequence induction problem, would represent a significant advance in the theory of general reasoning.

In Solomonoff’s induction, an algorithm learns about the world (modeled as a Turing machine) from observations (modeled as output from that Turing machine). Solomonoff’s induction is uncomputable, however there are computable approximations.

1) Suggestion of agent design

Consider an agent fully embedded in an environment. Every turn, the agent receves one bit of observation and can preform one bit of action. The design we propose for this agent comes in two steps, learning and deciding.

1.1) Learning:
The agent models the entire wrold, including the agent it self, as one unknown, output only Turing machine. Both observations and the agents own actions, are seen as world outputs. The agent calculate the probability distribution over hypotheses for the entire world, using Solomonoff’s induction (computable approximation).

This suggestion completely removes any boundary between the agent and the rest of the world, in the agents wold model.

1.2) Deciding
The agent must also have a decision proses for choosing what action to take. Because the agent model its own actions as deterministic outputs from a complete wold model, we can not use the decision procedure used by Hutter’s AXIX. This is not just a problem of counterfactuals. Our agents internal world model has no input channel.

Instead we suggest the flowing: For each available action, the agent calculate the expected utility, conditional on observing itself preform that action. The agent chooses the action that gives the highest expected utility. Alternatively, the agent choses semi-randomly, with higher probability for actions that results in higher expected utility.

An advantage with this agent design is that the decision process does not have to care about sequences of actions. Instead different possible future actions are accoutered for by separate wold hypotheses. Further more, this naturally takes in to account situations where the agent looses control over its own actions, e.g. if it brakes down.

2) Scoring agents

Solomonoff’s induction is the formal optimal solution to an associated scoring rule. Having a such a scoring rule is useful for testing how well approximation do. (Right?)

I don’t know what the associated scoring rule would be for this suggestion. Sorry :(

3) Measuring utility

The decision proses is based on the agent being able to detect utility in any given world hypothesis. This is a very hard problem, which we do not attempt to solve here.

Comment author: 10 December 2016 12:15:31AM 0 points [-]

What do you mean by "our collective epistemology"?

Comment author: 15 August 2016 05:24:40PM 0 points [-]

A friend of mine is interested in reading this book, but would prefer a printed copy. Is there any chance that this book will be published any time soon?

In response to comment by on Trying to Try
Comment author: 14 August 2016 09:00:36AM 0 points [-]

Do you want to help? Any idea of a test you could give me?

Most tests I could give you would would result in you trying to find the right answer and thus not test intuitive language usage. If you had a corpus of English text you wrote previously you could search it for "try" and get the first X examples. Then why could analyse what you meant with the word try.

But I think I can work with the rest of your post.

This means that I can try my best at something, and you can still try harder, if you have more resources that can be invested.

This suggest that investing more resources mean trying harder.

In cases where investing more resources means that success is less likely that notion of trying harder isn't optimization for a goal.

The woman who's playing hard to get isn't "trying". She isn't investing resources. She might still use the strategy that produces the best results.

In the case of the hypnosis effect of forgetting the numbers, that's not something I can achieve while trying to optimize for it. For me that seminar was a reference experience. I sat there and knew that I can only achieve the goal if I would stop trying to optimize for it. The fact that I really wanted to optimize for it and succeed only made it worse.

Investing resources and optimizing is different from doing what's necessary.

Sometimes "Just be yourself" would be good advice if the answer person could accept it*, because it stops the optimization and the trying that are the biggest problem.

*In practice people can't accept it so it usually isn't effective advice.

In response to comment by on Trying to Try
Comment author: 15 August 2016 12:06:13PM 0 points [-]

This means that I can try my best at something, and you can still try harder, if you have more resources that can be invested.

This suggest that investing more resources mean trying harder.

Yes, I just said so

Amount of trying X = How much time, money and other resources one is spending directly on optimizing for X.

But only if the added resources actually goes towards optimizing for winning. More precisely: If and only if I think that adding more resources will improve my expected outcome, then adding more resources, is trying harder.

I know what you mean with the hypnosis, my experience was very similar. But I did less post analysis than you.

I am not going to get in to exactly why I hate the advise "Be yourself", because it is a bit too personal and also off topic. But because I thought it was such a terrible advise, and why would anyone say that, I did some asking and thinking. Next time you are giving advise, Instead of saying "Be yourself", say "Focus on others". As you have already realized, saying "Be yourself" is telling people what not to do, which is not helpful. So tell them what to do instead. The best way to avoid doing X is to do Y instead, and there are extremely few situations where there are no possible Y to focus on. Mediation and trying to be hypnotized are the only examples I can think of, and even in mediation instructions, you are toled to focus on you breathing, or something, because doing nothing is too hard. But in most situations there are things you can focus you attention and efforts on, that are actually useful, and not just an artificial distraction. The circumstance where "Be yourself", usually pop up is when someone needs advise on how to do a good impression on an other person (date, interview for a job, etc). In these situations, a good choice is to focus on the other person, to get to know them.

In response to comment by on Trying to Try
Comment author: 12 August 2016 07:43:21PM 1 point [-]

I think for most people if you ask them to define what "try" means they will tell you that it's about putting in effort to achieve a goal. Emprirically that's however doesn't describe well the circumstances in which they use the word.

Especially on LW it might be possible that you actually don't wouldn't describe the manager who works 80 hours as trying to do his best at his job, but what you said doesn't make me confident that's the case.

I was at a hypnosis seminar where one of the exercises is about temporily forgetting numbers. There no mental action that you can do where you exert effort that gets you to forget the numbers but if you are in a mental state where you don't try and follow the instructions of the hypnotherapist you will temporarily forget the numbers.

At the end of the seminar I think of roughly 20 people there were two for which it didn't work. It didn't work for me because I wanted to have the effect happen and therefore I couldn't let go enough to stop trying to make it work. There was another person who happened to be a professional hypnotherapist for whom the same was true.

The mental state of just working towards a goal and not putting in any effort isn't easy to achieve.

In response to comment by on Trying to Try
Comment author: 13 August 2016 08:42:27PM *  0 points [-]

I agree that, from your stand point, you are correct in not entirely trusting me, when I claim to know my own brains working, in the case of this single word. And that is ok.

I wrote my fist post out of frustration over this way of interpreting "try":

I am rather offended by the the thought that when I say, "I am going to try", some one might interpret that as "I am going to try to try", or even "I am going to pretend to try". Because that was not what I said, and it was defiantly not what I meant. When I say "I am going to try", it means that I will put extra effort in to the task, just because I am aware of the risk of failure.

I usually succeed in keeping my rants of the Internet, but not always. Sorry about that, and for getting unnecessarily defensive at your responses.

As said, I am ok with you doubting me on weather I know my own brains working, in the case of this single word. But it would be fun if I could convince you. Do you want to help? Any idea of a test you could give me?

Regarding your example with Bob and Dave. How do I think is trying hardest? I do not know. To judge this, I would need to know the reasons for why they are doing what they are doing.

I have not yet defined how I want to measure the amount of trying. I have an intuitive idea, but it is less prices than my concept of trying. When I try to formalize my thoughs I get something like this:

Try X = Optimizing for X, usually given some constraints (e.g. unacceptable actions or risks, limited amount of time, money and other resources, that one is willing to spend on the try)

Amount of trying X = How much time, money and other resources one is spending directly on optimizing for X.

Trying ones best = Optimizing for X

Additionally, all the optimizations happens in the real world. Aside from deliberate constraints, there are always the real constraints of the real world, including how smart one is. (Edit: Shit, do I run in to the problem with determinism here? It should not matter, but I am not entirely sure. I need think more about this.)

This means that I can try my best at something, and you can still try harder, if you have more resources that can be invested. I expect that this sounds odd to you, but it actually lines up nicely with my intuition.

Comment author: 13 August 2016 07:09:34PM 2 points [-]

I just finished listening to the Audiobook version of Rationality: From AI to Zombie. Lots of thanks to Yudkowsky and everyone else that was involved in making this book and the audio book. I do not know who the reader of the audio book is, but thanks all the same.

I am writing this comment as my way of prizing this book. I will try to summarize what I have personalty learned from it, in the hope that someone who was involved, will read this post and fell some pride in having helped me in my self improvement. But I am also writing this comment because I just want to express my thoughts after finishing the book.

I have not have any major change of mind, but I have several minor ones, which might very well continue to grow.

Listening to Yudkowsky's words have made me more confident, because he is saying many things that I already intuitively knew, but I could not properly explain it my self, and could therefore not be sure I was right. I am still not 100% certain I am right, but I am more confident, and I believe that this is a good thing. Smart people should be confident. No, this is not hind site bias, because:

• I did not allays instantly agree, so I do know the difference.
• I been actively introspecting since I was 12, so I know most of my brains tricks.

I never set out to be a rationalist. I don't even remember having a pre-LessWrong concept for the word "rationalist". There where just, correct thinking and in-correct thinking, and obviously correct thinking is the way that systematically leads you to the truth, because how else would you measure correctness. Maybe this saved me from falling in to some of the rationalist tropes that Yudkovsky warns about. Or maybe I avoided them because I have read to little science fiction. Or maybe it was because I looked at these types of tropes and saw an author who kinged to the, obviously wrong, but warm and fussy, idea that every human has the same number of skill points.

I wonder who setts out to be rational, with out having something specific they need rationality for. Maybe the same kind of people that identifies as an atheist? I am an atheist, but I don't identify as such, because in my country, this is mostly a non issue.

I found LessWrong because my new boyfriend encouraged me to read here, and I actually got through the book, because I like audiobooks.

The pre-LessWrong me was a truth seeker, and as such, I though a lot about the way as applied to truth seeking. I had a crisis of faith, a several years a go, questioning the validity of science. But never really though about applying systematic reasoning to decision under uncertainty. When, in my past, I was confronted with a decision, which I did not know how to reason out, I used to deliberately hand over the decision to my feelings. Because, I reasoned, if I don't know what is right anyway, I might as well save me the fight of going against my impulses. I hope that I can use what I have learned here to do better.

An other thing I have realized is that I am such a pushover for perceived social norms. I have notice a significant mental shift in my brain, just from having some one in my ear, who casually mentions many words and cryonics, as if these where the most normal things in the world. Intellectually I was already convinced, I already knew the right answer before listening to the book, but I still needed the extra nagging, to get all of may brain on board with it. I think that this has been the single most important insight I got from the book.

One reason I have not tried to develop the art of rational decision making before, is that I knew that I was not strong enough to counter my emotional preferences. But I was wrong. I now have one, systematically applicable self hack, and probably there are more out there to find. I have hope to be able to take charge of my motivation, and I have reasons to fight for control.

Current me is an aspiring effective altruist. I do not strive to be a perfect altruist, because I do have some selfish preferences, that I do not expect to go away. But I am going to get my ass out of the comfortable bubble of I can''t do anything anyway, and do something. Though I have not decided yet if I am going to take the path of earn to give, or if I should get directly involved in some project my self. I am looking in to both ways.

Finlay, here is one my favorite quotes from the book:

I pause. “Well…” I say slowly. “Frankly, I’m not entirely sure myself where this ‘reality’ business comes from. I can’t create my own reality in the lab, so I must not understand it yet. But occasionally I believe strongly that something is going to happen, and then something else happens instead. I need a name for whatever-it-is that determines my experimental results, so I call it ‘reality’. This ‘reality’ is somehow separate from even my very best hypotheses. Even when I have a simple hypothesis, strongly supported by all the evidence I know, sometimes I’m still surprised. So I need different names for the thingies that determine my predictions and the thingy that determines my experimental results. I call the former thingies ‘belief’, and the latter thingy ‘reality’.”

In response to comment by on Trying to Try
Comment author: 11 August 2016 09:11:29AM 0 points [-]

There's the classic example of "don't try to think of a pink elephant". Most people you give that task will exert effort into not thinking of a pink elephant but that effort won't lead to them not thinking of a pink elephant.

To my best understanding, hypnosis bypasses the conscious decision center of the brain, which would explain why there is no trying.

There's trying. The person often does tense up their arm. It's just that the arm doesn't move as other muscles hold the arm in place.

In hypnosis you take certain metacognition away. If you tell someone to try they just try and exert effort but they don't work towards a goal if you don't give them a goal.

In addition to hypnosis the Alexander Technique is a system for movement where having a clear goal for movement and not trying to move is an important concept. It leads to people moving with less tension and more ergonomical.

I think in a variety of contexts where the effects of mental states matter naive people engage in effort when you tell them to try but not necessarily effort that works effectively towards a goal.

To move again to a more general level, Bob the manager who works 80 hours per week and sleeps 4 hours per day is trying really hard to do a good job. Certainly more than Dave who works 40 hours per week and sleeps 8 hours. It's certainly possible that Bob is more productive than Dave as a result of putting in more effort but it isn't certain. Maybe he spends too much time in busy work and isn't rested enough to concentrate on what matters.

In response to comment by on Trying to Try
Comment author: 12 August 2016 05:23:58PM 0 points [-]

Ok, you, and possibly most people, associate the word "try", only with putting in effort. For me "try" means something different (as I have tried to explain), because your try, is not a natural concept for me. I will just have to keep this difference in mind in future conversation, when ever it is important for the communication.

In response to Church vs. Taskforce
Comment author: 11 August 2016 11:39:48PM *  0 points [-]

I already live in that post religious world. Yes, we do have religion, of course. But to most people here, religion is a private matter. There are people here that goes to church regularly, and for whom this is their prime community, but those are a small minority.

Instead our communities are the workplace or organized hobbies, or just fiends that one drinks alcohol with regularly. My parents prime community (except family) is the local Orienteering club. My self, I would not say that I have a community right now. Possibly my flat mates, if five people can be called a community. Mostly I have a network of friends, and I think that this is the most common form of "community" in my country.

This kind of network of workplaces, relatives, hobbies, friends, means that many people have one foot here and one there, which makes weaker communities. And there are people that just do not have a community.

But there are some efforts to patch the holes. In my neighborhood there are something called "Fika common", which is a social meat up where everyone is invited. Anyone who wants to, brings cake, or similar, and share with everyone else. I rely like that this exist, but I have not been there, because I don't like strangers much.

In response to comment by on Trying to Try
Comment author: 10 August 2016 11:37:41AM 1 point [-]

Do other people really work like that? I thought that the thing with the Yoda quote was that the Force only works if you 100% believe in it. Unlike the nature of our wold, Force do care about your state of mind, and not only about your actions. But we do not live in that word.

We do live a world where if you tell someone in hypnosis to move their arm up they will move their arm up but if you tell them to try to move their arm up they won't move their arm up.

When I say "I am going to try", it means that I will put extra effort in to the task, just because I am aware of the risk of failure.

Yes, it general means that you put effort into the task. But effort doesn't always mean effective action.

In response to comment by on Trying to Try
Comment author: 10 August 2016 06:59:43PM 0 points [-]

When I say "I am going to try", it means that I will put extra effort in to the task, just because I am aware of the risk of failure.

Yes, it general means that you put effort into the task. But effort doesn't always mean effective action.

Let me clarify even more. To me, the word "try" referees to the conscious process of optimizing for succuss, with or with out constraints. Constraints, may be that I only want to put so much effort in to the problem, or that I am not willing to take certain risks, etc.

Also, to me the word "do" means that I intend to preform an action, that is trivial enough so that I do not feel a need to optimize.

For example, right now I am trying to explain my self. I am optimizing this text for clarity, under the ill defined, but very real constraint, that I am only willing to put in a limited amount of effort. But I am doing the accrual typing on the keyboard. I do not try to hit the right keys. Hitting keys are trivial, I don't it flawlessly, but I do it well enough not to bother to optimize the effort thunder. Trying is more costly than doing, in my meaning of the words.

I never ever tried to try. I am not really sure what that would mean even using my meaning of the word "try". I try or I do not try, there is no try to try. How ever, I did just spend some time trying to try to try, and failed.

When I say, I failed to try to try, I mean that, using my meaning of the word "try". The action that Yudkovsky call "trying to try", I would call "pretending to try".

However, trying, as I use the word, does not necessarily mean trying hard. Some times a solution is worth the effort if and only if it is cheap. In this situation, if I do not know the difficulty of the problem, I will give it a light try. It can be, thinking about a problem for X minutes, and if I did not make any progress, I drop it. But that is still a try. During those minutes I optimist to win. Because I do want that win. I just don't want it very strongly.

We do live a world where if you tell someone in hypnosis to move their arm up they will move their arm up but if you tell them to try to move their arm up they won't move their arm up.

That is interesting. Do you know the underlying reason for this? I am guessing that it has to do with conscious and non conscious actions. Trying is a conscious effort, but doing is mostly not conscious. To my best understanding, hypnosis bypasses the conscious decision center of the brain, which would explain why there is no trying. But don't trust me on this, because I know very little about hypnosis, and I am very good att making up explanations on the spot.

I was to a workshop once that involved hypnosis. It turned out that I am not very receptive to hypnosis. I am not saying I a immune, but it just did not work on me that time, and it did work on most of the others. I was really disappointed.

I am not convinced that there is a strong connection between the two mental phenomena, hypnosis and pretending to try, but the fact that my mind refuses both of them is evidence in this direction.

View more: Next