Previously: Raising the waterline, see also: 1001 PredictionBook Nights (LW copy), Techniques for probability estimates

Low waterlines imply that it's relatively easy for a novice to outperform the competition. (In poker, as discussed in Nate Silver's book, the "fish" are those who can't master basic techniques such as folding when they have a poor hand, or calculating even roughly the expected value of a pot.) Does this apply to the domain of making predictions? It's early days, but it looks as if a smallish set of tools - a conscious status quo bias, respecting probability axioms when considering alternatives, considering references classes, leaving yourself a line of retreat, detaching from sunk costs, and a few more - can at least place you in a good position. 

Dewey 2011 lays out the rules for one kind of agent with a mutable value system. The agent has some distribution over utility functions, which it has rules for updating based on its interaction history (where "interaction history" means the agent's observations and actions since its origin). To choose an action, it looks through every possible future interaction history, and picks the action that leads to the highest expected utility, weighted both by the possibility of making that future happen and the utility function distribution that would hold if that future came to pass.

We might motivate this sort of update strategy by considering a sandwich-drone bringing you a sandwich. The drone can either go to your workplace, or go to your home. If we think about this drone as a value-learner, then the "correct utility function" depends on whether you're at work or at home - upon learning your location, the drone should update its utility function so that it wants to go to that place. (Value learning is unnecessarily indirect in this case, but that's because it's a simple example.)

Suppose the drone begins its delivery assigning equal measure to the home-utility-function and to the work-utility-function (i.e. ignorant of your location), and can learn your location for a small cost. If the drone evaluated this idea with its current utility function, it wouldn't see any benefit, even though it would in fact deliver the sandwich properly - because under its current utility function there's no point to going to one place rather than the other. To get sensible behavior, and properly deliver your sandwich, the drone must evaluate actions based on what utility function it will have in the future, after the action happens.

If you're familiar with how wireheading or quantum suicide look in terms of decision theory, this method of deciding based on future utility functions might seem risky. Fortunately, value learning doesn't permit wireheading in the traditional sense, because the updates to the utility function are an abstract process, not a physical one. The agent's probability distribution over utility functions, which is conditional on interaction histories, defines which actions and observations are allowed to change the utility function during the process of predicting expected utility.

Dewey also mentions that so long as the probability distribution over utility functions is well-behaved, you cannot deliberately take action to raise the probability of one of the utility functions being true. But I think this is only useful to safety when we understand and trust the overarching utility function that gets evaluated at the future time horizon. If instead we start at the present, and specify a starting utility function and rules for updating it based on observations, this complex system can evolve in surprising directions, including some wireheading-esque behavior.

The formalism of Dewey 2011 is, at bottom, extremely simple. I'm going to be a bad pedagogue here: I think this might only make sense if you go look at equations 2 and 3 in the paper, and figure out what all the terms do, and see how similar they are. The cheap summary is that if your utility is a function of the interaction history, trying to change utility functions based on interaction history just gives you back a utility function. If we try to think about what sort of process to use to change an agent's utility function, this formalism provides only one tool: look out to some future time horizon, and define an effective utility function in terms of what utility functions are possible at that future time horizon. This is different from the approximations or local utility functions we would like in practice.

If we take this scheme and try to approximate it, for example by only looking N steps into the future, we run into problems; the agent will want to self-modify so that next timestep it only looks ahead N-1 steps, and then N-2 steps, and so on. Or more generally, many simple approximation schemes are "sticky" - from inside the approximation, an approximation that changes over time looks like undesirable value drift.

Common sense says this sort of self-sabotage should be eliminable. One should be able to really care about the underlying utility function, not just its approximation. However, this problem tends to crop up, for example whenever the part of the future you look at does not depend on which action you are considering; modifying to keep looking at the same part of the future unsurprisingly improve the results you get in that part of the future. If we want to build a paperclip maximizer, it shouldn't be necessary to figure out every single way to self-modify and penalize them appropriately.

We might evade this particular problem using some other method of approximation that does something more like reasoning about actions than reasoning about futures. The reasoning doesn't have to be logically impeccable - we might imagine an agent that identifies a small number of salient consequences of each action, and chooses based on those. But it seems difficult to show how such an agent would have good properties. This is something I'm definitely interested in.

One way to try to make things concrete is to pick a local utility function and specify rules for changing it. For example, suppose we wanted an AI to flag all the 9s in the MNIST dataset. We define a single-time-step utility function by a neural network that takes in the image and the decision of whether to flag or not, and returns a number between -1 and 1. This neural network is deterministically trained for each time step on all previous examples, trying to assign 1 to correct flaggings and -1 to mistakes. Remember, this neural net is just a local utility function - we can make a variety of AI designs involving it. The goal of this exercise is to design an AI that seems liable to make good decisions in order to flag lots of 9s.

The simplest example is the greedy agent - it just does whatever has a high score right now. This is pretty straightforward, and doesn't wirehead (unless the scoring function somehow encodes wireheading), but it doesn't actually do any planning - 100% of the smarts have to be in the local evaluation, which is really difficult to make work well. This approach seems unlikely to extend well to messy environments.

Since Go-playing AI is topical right now, I shall digress. Successful Go programs can't get by with only smart evaluations of the current state of the board, they need to look ahead to future states. But they also can't look all the way until the ultimate time horizon, so they only look a moderate way into the future, and evaluate that future state of the board using a complicated method that tries to capture things important to planning. In sufficiently clever and self-aware agents, this approximation would cause self-sabotage to pop up. Even if the Go-playing AI couldn't modify itself to only care about the current way it computes values of actions, it might make suboptimal moves that limit its future options, because its future self will compute values of actions the 'wrong' way.

If we wanted to flag 9s using a Dewian value learner, we might score actions according to how good they will be according to the projected utility function at some future time step. If this is done straightforwardly, there's a wireheading risk - the changes to its utility function are supplied by humans who might be influenced by actions. I find it useful to apply a sort of "magic button" test - if the AI had a magic button that could rewrite human brains, would it pressing that button have positive expected utility for it? If yes, then this design has problems, even though in our current thought experiment it's just flagging pictures.

To eliminate wireheading, the value learner can use a model of the future inputs and outputs and the probability of different value updates given various inputs and outputs, which doesn't model ways that actions could influence the utility updates. This model doesn't have to be right, it just has to exist. On one hand, this seems like a sort of weird doublethink, to judge based on a counterfactual where your actions don't have impacts you could otherwise expect. On the other hand, it also bears some resemblance to how we actually reason about moral information. Regardless, this agent will now not wirehead, and will want to get good results by learning about the world, if only in the very narrow sense of wanting to play unscored rounds that update its value function. If its value function and value updating made better use of unlabeled data, it would also want to learn about the world in the broader sense.

Overall I am somewhat frustrated, because value learners have these nice properties, but are computationally unrealistic and do not play well with approximation. One can try to get the nice properties elsewhere, such as relying on an action-suggester to not suggest wireheading, but it would be nice to be able to talk about this as an approximation to something fancier.

Thanks to the reaction to this article and some conversations, I'm convinced that it's worth trying to renovate and restore LW. Eliezer, Nate, and Matt Fallshaw are all on board and have empowered me as an editor to see what we can do about reshaping LW to meet what the community currently needs. This involves a combination of technical changes and social changes, which we'll try to make transparently and non-intrusively.

I like to read posts on "Main" from time to time, including ones that haven't been promoted. However, lately, these posts get drowned out by all the meetup announcements.

It seems like this could lead to a cycle where people comment less on recent non-promoted posts (because they fall off the Main non-promoted area quickly) which leads to less engagement, and less posts, etc.

Meetups are also very important, but here's the rub: I don't think a text-based announcement in the Main area is the best possible way to showcase meetups.

So here's an idea: how about creating either a calendar of upcoming meetups, or map with pins on it of all places having a meetup in the next three months?

This could be embedded on the front page of leswrong.com -- that'd let people find meetups easier (they can look either by timeframe or see if their region is represented), and would give more space to new non-promoted posts, which would hopefully promote more discussion, engagement, and new posts.

Thoughts?

This will be brief.

Inbox zero is a valuable thing to maintain.  Roughly promoted around the web as having an empty inbox.

An email inbox collects a few things:

• junk
• automatic mail sent to you
• personal mail sent to you
• work sent to you
• (maybe - work you send to yourself because that's the best way to store information for now)

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.

This is mainly of interest to Effective Altruists, and was cross-posted on the EA forum

Why separate effective giving from Effective Altruism? Isn't the whole point of EA about effective giving, meaning giving to the most impactful charities to advance human flourishing? Sure, effective giving is the point of EA, but there might be a lot of benefit to drawing a distinct line between the movement of Effective Altruism itself, and the ideas of effective giving that it promotes. That's something that Kerry Vaughn, the Executive Director of Effective Altruism Outreach, and I, the President of Intentional Insights, discussed in our recent phone call, after having an online discussion on this forum. To be clear, Kerry did not explicitly endorse the work of Intentional Insights, and is not in a position to do so - this just reflects my recollection of our conversations.

Why draw that line? Because there's quite a bit of danger in rapid movement growth of attracting people who might dilute the EA movement and impair the building of good infrastructure down the road (see this video and paper). This exemplifies the dangers of simply promoting Effective Altruism indiscriminately, and just trying to grow the movement as fast as possible.

Thus, what we can orient toward is using modern marketing strategies to spread the ideas of effective altruism - what Kerry and I labeled effective giving in our conversations - without necessarily trying to spread the movement. We can spread the notion of giving not simply from the heart, but also using the head. We can talk about fighting the drowning child problem. We can talk about researching charities and using GiveWell, The Life You Can Save, and other evidence-based charity evaluators to guide one's giving. We can build excitement about giving well, and encourage people to think of themselves as Superdonors or Mega-Lifesavers. We can use effective marketing strategies such as speaking to people's emotions and using stories, and contributing to meta-charities such as EA Outreach and others that do such work.  That's why we at Intentional Insights focus on spending our resources on spreading the message of effective giving, as we believe that getting ten more people to give effectively is more impactful than us giving of our resources to effective charities ourselves. At the same time, Kerry and I spoke of avoiding heavily promoting effective altruism as a movement or using emotionally engaging narratives to associate positive feelings with it - instead, just associating positive feelings with effective giving, and leaving bread crumbs for people who want to explore Effective Altruism through brief mentions and links.

Let's go specific and concrete. Here's an example of what I mean: an article in The Huffington Post that encourages people to give effectively, and only briefly mention Effective Altruism. Doing so balances the benefits of using marketing tactics to channel money to effective charities, while not heavily promoting EA itself to ameliorate the dangers of rapid movement growth.

Check out the sharing buttons on it, and you'll see it was shared quite widely, over 1K times. As you'll see from this Facebook comment on my personal page, it helped convince someone to decide to donate to effective charities. Furthermore, this comment is someone who is the leader of a large secular group in Houston, and he thus has an impact on a number of other people. Since people rarely make actual comments, and far from all are fans of my Facebook page, we can estimate that many more made similar decisions but chose not to comment about it.

Another example. Here is a link to the outcome of an Intentional Insights collaboration with The Life You Can Save to spread effective giving to the reason-oriented community through Giving Games. In a Giving Game, participants in a workshop learn about a few pre-selected charities, think about and discuss their relative merits, and choose which charity will get a real donation, $10 per participant. We have launched a pilot program with the Secular Student Alliance to bring Giving Games to over 300 secular student groups throughout the world, with The Life You Can Save dedicating $10,000 to the pilot program, and easily capable of raising more if it works well. As you'll see from the link, it briefly mentions Effective Altruism, and focuses mainly on education in effective giving itself.

Such articles as the one in The Huffington Post, shared widely in social media, attest to the popularity of effective giving as a notion, separate from Effective Altruism itself. As you saw, it is immediately impactful in getting some people to give to effective charities, and highly likely gets others to think in this direction. I had a conversation with a number of leaders of local EA groups, for example with Alfredo Parra in Munich, excited about the possibility of translating and adapting this article to their local context, and all of you are free to do so as well - I encourage you to cite me/Intentional Insights in doing so, but if you can't, it's fine as well.

That gets to another point that Kerry and I discussed, namely the benefits of having some EAs who specialize in promoting ideas about effective giving, and more broadly integrating promotion of effective giving as something that EAs do in general. Some EAs can do the most good by working hard and devoting 10% of their money to charity. Some can do the most good by thinking hard about the big issues. Some can do the most good by growing the internal capacity and infrastructures of the movement, and getting worthy people on board. Others can do the most good by getting non-EAs to channel their money toward effective charities through effective marketing and persuasion tactics.

Intentional Insights orients toward providing the kind of content that can be easily adapted and shared by these EAs widely. It's a work in progress, to create and improve this content. We are also working with other EA meta-charities such as The Life You Can Save and others. Another area to work on is not only content creation, but content optimization and testing - I talked with Konrad Seifert from Geneva about testing our content at a university center there. Moreover, we should develop the infrastructure to integrate spreading effective giving into EA activities, something EA Outreach may potentially collaborate with us on, depending on further discussions.

So these are some initial thoughts, which I wanted to bring to the community for discussion. What do you think of this line of work, and what are your ideas for optimization? Thanks!

**EDIT** Edited to clarify that Kerry Vaughn did not explicitly endorse the work of Intentional Insights.

The beginning of the new year is a natural Schelling Point and swiftly approaching. With that in mind I have created a handy go-to list of things worth considering for next year.

Alongside this process; another thing you might like to do is conduct a review of this year, confirming your progress on major goals; double checking that you are on track.  and conduct any last-minute summaries of potential failures or learning-cases.

This list is designed to be used for imagination, opportunity, and potential planning purposes.  If you find yourself having the feelings of (disappointment, failure, fear, regret, burdens, guilt and others) reconsider looking at this list and instead do something that will not lead to negative feelings about the future.  If you are not getting something positive out of doing this exercise, don't.  That's a silly idea.  I am banking on the fact that it will be more helpful than not; for most people.  If you are in the category of people that it does not help - I am sorry; I assume you know your priorities and are working on them as reasonably effectively as possible - good luck with that task.

This list is going to look a bit like my List of common human goals because it was written concurrenlty with the ideas listed there (and by the same person).

You might want a pen and paper; and 10 minutes to go through this list and consider what things you want to do over the next year that fall into these categories.  This time is not for you to plan out an entire year, but something of a chance to consider the playing field of "a year of time".  After you have a list of things you want to do; there are lots of things you can do with them.  i.e. time planning, research, goal factoring, task-generating.

without further ado; the list:

1. things I might want to study or learn next year

Often people like learning.  Are you thinking of higher level study?  Or keen to upskill?  Thinking of picking up a textbook (our list of best textbooks on every subject) on a topic.  Or joining a learning group for a skill

2. life goals I would like to have completed by next year

Do you already have a list of life goals?  Should you review them and do you want to particularly work on one over the next year?  Is something overdue?  Is there something you have been putting off starting?

3. health goals

Are there health targets that you let get away from you this year?  Are you looking to set future health targets?  Start new habits for the year?  beeminder suggests setting actionable goals as beeminding tasks, i.e. "eat carrots today" rather than targets "lose 1kg this month".

4. savings I want to achieve by next year.

Do you want to save money towards something?  You need a budget has a free course on getting ahead of the paycheck cycle, pocketbook can also help you manage your money.  The best advice seems to be to open a savings account and initiate automatic transactions each week of $n.  After several weeks (provided you don't pull money out) you will have accrued several*n dollars of savings.  (relevant to people who have a tendency to spend any money in their account at any given time.  It's a bit harder to spend money not in your spending-account) In any case; having savings and putting it towards owning a passive income stream is a good goal to have or consider getting in on.

This post may also be of use.

5. job/earning goals

Are you planning to get a new job?  Hoping to get a raise?  transfer to a new department?  work less hours?  work more hours?  land a few big gigs? While I can't tell you what is worthwhile; it's worth knowing that in the process of interviewing for a new job - you should ask for more pay.  for that 5-10 uncomfortable minutes of your life (asking for a raise) you have the potential to earn $5-10 thousand dollars more (or more) for the exact same work.

6. relationship goals + family goals

Married; Kids; Poly; single->not transition; break-up? Divorce? moving away from your parents?  Getting better Friends?  Thanking your current friends for being so awesome?  Doing something different to previously - now is the chance to give it a few minutes thought.  There's never a good time to stage a break-up but also living in a bad state of affairs is also not a good thing to prolong.  (Disclaimer: before quitting a relationship; first improve communication, if needed contact a professional counsellor)

About families and friends - A lot of people feel like their family holds a stronger bond than their friends by default.  For an excellent family that is supportive in your darkest hour that is an excellent situation to be in.  However for a burdensome family that drags you down; often it can be hard to get away.  In contrast to friends; where good ones can be better than family and bad ones can be walked away from.  Specifically what's worth considering is that friends OR family can be a result of how you choose to treat them.  in the sense that if you have a preference that your friends be stronger than the strongest family ties then you can carry that into reality and achieve friendships to the envy of most families, and the same goes for a strong supportive family.  Your choice of what shape of reality you want to make for yourself will influence (on some levels) what sort of mess you get yourself into, and what sort of support network you have around.  Make that consideration over the next year of what sort of friendships and families you want to make for yourself and keep for yourself.

7. lifestyle goals

Start exercising daily (do you even lift)? Quitting smoking?  Do you go clubbing too often?  maybe you want to get out more? Addicted to curry puffs?  Hate hanging out with that group of friends?  Don't like going to pub trivia but do it anyway?  Too many doughnuts?  Go hiking?  Thinking of trying out a new hobby? holding out for "the right time". take that leap, sign up for a class.  Now is the time to make lifestyle changes.  (fair warning: most new year's resolutions fail, look into SMART goals)

8. holiday goals/ travelling goals

looking at doing a month-long holiday?  Visiting someone in another place?  Maybe consider planning from now.  Studies have shown that anticipation and putting energy towards planning positive things leads to happiness (in the journey) the ability to look forward to your next holiday is going to have positive impacts on the way you live.

9. donations 

Have you had intention to make donations but haven't made the plunge?  Maybe put some thought into how much you might like to donate and when/where to?  Many LW'ers are also EA's and have interests in motivated and purposeful giving for maximising possible outcomes.  This could be an opportunity to join the group of EA's that are actively giving.

10. volunteering

Have you always wanted to volunteer but never looked into it?  Maybe next year is the year to try.  Put some research in and find a group in need of volunteers.  Volunteering has the potential to give you a lot of positive feelings as well as a sense of community; being part of something bigger, and more.

You could stop here but there are a few more.  Out of the more general List of common human goals comes the following list of other areas to consider.  They are shorter in description and left open to imagination than those above.

11. Revenge

Is next year your chance to exact revenge on your foes?

12. Virtual reality success

Is next year the chance to harvest your gemstones?

13. Addiction

Is next year the year to get addicted (to something healthy or good for you, like exercise), or un-addicted (to something unhealthy for you)?

14. Ambassador 

Are there things you want to do next year which will leave you as a representative of a group?  Is there a way to push that forward?  Or better prepare for that event?

15. Help others?

Do you know how you might go about helping others next year?

16. Keeping up with the joneses

Are you competing with anyone?  Is there something you are likely to need to prepare for throught the year?

17. Feedback

Are you looking for feedback from others?  Are you looking to give feedback to others?  Is this the year for new feedback?

18. Influence

Do you want to influence the public?

19. fame

Do you want to achieve some level of fame?  We live in a realm of the internet!  You wouldn't believe how easy that is these days...

20. being part of something greater

Joining a movement?  Helping to create a revolution?  This could be the year...

21. Improve the tools available

As scientists we stand on the shoulders of the knowledge before us in order to grow.  We need sharp tools to make accurate cuts and finely tuned instruments to make exact measurements.  Can you help the world by pushing that requirement forward?

22. create something new

Is there something new that you want to do; is next year appropriate for doing it?

23. Break a record

Have your eye on a record?  How are you going to make it happen?

24. free yourself of your shackles

Are there things holding you back or tying you down?  Can you release those burdens?

25. experience

hoping to have a new experience, can you make it happen with thinking about it in advance?

26. Art

Want to have developed a creation?  Can you put wheels into motion?

27. Spirituality

Anything from a religion based spiritual appreciation to a general appreciation of the universe.  Revel in the "merely real" of our universe.

28. community

Looking to make a community, looking to be part of an existing community.  Looking to start a lesswrong branch?  Do it!

Meta:

about 2.5 hours of writing plus feedback from the https://complice.co/room/lesswrong room and the Slack channel

If you are looking for some common ways to work on these possible goals?  That sounds like a great title for the next post in a matching series (one I have not written yet).  If you want to be a munchkin and start compiling thoughts on the idea, feel free to send me a message with a link to a google doc, otherwise you might have to wait.  This post was written out of necessity for the new-year, and wasn't on my to-do list so the next one might take time to create.

Feel free to comment on goals; plans; progress or post your plans for the next year below.

If you can see improvements to this post - don't be afraid to mention them!

To see more posts I have written see my Table of contents

This summary was posted to LW Main on November 27th. The following week's summary is here.

Irregularly scheduled Less Wrong meetups are taking place in:

• Cologne meetup: 28 November 2015 05:00PM
• Prague Less Wrong Meetup: 02 December 2015 07:00PM
• San Antonio Meetup!: 29 November 2015 02:00PM

The remaining meetups take place in cities with regular scheduling, but involve a change in time or location, special meeting content, or simply a helpful reminder about the meetup:

• NYC Solstice: 19 December 2015 05:30PM
• Seattle Solstice: 19 December 2015 05:00PM
• [Vienna] Five Worlds Collide - Vienna: 04 December 2015 08:00PM

Locations with regularly scheduled meetups: Austin, Berkeley, Berlin, Boston, Brussels, Buffalo, Canberra, Columbus, Denver, London, Madison WI, Melbourne, Moscow, Mountain View, New Hampshire, New York, Philadelphia, Research Triangle NC, Seattle, Sydney, Tel Aviv, Toronto, Vienna, Washington DC, and West Los Angeles. There's also a 24/7 online study hall for coworking LWers and a Slack channel for daily discussion and online meetups on Sunday night US time.