Filter Last three months
This hour Today This week This month Last three months This year All time

Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Privileging the Question

91 Qiaochu_Yuan 29 April 2013 06:30PM

Related to: Privileging the Hypothesis

Remember the exercises in critical reading you did in school, where you had to look at a piece of writing and step back and ask whether the author was telling the whole truth? If you really want to be a critical reader, it turns out you have to step back one step further, and ask not just whether the author is telling the truth, but why he's writing about this subject at all.

-- Paul Graham

There's an old saying in the public opinion business: we can't tell people what to think, but we can tell them what to think about.

-- Doug Henwood

Many philosophers—particularly amateur philosophers, and ancient philosophers—share a dangerous instinct: If you give them a question, they try to answer it.

-- Eliezer Yudkowsky

Here are some political questions that seem to commonly get discussed in US media: should gay marriage be legal? Should Congress pass stricter gun control laws? Should immigration policy be tightened or relaxed? 

These are all examples of what I'll call privileged questions (if there's an existing term for this, let me know): questions that someone has unjustifiably brought to your attention in the same way that a privileged hypothesis unjustifiably gets brought to your attention. The questions above are probably not the most important questions we could be answering right now, even in politics (I'd guess that the economy is more important). Outside of politics, many LWers probably think "what can we do about existential risks?" is one of the most important questions to answer, or possibly "how do we optimize charity?" 

Why has the media privileged these questions? I'd guess that the media is incentivized to ask whatever questions will get them the most views. That's a very different goal from asking the most important questions, and is one reason to stop paying attention to the media. 

The problem with privileged questions is that you only have so much attention to spare. Attention paid to a question that has been privileged funges against attention you could be paying to better questions. Even worse, it may not feel from the inside like anything is wrong: you can apply all of the epistemic rationality in the world to answering a question like "should Congress pass stricter gun control laws?" and never once ask yourself where that question came from and whether there are better questions you could be answering instead.

I suspect this is a problem in academia too. Richard Hamming once gave a talk in which he related the following story:

Over on the other side of the dining hall was a chemistry table. I had worked with one of the fellows, Dave McCall; furthermore he was courting our secretary at the time. I went over and said, "Do you mind if I join you?" They can't say no, so I started eating with them for a while. And I started asking, "What are the important problems of your field?" And after a week or so, "What important problems are you working on?" And after some more time I came in one day and said, "If what you are doing is not important, and if you don't think it is going to lead to something important, why are you at Bell Labs working on it?" I wasn't welcomed after that; I had to find somebody else to eat with!

Academics answer questions that have been privileged in various ways: perhaps the questions their advisor was interested in, or the questions they'll most easily be able to publish papers on. Neither of these are necessarily well-correlated with the most important questions. 

So far I've found one tool that helps combat the worst privileged questions, which is to ask the following counter-question:

What do I plan on doing with an answer to this question?

With the worst privileged questions I frequently find that the answer is "nothing," sometimes with the follow-up answer "signaling?" That's a bad sign. (Edit: but "nothing" is different from "I'm just curious," say in the context of an interesting mathematical or scientific question that isn't motivated by a practical concern. Intellectual curiosity can be a useful heuristic.)

(I've also found the above counter-question generally useful for dealing with questions. For example, it's one way to notice when a question should be dissolved, and asked of someone else it's one way to help both of you clarify what they actually want to know.)

Comments (308)

Maximizing Your Donations via a Job

87 Alexei 05 May 2013 11:19PM

In November of 2012 I set a goal for myself: find the most x-risk reducing role I can fill. At first I thought it would be by working directly with MIRI, but after a while it became clear that I could contribute more by simply donating. So my goal became: find the highest paying job, so I can donate lots of money to CFAR and MIRI.

A little bit of background on me. Started programming in 2000. Graduated in 2009 with Bachelor's in computer science. Worked for about a year and a half at a game company. Then did my own game startup for about a year. Then moved to the bay area and joined a game startup here, which was acquired 10 months later. Worked a bit at the new company and then left. So, just under four years of professional programming experience, but primarily in the game industry. Almost no leadership / managerial experience, aside from the startup I did where I hired freelancers.

Below is my experience of finding a software engineering job in the Silicon Valley. If you are not an engineer or not in the Silicon Valley, I think you'll still find a lot of useful information here.

 

Pre-game

Before sending out my resume, I spent about a month preparing. I read Intro to Algorithms, which was very good overall, but not a huge help in preparing for interviews.[1] I read Cracking the Coding Interview, which was extremely helpful. (If you read only one book to prepare, make it this one.) The book has a lot of questions that are similar to the ones you'll actually see during interviews. I also did TopCoder problems, which were pretty helpful as well.[2] Looking back, I wish I spent more time finding actual interview questions online and doing more of those (that's why CCI book was so helpful).

After several weeks of preparation, I compiled a long list of companies I was going to apply to. I checked on GlassDoor to see what kind of salary I could expect at each one. I then rated all the companies. Companies with low salaries and poor personal fit received the lowest rating.

I started by applying to companies with the lowest ratings. This way I could use them as practice for the companies I thought would actually make a competitive offer. This was the right move and worked very well. (Another friend of mine did the same approach with good results as well.) Remember, you are not just doing those interviews to practice the coding problems, you are practicing pitching yourself as well.

 

Interviewing with a company

Standard procedure for applying to a tech company:

1. Send them your resume.

  • Proofread your resume. Let your friends proofread it.
  • Make sure there are only relevant things on it. When I applied to tech companies, I removed a lot of game-specific things from my resume. When I applied to companies that did 3D graphics, I made sure I had all my 3D graphics experience listed. I ended up with two version of my resume.
  • Have your resume in DOC, PDF, and TXT formats. This way you'll always have the right one when you upload / paste it.
  • For a few companies, I had a friend or friend of a friend who referred me. This REALLY HELPS in two ways: 1) your resume will be processed a lot faster, 2) if your friend is a great engineer/employee, you'll be taken a lot more seriously, and the company will fight for you a lot harder.

2. You'll get an email from the recruiter and setup a time to speak, where you'll talk about yourself, what you've done, why you are interested in their company, and so on. You can and should ask them questions as well.

  • When you start getting multiple calls each day, make sure you know who is calling. There is nothing worse than talking about the challenges of streaming music to a car sharing startup. (True story.)
  • Read about the company on Wikipedia before the call. Know the basic stuff. Look at their website and read the About page.
  • Find the thing that makes the company special and successful. Find the thing that you actually think is cool about the company. Those are your answers for why you want to work there.
  • Ask non-technical questions: How is the company structured? How many teams are there? How many employees? Engineers? Think of other intelligent questions to ask.
  • In my experience, it's not very beneficial to tell them you are interviewing with a dozen other companies. When they ask who else you are interviewing with, just name a few companies, especially the competitors / similar companies.
  • Be SUPER NICE to your recruiter. They are your main point of contact with the company. They'll be the one fighting to get you the best offer.

3. You'll have a technical phone interview with a software engineer where you'll solve a problem or two on collabedit or some similar website. At the end, you'll get a few minutes to ask them questions too.

  • All the usual interviewing tips apply here. E.g. talk out loud, your interviewer doesn't know what you are thinking.
  • Most companies don't care what language you use, as long as it's mainstream. (I used C# for almost all my coding questions.)
  • DO NOT start answering the question by writing code. If the questions seems vague, ask about the context. Who'll be using this solution? Definitely ask about the kind of data you are working with. If it's integers, are they random? Over some small range or over all possible integers?
  • List out metrics for various approaches: brute-force solution, optimized for speed solution, optimized for memory solution. Here is a question I saw a few times: Write a data structure which can accept and store integers, and can check if there exist two integers that sum up to a given number. There are multiple solutions, and the best one depends on the ratio of addInteger to checkForSum calls.
  • The previous steps should only take you a minute or two. Once you've decided what the best approach is, then you can write the solution. When you are done, check for errors, then run through several examples. Do a simple example and a slightly complicated example. When you find a bug, don't be hasty in fixing it. Understand why it happened and make sure you won't introduce new bugs by fixing it.
  • If everything works, make sure you handle errors correctly. Can you handle invalid input? Input that violates your assumptions? (As a reminder, I leave “\\Check for errors” comments in appropriate spots as I code the solution.)
  • When you are done, ask the interviewer questions. Ask them to tell you about what they do, if they haven't already. What have they been working on recently? What technologies/languages do they use at the company? Do they use Scrum/Agile? Pair-programming? Come up with other intelligent questions to ask.

 

4. You'll be invited for an on-site interview which will be 3-6 hours long, at least half of which will be coding on a white-board. (Although, a friend told me he brought his laptop with him, and most people were fine with him coding on it.)

  • All the previous tips apply.
  • Be on time. Take bathroom breaks when you need them. I found that drinking water during the interview keeps me refreshed. Remember your posture, body-language, and eye-contact skills.
  • Learn how to talk out loud as you are writing out your solution. If you are stuck, explain what you are thinking, and what your intuition is telling you.
  • Learn how to read your interviewers. If you say, "Here we should check for the null case or for empty array," and they go "Yeah, yeah, okay," they are not the type of interviewer that really cares about error conditions, so you can be somewhat more lax there. By the time I was finishing my on-site interviews, I could tell if my solution was right just by the interviewer's body language.
  • When you are done, ask them questions. What are they working on? What's the thing they like most about the company? What's their least favorite thing about the company? (Another way to phrase that: What's one thing you would change if you could change anything about the company?) Do they have to work overtime? How are the people here? Can you switch between projects? Are there company wide events? In all my interviews I've never met an interviewer that didn't try to sell their company really hard. People will always tell you their company is the best place to work.
  • If the person is a manager or a director, ask them higher level questions. What kind of culture are they trying to create? What are the current big challenges? Where do they want the company to be in the next 5 years? How does one advance in the company? (Usually there is a managerial and a technical track.) How often are reviews done? How are they structured?

 

5. You'll get a call from the recruiter congratulating you on an offer. They'll go over the offer details with you.

  • Before they make you an offer, they'll check if you are actually seriously considering their company. If you told a startup you are also interviewing with Google, they might suspect that you are not seriously considering them. Unless you dissuade those fears, they might actually not even make you an offer. (Happened to me with Rdio.)
  • If you didn't get an offer, try to get as much info as you can. What happened? What can you improve on? Below are the reasons why I didn't get an offer after an on-site interview:
    • Not doing well on a technical question. (Happened twice; one time because of a very obnoxious interviewer.)
    • Not interviewing for quite the right position (that on-site interview ended early).
    • Not having the necessary experience (a lot more important to startups than bigger companies).
    • Not being passionate enough about the company.
  • If this is not a good timing for the offer, e.g. it's one of your first interviews, then tell them so. They will probably wait to give you the offer details until you are ready to consider it.
  • The recruiter will likely ask what's important to you in an offer. How are you going to make your decision? What I've said is that compensation will be an important factor in my decision, but that the team/project/etc. are important considerations as well.

 

6. You have a few days (usually around 5 business days) until the offer expires to decide if you want to accept it.

  • Sometimes the offer will expire before you've received offers from other companies. This is why it's important to interview in rough order of ranking, so that you can just let those offers go, knowing you'll have much better ones soon. If you want to hold on to the offer, just ask your recruiter for an extension. It'll be much easier to get an extension at big companies, especially if you are interviewing for a generic position.
  • If you decline the offer, let them know.

 

Always be very nice, friendly, and polite. Walk the fine line between telling the truth and saying the right thing. Ideally, make sure those are the same. Even if you are interviewing with a company you have no intention of working at, make sure to find something you really like about them, something that makes them stand out to you. Always have a good answer to: "Why do you want to work here?"

Before each on-site interview make sure you research the company thoroughly. Use their product. Think of ways to improve it. It's very helpful if you can meet with someone that works there and talk to them. See if they can give you any tips on the interview process. Some companies (e.g. AirBnB) want people that are extremely passionate about their product. Some companies focus more than usual on architectural questions. Many companies expect the engineers to have some familiarity with UI/UX and the ability to think about a feature from all angles.

 

Managing your time

I sent my resume to 78 companies, had at least a phone conversation with a recruiter with 27 of them, had an on-site interview with 16 companies, and received 12 offers. Out of those, I've only seriously considered 3. (Companies with lower ratings had an atrocious response rate.)

My time-line ended up looking something like this:

  • Week 1: Started applying to low-rated companies. About 2 phone interviews.
  • Week 2: About 7 phone interviews. One on-site interview. Sending out more resumes.
  • Week 3: About 3 phone interviews.
  • Week 4: About 15 phone interviews. A few meetings with friends of friends, who ended up referring me. 1 on-site interview. Sent my resume to all the high-rated companies. (During this week interviewing became a full-time job.)
  • Week 5: About 10 phone interviews. 4 on-site interviews.
  • Week 6: 8 phone interviews. 4 on-site interviews.
  • Week 7: 4 phone calls. 5 on-site interviews.
  • Week 8: 12 phone calls. 2 on-site interviews.
  • Week 9: About 8 calls a day for a few days, while I negotiated with my top companies.
  • (These are strictly lower bounds for phone calls. On-site data is pretty accurate.)

Some companies move fast, some companies move slow. Google took 2 weeks from the on-site interview to the offer call. This is very common for them, but most other companies move faster. With Amazon, I actually interviewed with two different branches. With one branch things were going well, until they dropped the ball and never got back to me, even after I pestered them. This is unusual; although, Twitter did something similar, but then ended up responding with an on-site invitation. With the other Amazon branch, when I got home from the on-site interview, I already had an email saying they were going to make an offer. This is extremely fast. (I had a very good reference for that position.) Most companies take about a week between on-site and offer. The whole process, from first call to offer, takes about three weeks.

If your recruiter doesn't respond to you during 4 days or longer, shoot them an email. They might have forgotten to respond, or thought they did, or may be things are moving slowly, or may be they decided not to pursue. You want to be clear on where you stand with all the companies you are applying to.

The timing is pretty important here. You want your top-rated companies to give you an offer within a span of a week. This way you'll be able to leverage all those offers against each other.

If your current job position is already almost optimal for your goals, then it's possible you can do a few interviews, get a few offers and pick the best one, which will give you some marginal improvement. Or use those offers to leverage a raise at your existing company. But if you are pretty sure your current job has not been optimized for your goal, then I'd say, contrary to popular wisdom, just leave and spend a full month interviewing. (Or, even better, if you can, take a long "vacation".) You just can't do this kind of intense interviewing while holding another job. The one exception to this rule I can think of is if one of your highest-rated companies is a competitor with your current employer. Then you can leverage that!

Value of information is extremely high during this process. Talk to all the companies you can, talk to all the people you can. Once you have the final list of companies you are considering, reduce your uncertainty on everything. Validate all your assumptions. (Example: I was sure Google matched donations up to $12k, but turns out it's only up to $6k.)

 

How to evaluate your offer

There are 4 basic components in an offer: sign-on bonus, base salary, equity, and bonus.

Sign-on bonus. Most companies will be okay offering something like $12k sign-on bonus. Some will offer more. Most startups probably won't offer any.

Base salary. This is pretty consistent across most companies. Based on your experience, you'll be given a title (e.g. Senior Software Engineer or SE 2), and that title will determine the range of the salary you can expect. If you are good, you can demand a salary at the top of that range, but it's extremely hard to go higher.

Equity. This is the most interesting part. A good amount of value will come from this portion. With a startup, it'll be most of it. Here are two things to pay attention to:

  • Is the company public or private? If it's public, you are most likely going to be given RSUs (restricted stock units), which will basically convert to normal company shares when they vest. For private companies, see the section below.
  • What's the vesting schedule? For almost all companies you'll get 25% of your shares right after your first year. (This is called a 'cliff'.) After that you'll be given the appropriate fraction either monthly (e.g. at Google) or quarterly (e.g. at Facebook). Amazon is an example of a company where the vesting schedule is somewhat different: 5% after year 1, 15% after year 2, and then 20% each semester for the next two years.

Bonus. This is the bonus system the company has setup. You can't negotiate it, but it's important to take it into account.

  • There will usually be a cash bonus that's based on your salary. It'll have a target percent (e.g. 15%). If you can find out how many people hit their target, that will be very helpful. However, most companies don't share or simply don't have that information.
  • Some companies also have equity bonuses. Try to get as much info on those as you can. Don't assume that you'll get the maximum bonus even if you work hard. If you have friends working at that company, ask them what kind of bonuses they've been getting.
  • Lots of startups don't have bonus systems in place.

Other factors.

  • Donation matching: Google matches up to $6k (you donate $6k to any charity, they'll donate another $6k). Craigslist matches 3:1 up to 10% of your salary. Most companies don't have anything like that, and you can't negotiate it.
  • Paid Time Off: Google offers 2 weeks, all other companies I was considering offer 3 weeks, and some even have unlimited PTO. This is not negotiable in most companies.
  • Commute: how far will you have to travel to work? Are you okay moving closer to work? (Google and Facebook have shuttles that can pick you up almost anywhere, so you could work while you commute.)
  • People/culture/community/team/project are all important factors as well, depending on what you want. If you are going to spend the next several years working on something, you should be building up skills that will still be valuable in the future.

 

Thinking about private companies

If the company is private, you might be given RSUs or you might be given stock options. With stock options, you'll have to pay the strike price to exercise your options. So the total value your options have is: (price of a share - strike price) * number of shares.

You can't do anything with your shares until the company gets acquired or goes public. Some companies have liquidation events, but those are pretty rare. Most companies don't have them, and the ones that do only extend the opportunity to people that have been with the company for a while. There are also second-hand markets, but I don't know much about those.

If you are completely risk-intolerant, then just go with a public company, and don't consider private companies. (This is actually not exactly true. Just because a company is public, doesn't mean its risk-free, and just because a company is private doesn't mean there is a lot of risk. There are other important factors like the size of the company, their market diversity, and how long they've been around.) If you are okay with some risk, then you want a company that's close to an IPO or is likely to get acquired soon. If you want to have a chance to make more than a few million dollars, either start your own company or join a very early stage startup (my top pick would be Ripple). Before doing so, check out the stats on startups to make sure you understand how likely any given startup is to fail and make sure you understand the concepts of inside/outside view.

 

Taxes

It's crucial to understand all the tax implications of your salary, equity, and donations. I'm not going to go into all the details, there are a lot of resources out there for this, but you should definitely read them until it's crystal clear how you will be taxed. I'll highlight a few points:

  • Understand the tax rate schedule and notice the new 39.6% tax bracket. If your income is $100k, that doesn't mean you get taxed 28% on all of it. 28% applies only to the income portion above $87,850. Also note that this is only the federal tax. Your state will have additional taxes as well. Aside from those percentages, there are a few other flat taxes, but they are considerably smaller in magnitude.
  • The money you donate to a nonprofit (aka. 501(c)(3)) organization can be subtracted from your taxable income. This means that you will most likely get a refund when you file your taxes. Why? Because when you fill out your W4 form, you'll basically tell your employer how much money to withhold from your paycheck for tax purposes. If you don't account for your future donations, more money will be withheld than is appropriate and the discrepancy will be paid back to you after you file your taxes. Ideally, you want to take your donations into account and fill out the W4 form such that there are no discrepancies. That means you'll get your money now rather than later. (I haven't gone through this process myself, so there is some uncertainty here.)
  • You can claim tax deduction for up to 50% of your wages. That means if you make a lot of money in one year, even if you donate most of it, you'll be able to reduce your taxable income by a maximum of 50%. The rest goes over to the next year.
  • When RSUs vest, their value is treated as ordinary income for tax purposes. When you sell them, the difference is taxed as a capital gain (or loss).
  • Stock options have a more complicated set of tax rules, and you should understand them if you are considering a company that offers them.
  • You can't have your employer donate money or stock for you to bypass the taxes. I've asked.

 

Calculating donations

To calculate exactly how much I could donate if I worked at a given company, I've created this spreadsheet. (This is an example with completely fictitious company offers with very low numbers, but the calculations should be correct.) Let me walk you through the spreadsheet.

 

Time discounting (Cell B1)

Money now is more valuable than money later. By how much? That's a very complicated question. If you invest your money now, you might be able to make something like 10% annually with some risk.[3] If you are donating to a charity, and they are growing very rapidly, then they can do a lot with your money right now, and you should account for that as well. If you expect the charity to double in size/effectiveness/output in the next year, then you might use a discount rate as high as 50%. I chose to use 20% annual discount rate based on my own estimates. Since I'm doing monthly compounding, the spreadsheet value is slightly higher (~22%). You can look at the column K to see how the future value of a dollar is being discounted. Note, for example, that a dollar in 12 months is worth 80¢ to me now. This discounting rate is especially important to keep in mind when examining startups, because almost all their compensation lies in the future. The further away it is, the more heavily you have to discount it.

 

Cost of living (Cell B2)

This is how much pre-tax money a year I'm not going to donate. See column L for the monthly expenses. We time-discount those dollars as well.

 

Offers (Cells A4-I15)

This is where you plug-in the offers you get. Bonus row is for cash bonus. Equity row is for the total equity the company offers you. I use the dollar amount, but you'll notice that for some of them I'm computing the dollar amount as: RSUs the company is giving me * current share price. For private companies, this is value I expect my equity to have when the company goes public. For Square it looks like: (percent of the company I'll own) * (my guess at valuation of the company at IPO) - (cost to exercise my options). For Twitter it looks like: (growth factor up to IPO) * (current price per share) * (RSUs I am granted). (Again, the numbers are completely made up.) In my calculations I'm not expecting public companies' share price to rise or fall. If you disagree, you should adjust for that as well.

 

Monthly projections (Cells A18-I66)

We are going to look at how much money we'll be making per month for the next four years. (Four years because our equity should be fully vested by that time.) If you are certain that you will stay at the company for less time than that, then you should consider a shorter timeline. This might affect companies differently. For example, most of the equity you get at Amazon comes during the last two years. If you are not going to be there, you are missing out on a big part of your offer.

For companies that I was seriously considering, I created two columns: one for cash wages and one for equity wages. This way I can do taxes on them more precisely.

Let's go through the Google's offer:

  • For the first year we'll be only making our standard salary.

  • After the first year, we get our cash bonus (green font). Here we are assuming it'll be 15% of our salary. We also get 25% of our RSUs vested (salmon background).

  • For the remainder of the second year, we are making our normal salary. Each month we also get 1/48th of our original equity offer.

  • Google also has an equity bonus system, where each year you can get a bonus of up to 50% of your original equity offer. This bonus will be paid in RSUs, and it vests over 4 years, but with no cliff. So we count that as well, but I'm assuming I'm only going to get 15%, not the full 50%.

  • In year 3 everything is basically the same, except now we got our second equity bonus, so we have two of them running simultaneously.

  • In year 4, we have three of them running simultaneously.

For pre-IPO companies, I've estimated when they'll go IPO. Most have clauses in place that don't allow you to sell your shares until after half a year or so after the IPO. I'm assuming I will sell/donate all my shares then, and then continue selling/donating them as they continue vesting.

 

Sum (Cells A68-I71)

In row 68 we have the total sum. This is the amount of pre-tax dollars we expect to earn in the next four years (remember that this amount has been adjusted for time-discounting, so it'll seem much lower than you'd normally expect). L68 is how much money we are spending on ourselves during those four years.

In row 69 we subtract our living expenses to get the amount of money we'll be able to donate. Note that I'm subtracting it from the cash column, leaving the equity column alone (for the companies where I split the two).

In row 70 we account for taxes. Note that our living expenses already accounted for the taxes we pay up to $65k, so the rest of it will be taxed at around 28% or higher. You could sell your shares, or you could just donate your shares directly to your charity. (That's what we are doing with our Google offer.)

In row 71 we simply sum up the donations from cash and equity.

 

Disclaimer 1: while I tried as hard as I could to double check this spreadsheet, there might still be mistakes there, so use it with caution and triple check everything. The tax calculations as they are right now are wrong, and you'll have to redo them (basically the whole Row 70) based on your own numbers.

Disclaimer 2: this spreadsheet is not great for evaluating an offer from a startup, since it doesn't capture the associated uncertainty and risk. Furthermore, if you expect the startup to succeed after more than 4 years, to correctly compare it to other companies you'll have to compute more than 48 months and potentially start accounting for things like promotions and raises.

 

Picking the one

All right, so how do you actually pick the best company? It's not as simple as picking the one with the highest EV, since you have to account for risk involved with startups and even pre-IPO companies. In fact, you should be surprised if your offers from public companies have a higher EV than offers from startups. If that's the case, I'd double check your calculations.

This is where it becomes extremely crucial to narrow down your uncertainty. When is the company going to IPO? What is the likely valuation? Does the company have a lot of competitors? Does the company have the necessary talent to execute on their plan? What's the company's history? What is the employee churn rate (especially for executives)? How well is the company doing financially? Who are the investors? Etc, etc, etc... There is a ton of questions you should be asking, and you should be asking them to everyone whose opinion on this issue you can respect. Honest opinion from an informed and knowledgeable neutral party is worth a LOT here!

You should also talk to the people at the company. Your recruiter will connect you to the right people if you ask. Keep in mind that nobody there will tell you that the company is going to go bankrupt or fail. But you can still get some valuable estimates, and then potentially discount them down a bit. You can even ask for their opinion on other companies you are interviewing with. Expect them to completely throw the other company under the bus though, but even so, you could get a lot of valuable criticism and bring it up when you talk to that other company. Overall, expect a lot of conflicting messages.

Keep in mind the charities you'll be donating to. What kind of donors do they have already? Are most people donating a bit from their salary? In that case, a more risky venture might be reasonable. Can they really use some money right now, or would they be a lot more effective later on with a large capital? What's their time discount rate? If you care about your charity, you can help them diversify their donor pool.

For me, it was a hard choice between big public companies (primary candidate: Google) and close to IPO companies (primary candidates: Twitter and Square).

 

Negotiating

You have to negotiate your offer. You have to have to have to HAVE TO. For any given company, you'll be able to get them to up their offer at least once and potentially thrice. Example: Google upped my offer three times.

  • Some companies will tell you their offer is not negotiable. That's not true.
  • It's much easier to leverage similar companies against each other. Leverage big public companies against each other; leverage pre-IPO companies against each other; etc... Leveraging between those categories is a bit more difficult, because startups know they can't compete with the raw cash value you are offered at bigger companies. The only thing they can do is up their equity offer and hope that they are a much better personal fit for you than the large companies.
  • Recruiters will ask you very directly what the other companies are offering you. You can choose to disclose or not to disclose. If you don't disclose, the company will come back to you with their standard offer. That offer might be higher or lower than you expected. (Example: The first offer I got from Google was significantly worse than initial offers I got from Facebook and Amazon.) If you tell them what offers you have (and you should only disclose details of your very best offers), then they'll very likely match or come in a bit stronger. Usually you don't have much to gain by disclosing your other offers upfront. You can always do so later. However, you should let your recruiters know that other companies did make an offer, or you are expecting them to. That gives you more leveraging power.
  • Sign-on bonus is very easy to negotiate. You can easily convince a company to match a sign-on bonus their competitor has offered.
  • Negotiating salary is much harder, but, again, usually you can convince a company to match a salary their competitor has offered or at least come closer to it. If you are interviewing with startups, their salary offer will usually be lower than at bigger companies and even harder to negotiate. ("Cash is king" is the common phrase used there.)

First negotiating phase: simply email / call back your recruiter (who is now your best friend, right?) and tell them that the offer is somewhat lower than you expected, you have other better offers from other companies, and you are wondering if they can increase their offer. If the company made you a clearly worse offer than another similar company, you should be very open about it.

Second negotiating phase: matching other companies. This is when it makes the most sense to disclose your other offers. For example, I used my Amazon and Facebook offers to convince Google to up their offer significantly. For some reason their original offer was very low, but seeing their competitors with much better offers convinced them to update pretty quickly. You can also bring up the perks one company has that the other doesn't (e.g. donation matching or unlimited PTO). The company can make up for that with salary/equity. There is some difficulty in using offers from private companies as leverage, because there is not much information you can disclose about them. You can talk about the number of shares you'll have, but it might not mean anything to the other recruiters if they are not familiar with the startup.

Third negotiating phase: once you picked the company you'll work for, go back to them and say something along the lines of "I really like the offer and the company, but there are a few things that don't make it ideal for me. One of your competitors did this, and another company has that. Right now I'm inclined to go with your competitor, but it's a tough decision, and I would rather go with you. I think if you can make me an offer with the following parameters, it'll make my decision extremely easy, and I'll sign on the spot." Include offer letters from other companies, especially the ones that have them beat or beat on some parameters. Notice the key promise at the end: you will sign with them. Your recruiter will have a lot more leverage in fighting for you if you make that promise. You are not legally obligated to follow through with your promise, but I wouldn't advise breaking it or using it just to extract more value to use as leverage against other companies. Use this tactic at the very end to extract that last bit of value from the company that's already the best. This is what I did with Google. I asked for about 3% higher salary and 12% more equity than what they were offering, and they came back with the exact numbers I requested, which means I should have asked for more. My advice would be to ask for about twice or may be even three times as much (6% and 30% respectively). Even if they come back with a compromise, it'll very likely be more than 3% and 12% increase. If not, you can try to barter one more time.

I'm sure some people will cringe at this kind of haggling, but, in all honesty, this is what recruiters expect, and they are very much used to it. Nobody even blinked an eye when I started negotiating, even on second and third rounds. However, some recruiters might try to make you feel guilty. They'll say that if you really want to work at their startup, then you shouldn't really care about your compensation. Most points they'll make will even be valid, but if you are trying to optimize for donations, then you have to make the compensation the most important factor in your decision. I've actually told most of my recruiters that I plan to donate most of my salary to charities. I don't think that got me higher offers, but it made me come off less like a greedy jerk.

At the end of the day, the company wants you, but they want to pay you as little as possible. But, given the choice of having you and paying you the most you deserve VS. not having you, all companies will pick the first option. ALL OF THEM. This is one of the best perks of being a talented software engineer in the bay area.

Once you accept the offer, don't forget to email everyone else and let them know. Thank everyone that helped you. Some recruiters will be surprised by your decision, and some will even fight really hard to get you to reconsider.

 

 

 

[1] None of the interviews required a data structure more complicated than a heap. All the answers had a very easy to compute complexity, either polynomial, polynomial * logarithmic, or factorial. The most weird one was probably O(√n) for computing prime numbers.

[2] Some problems I did during actual single-round match-up (SRM) competitions, which is good for training yourself how to code and think faster than you are used to. I also did a lot of old SRM problems, which have solutions and explanations posted in case I couldn't get them. I could easily do problem 1 & 2 in the easy division, and could do problem 3 most of the time. I didn't really bother with the hard division, and none of the interview questions were ever as hard as problem 3 in the easy division.

[3] According to the comments, this number is too high. Pick your own best estimate.

Comments (42)

Recent updates to gwern.net (2012-2013)

62 gwern 18 March 2013 07:54PM

Previous: Recent updates to gwern.net (2011)

“But where shall wisdom be found? / And where is the place of understanding? / Man knoweth not the price thereof; neither is it found in the land of the living…for the price of wisdom is above rubies.”

As before, here is material I’ve worked on in the 477 days since my last update which LWers may find interesting. In roughly chronological & topical order, here are the major additions to gwern.net:

Transcribed or translated:

More technical:

Personal:

Comments (32)

Reflection in Probabilistic Logic

61 Eliezer_Yudkowsky 24 March 2013 04:37PM

Paul Christiano has devised a new fundamental approach to the "Löb Problem" wherein Löb's Theorem seems to pose an obstacle to AIs building successor AIs, or adopting successor versions of their own code, that trust the same amount of mathematics as the original.  (I am currently writing up a more thorough description of the question this preliminary technical report is working on answering.  For now the main online description is in a quick Summit talk I gave.  See also Benja Fallenstein's description of the problem in the course of presenting a different angle of attack.  Roughly the problem is that mathematical systems can only prove the soundness of, aka 'trust', weaker mathematical systems.  If you try to write out an exact description of how AIs would build their successors or successor versions of their code in the most obvious way, it looks like the mathematical strength of the proof system would tend to be stepped down each time, which is undesirable.)

Paul Christiano's approach is inspired by the idea that whereof one cannot prove or disprove, thereof one must assign probabilities: and that although no mathematical system can contain its own truth predicate, a mathematical system might be able to contain a reflectively consistent probability predicate.  In particular, it looks like we can have:

∀a, b: (a < P(φ) < b)          ⇒  P(a < P('φ') < b) = 1
∀a, b: P(a ≤ P('φ') ≤ b) > 0  ⇒  a ≤ P(φ) ≤ b

Suppose I present you with the human and probabilistic version of a Gödel sentence, the Whitely sentence "You assign this statement a probability less than 30%."  If you disbelieve this statement, it is true.  If you believe it, it is false.  If you assign 30% probability to it, it is false.  If you assign 29% probability to it, it is true.

Paul's approach resolves this problem by restricting your belief about your own probability assignment to within epsilon of 30% for any epsilon.  So Paul's approach replies, "Well, I assign almost exactly 30% probability to that statement - maybe a little more, maybe a little less - in fact I think there's about a 30% chance that I'm a tiny bit under 0.3 probability and a 70% chance that I'm a tiny bit over 0.3 probability."  A standard fixed-point theorem then implies that a consistent assignment like this should exist.  If asked if the probability is over 0.2999 or under 0.30001 you will reply with a definite yes.

continue reading »
Comments (163)

Robust Cooperation in the Prisoner's Dilemma

59 orthonormal 07 June 2013 08:30AM

I'm proud to announce the preprint of Robust Cooperation in the Prisoner's Dilemma: Program Equilibrium via Provability Logic, a joint paper with Patrick LaVictoire (me), Mihaly Barasz, Paul Christiano, Benja Fallenstein, Marcello Herreshoff, and Eliezer Yudkowsky.

This paper was one of three projects to come out of the 2nd MIRI Workshop on Probability and Reflection in April 2013, and had its genesis in ideas about formalizations of decision theory that have appeared on LessWrong. (At the end of this post, I'll include links for further reading.)

Below, I'll briefly outline the problem we considered, the results we proved, and the (many) open questions that remain. Thanks in advance for your thoughts and suggestions!

Background: Writing programs to play the PD with source code swap

(If you're not familiar with the Prisoner's Dilemma, see here.)

The paper concerns the following setup, which has come up in academic research on game theory: say that you have the chance to write a computer program X, which takes in one input and returns either Cooperate or Defect. This program will face off against some other computer program Y, but with a twist: X will receive the source code of Y as input, and Y will receive the source code of X as input. And you will be given your program's winnings, so you should think carefully about what sort of program you'd write!

Of course, you could simply write a program that defects regardless of its input; we call this program DefectBot, and call the program that cooperates on all inputs CooperateBot. But with the wealth of information afforded by the setup, you might wonder if there's some program that might be able to achieve mutual cooperation in situations where DefectBot achieves mutual defection, without thereby risking a sucker's payoff. (Douglas Hofstadter would call this a perfect opportunity for superrationality...)

Previously known: CliqueBot and FairBot

And indeed, there's a way to do this that's been known since at least the 1980s. You can write a computer program that knows its own source code, compares it to the input, and returns C if and only if the two are identical (and D otherwise). Thus it achieves mutual cooperation in one important case where it intuitively ought to: when playing against itself! We call this program CliqueBot, since it cooperates only with the "clique" of agents identical to itself.

There's one particularly irksome issue with CliqueBot, and that's the fragility of its cooperation. If two people write functionally analogous but syntactically different versions of it, those programs will defect against one another! This problem can be patched somewhat, but not fully fixed. Moreover, mutual cooperation might be the best strategy against some agents that are not even functionally identical, and extending this approach requires you to explicitly delineate the list of programs that you're willing to cooperate with. Is there a more flexible and robust kind of program you could write instead?

As it turns out, there is: in a 2010 post on LessWrong, cousin_it introduced an algorithm that we now call FairBot. Given the source code of Y, FairBot searches for a proof (of less than some large fixed length) that Y returns C when given the source code of FairBot, and then returns C if and only if it discovers such a proof (otherwise it returns D). Clearly, if our proof system is consistent, FairBot only cooperates when that cooperation will be mutual. But the really fascinating thing is what happens when you play two versions of FairBot against each other. Intuitively, it seems that either mutual cooperation or mutual defection would be stable outcomes, but it turns out that if their limits on proof lengths are sufficiently high, they will achieve mutual cooperation!

The proof that they mutually cooperate follows from a bounded version of Löb's Theorem from mathematical logic. (If you're not familiar with this result, you might enjoy Eliezer's Cartoon Guide to Löb's Theorem, which is a correct formal proof written in much more intuitive notation.) Essentially, the asymmetry comes from the fact that both programs are searching for the same outcome, so that a short proof that one of them cooperates leads to a short proof that the other cooperates, and vice versa. (The opposite is not true, because the formal system can't know it won't find a contradiction. This is a subtle but essential feature of mathematical logic!)

Generalization: Modal Agents

Unfortunately, FairBot isn't what I'd consider an ideal program to write: it happily cooperates with CooperateBot, when it could do better by defecting. This is problematic because in real life, the world isn't separated into agents and non-agents, and any natural phenomenon that doesn't predict your actions can be thought of as a CooperateBot (or a DefectBot). You don't want your agent to be making concessions to rocks that happened not to fall on them. (There's an important caveat: some things have utility functions that you care about, but don't have sufficient ability to predicate their actions on yours. In that case, though, it wouldn't be a true Prisoner's Dilemma if your values actually prefer the outcome (C,C) to (D,C).)

However, FairBot belongs to a promising class of algorithms: those that decide on their action by looking for short proofs of logical statements that concern their opponent's actions. In fact, there's a really convenient mathematical structure that's analogous to the class of such algorithms: the modal logic of provability (known as GL, for Gödel-Löb).

So that's the subject of this preprint: what can we achieve in decision theory by considering agents defined by formulas of provability logic?

continue reading »
Comments (106)

Tiling Agents for Self-Modifying AI (OPFAI #2)

51 Eliezer_Yudkowsky 06 June 2013 08:24PM

An early draft of publication #2 in the Open Problems in Friendly AI series is now available:  Tiling Agents for Self-Modifying AI, and the Lobian Obstacle.  ~20,000 words, aimed at mathematicians or the highly mathematically literate.  The research reported on was conducted by Yudkowsky and Herreshoff, substantially refined at the November 2012 MIRI Workshop with Mihaly Barasz and Paul Christiano, and refined further at the April 2013 MIRI Workshop.

Abstract:

We model self-modication in AI by introducing 'tiling' agents whose decision systems will approve the construction of highly similar agents, creating a repeating pattern (including similarity of the offspring's goals).  Constructing a formalism in the most straightforward way produces a Godelian difficulty, the Lobian obstacle.  By technical methods we demonstrate the possibility of avoiding this obstacle, but the underlying puzzles of rational coherence are thus only partially addressed.  We extend the formalism to partially unknown deterministic environments, and show a very crude extension to probabilistic environments and expected utility; but the problem of finding a fundamental decision criterion for self-modifying probabilistic agents remains open.

Commenting here is the preferred venue for discussion of the paper.  This is an early draft and has not been reviewed, so it may contain mathematical errors, and reporting of these will be much appreciated.

The overall agenda of the paper is introduce the conceptual notion of a self-reproducing decision pattern which includes reproduction of the goal or utility function, by exposing a particular possible problem with a tiling logical decision pattern and coming up with some partial technical solutions.  This then makes it conceptually much clearer to point out the even deeper problems with "We can't yet describe a probabilistic way to do this because of non-monotonicity" and "We don't have a good bounded way to do this because maximization is impossible, satisficing is too weak and Schmidhuber's swapping criterion is underspecified."  The paper uses first-order logic (FOL) because FOL has a lot of useful standard machinery for reflection which we can then invoke; in real life, FOL is of course a poor representational fit to most real-world environments outside a human-constructed computer chip with thermodynamically expensive crisp variable states.

As further background, the idea that something-like-proof might be relevant to Friendly AI is not about achieving some chimera of absolute safety-feeling, but rather about the idea that the total probability of catastrophic failure should not have a significant conditionally independent component on each self-modification, and that self-modification will (at least in initial stages) take place within the highly deterministic environment of a computer chip.  This means that statistical testing methods (e.g. an evolutionary algorithm's evaluation of average fitness on a set of test problems) are not suitable for self-modifications which can potentially induce catastrophic failure (e.g. of parts of code that can affect the representation or interpretation of the goals).  Mathematical proofs have the property that they are as strong as their axioms and have no significant conditionally independent per-step failure probability if their axioms are semantically true, which suggests that something like mathematical reasoning may be appropriate for certain particular types of self-modification during some developmental stages.

Thus the content of the paper is very far off from how a realistic AI would work, but conversely, if you can't even answer the kinds of simple problems posed within the paper (both those we partially solve and those we only pose) then you must be very far off from being able to build a stable self-modifying AI.  Being able to say how to build a theoretical device that would play perfect chess given infinite computing power, is very far off from the ability to build Deep Blue.  However, if you can't even say how to play perfect chess given infinite computing power, you are confused about the rules of the chess or the structure of chess-playing computation in a way that would make it entirely hopeless for you to figure out how to build a bounded chess-player.  Thus "In real life we're always bounded" is no excuse for not being able to solve the much simpler unbounded form of the problem, and being able to describe the infinite chess-player would be substantial and useful conceptual progress compared to not being able to do that.  We can't be absolutely certain that an analogous situation holds between solving the challenges posed in the paper, and realistic self-modifying AIs with stable goal systems, but every line of investigation has to start somewhere.

Parts of the paper will be easier to understand if you've read Highly Advanced Epistemology 101 For Beginners including the parts on correspondence theories of truth (relevant to section 6) and model-theoretic semantics of logic (relevant to 3, 4, and 6), and there are footnotes intended to make the paper somewhat more accessible than usual, but the paper is still essentially aimed at mathematically sophisticated readers.

Comments (179)

Post ridiculous munchkin ideas!

47 D_Malik 15 May 2013 10:27PM

Thus spake Eliezer:

A Munchkin is the sort of person who, faced with a role-playing game, reads through the rulebooks over and over until he finds a way to combine three innocuous-seeming magical items into a cycle of infinite wish spells.  Or who, in real life, composes a surprisingly effective diet out of drinking a quarter-cup of extra-light olive oil at least one hour before and after tasting anything else.  Or combines liquid nitrogen and antifreeze and life-insurance policies into a ridiculously cheap method of defeating the invincible specter of unavoidable Death.  Or figures out how to build the real-life version of the cycle of infinite wish spells.

It seems that many here might have outlandish ideas for ways of improving our lives. For instance, a recent post advocated installing really bright lights as a way to boost alertness and productivity. We should not adopt such hacks into our dogma until we're pretty sure they work; however, one way of knowing whether a crazy idea works is to try implementing it, and you may have more ideas than you're planning to implement.

So: please post all such lifehack ideas! Even if you haven't tried them, even if they seem unlikely to work. Post them separately, unless some other way would be more appropriate. If you've tried some idea and it hasn't worked, it would be useful to post that too.

Comments (1110)

Fermi Estimates

46 lukeprog 11 April 2013 05:52PM

Just before the Trinity test, Enrico Fermi decided he wanted a rough estimate of the blast's power before the diagnostic data came in. So he dropped some pieces of paper from his hand as the blast wave passed him, and used this to estimate that the blast was equivalent to 10 kilotons of TNT. His guess was remarkably accurate for having so little data: the true answer turned out to be 20 kilotons of TNT.

Fermi had a knack for making roughly-accurate estimates with very little data, and therefore such an estimate is known today as a Fermi estimate.

Why bother with Fermi estimates, if your estimates are likely to be off by a factor of 2 or even 10? Often, getting an estimate within a factor of 10 or 20 is enough to make a decision. So Fermi estimates can save you a lot of time, especially as you gain more practice at making them.

 

Estimation tips

These first two sections are adapted from Guestimation 2.0.

Dare to be imprecise. Round things off enough to do the calculations in your head. I call this the spherical cow principle, after a joke about how physicists oversimplify things to make calculations feasible:

Milk production at a dairy farm was low, so the farmer asked a local university for help. A multidisciplinary team of professors was assembled, headed by a theoretical physicist. After two weeks of observation and analysis, the physicist told the farmer, "I have the solution, but it only works in the case of spherical cows in a vacuum."

By the spherical cow principle, there are 300 days in a year, people are six feet (or 2 meters) tall, the circumference of the Earth is 20,000 mi (or 40,000 km), and cows are spheres of meat and bone 4 feet (or 1 meter) in diameter.

Decompose the problem. Sometimes you can give an estimate in one step, within a factor of 10. (How much does a new compact car cost? $20,000.) But in most cases, you'll need to break the problem into several pieces, estimate each of them, and then recombine them. I'll give several examples below.

Estimate by bounding. Sometimes it is easier to give lower and upper bounds than to give a point estimate. How much time per day does the average 15-year-old watch TV? I don't spend any time with 15-year-olds, so I haven't a clue. It could be 30 minutes, or 3 hours, or 5 hours, but I'm pretty confident it's more than 2 minutes and less than 7 hours (400 minutes, by the spherical cow principle).

Can we convert those bounds into an estimate? You bet. But we don't do it by taking the average. That would give us (2 mins + 400 mins)/2 = 201 mins, which is within a factor of 2 from our upper bound, but a factor 100 greater than our lower bound. Since our goal is to estimate the answer within a factor of 10, we'll probably be way off.

Instead, we take the geometric mean — the square root of the product of our upper and lower bounds. But square roots often require a calculator, so instead we'll take the approximate geometric mean (AGM). To do that, we average the coefficients and exponents of our upper and lower bounds.

So what is the AGM of 2 and 400? Well, 2 is 2×100, and 400 is 4×102. The average of the coefficients (2 and 4) is 3; the average of the exponents (0 and 2) is 1. So, the AGM of 2 and 400 is 3×101, or 30. The precise geometric mean of 2 and 400 turns out to be 28.28. Not bad.

What if the sum of the exponents is an odd number? Then we round the resulting exponent down, and multiply the final answer by three. So suppose my lower and upper bounds for how much TV the average 15-year-old watches had been 20 mins and 400 mins. Now we calculate the AGM like this: 20 is 2×101, and 400 is still 4×102. The average of the coefficients (2 and 4) is 3; the average of the exponents (1 and 2) is 1.5. So we round the exponent down to 1, and we multiple the final result by three: 3(3×101) = 90 mins. The precise geometric mean of 20 and 400 is 89.44. Again, not bad.

Sanity-check your answer. You should always sanity-check your final estimate by comparing it to some reasonable analogue. You'll see examples of this below.

Use Google as needed. You can often quickly find the exact quantity you're trying to estimate on Google, or at least some piece of the problem. In those cases, it's probably not worth trying to estimate it without Google.

continue reading »
Comments (105)

Prisoner's Dilemma (with visible source code) Tournament

44 AlexMennen 07 June 2013 08:30AM

After the iterated prisoner's dilemma tournament organized by prase two years ago, there was discussion of running tournaments for several variants, including one in which two players submit programs, each of which are given the source code of the other player's program, and outputs either “cooperate” or “defect”. However, as far as I know, no such tournament has been run until now.

Here's how it's going to work: Each player will submit a file containing a single Scheme lambda-function. The function should take one input. Your program will play exactly one round against each other program submitted (not including itself). In each round, two programs will be run, each given the source code of the other as input, and will be expected to return either of the symbols “C” or “D” (for "cooperate" and "defect", respectively). The programs will receive points based on the following payoff matrix:

“Other” includes any result other than returning “C” or “D”, including failing to terminate, throwing an exception, and even returning the string “Cooperate”. Notice that “Other” results in a worst-of-both-worlds scenario where you get the same payoff as you would have if you cooperated, but the other player gets the same payoff as if you had defected. This is an attempt to ensure that no one ever has incentive for their program to fail to run properly, or to trick another program into doing so.

Your score is the sum of the number of points you earn in each round. The player with the highest score wins the tournament. Edit: There is a 0.5 bitcoin prize being offered for the winner. Thanks, VincentYu!

Details:
All submissions must be emailed to wardenPD@gmail.com by July 5, at noon PDT. Your email should also say how you would like to be identified when I announce the tournament results.
Each program will be allowed to run for 10 seconds. If it has not returned either “C” or “D” by then, it will be stopped, and treated as returning “Other”. For consistency, I will have Scheme collect garbage right before each run.
One submission per person or team. No person may contribute to more than one entry. Edit: This also means no copying from each others' source code. Describing the behavior of your program to others is okay.
I will be running the submissions in Racket. You may be interested in how Racket handles time (especially the (current-milliseconds) function), threads (in particular, “thread”, “kill-thread”, “sleep”, and “thread-dead?”), and possibly randomness.
Don't try to open the file you wrote your program in (or any other file, for that matter). I'll add code to the file before running it, so if you want your program to use a copy of your source code, you will need to use a quine. Edit: No I/O of any sort.
Unless you tell me otherwise, I assume I have permission to publish your code after the contest.
You are encouraged to discuss strategies for achieving mutual cooperation in the comments thread.
I'm hoping to get as many entries as possible. If you know someone who might be interested in this, please tell them.
It's possible that I've said something stupid that I'll have to change or clarify, so you might want to come back to this page again occasionally to look for changes to the rules. Any edits will be bolded, and I'll try not to change anything too drastically, or make any edits late in the contest.

Here is an example of a correct entry, which cooperates with you if and only if you would cooperate with a program that always cooperates (actually, if and only if you would cooperate with one particular program that always cooperates):

(lambda (x)
    (if (eq? ((eval x) '(lambda (y) 'C)) 'C)
        'C
        'D))

Comments (168)

New report: Intelligence Explosion Microeconomics

44 Eliezer_Yudkowsky 29 April 2013 11:14PM

Summary: Intelligence Explosion Microeconomics (pdf) is 40,000 words taking some initial steps toward tackling the key quantitative issue in the intelligence explosion, "reinvestable returns on cognitive investments": what kind of returns can you get from an investment in cognition, can you reinvest it to make yourself even smarter, and does this process die out or blow up? This can be thought of as the compact and hopefully more coherent successor to the AI Foom Debate of a few years back.

(Sample idea you haven't heard before:  The increase in hominid brain size over evolutionary time should be interpreted as evidence about increasing marginal fitness returns on brain size, presumably due to improved brain wiring algorithms; not as direct evidence about an intelligence scaling factor from brain size.)

I hope that the open problems posed therein inspire further work by economists or economically literate modelers, interested specifically in the intelligence explosion qua cognitive intelligence rather than non-cognitive 'technological acceleration'.  MIRI has an intended-to-be-small-and-technical mailing list for such discussion.  In case it's not clear from context, I (Yudkowsky) am the author of the paper.

Abstract:

I. J. Good's thesis of the 'intelligence explosion' is that a sufficiently advanced machine intelligence could build a smarter version of itself, which could in turn build an even smarter version of itself, and that this process could continue enough to vastly exceed human intelligence.  As Sandberg (2010) correctly notes, there are several attempts to lay down return-on-investment formulas intended to represent sharp speedups in economic or technological growth, but very little attempt has been made to deal formally with I. J. Good's intelligence explosion thesis as such.

I identify the key issue as returns on cognitive reinvestment - the ability to invest more computing power, faster computers, or improved cognitive algorithms to yield cognitive labor which produces larger brains, faster brains, or better mind designs.  There are many phenomena in the world which have been argued as evidentially relevant to this question, from the observed course of hominid evolution, to Moore's Law, to the competence over time of machine chess-playing systems, and many more.  I go into some depth on the sort of debates which then arise on how to interpret such evidence.  I propose that the next step forward in analyzing positions on the intelligence explosion would be to formalize return-on-investment curves, so that each stance can say formally which possible microfoundations they hold to be falsified by historical observations already made.  More generally, I pose multiple open questions of 'returns on cognitive reinvestment' or 'intelligence explosion microeconomics'.  Although such questions have received little attention thus far, they seem highly relevant to policy choices affecting the outcomes for Earth-originating intelligent life.

The dedicated mailing list will be small and restricted to technical discussants.

continue reading »
Comments (239)

View more: Next

Powered by Reddit Powered by Reddit