I'd love to see a discussion between people like LeCun, Norvig, Yudkowsky and e.g. Russell. A discussion where they talk about what exactly they mean when they think about "AI risks", and why they disagree, if they disagree.

Right now I often have the feeling that many people mean completely different things when they talk about AI risks. One person might mean that a lot of jobs will be gone, or that AI will destroy privacy, while the other person means something along the lines of "5 people in a basement launch a seed AI, which then turns the world into computronium". These are vastly different perceptions, and I personally find myself somewhere between those positions.

LeCun and Norvig seem to disagree that there will be an uncontrollable intelligence explosion. And I am still not sure what exactly Russell believes.

Anyway, it is possible to figure this out. You just have to ask the right questions. And this never seems to happen when MIRI or FHI talk to experts. They never specifically ask about their controversial beliefs. If you e.g. ask someone if they agree that general AI could be a risk, a yes/no answer provides very little information about how much they agree with MIRI. You'll have to ask specific questions.

what blind spots did 12-14-year-old you have

Heh, if I was 12-14 these days, the main message I would send to me would be: Start making and publishing mobile games while you have a lot of free time, so when you finish university, you have enough passive income that you don't have to take a job, because having a job destroys your most precious resources: time and energy.

(And a hyperlink or two to some PUA blogs. Yeah, I know some people object against this, but this is what I would definitely send to myself. Sending it to other kids would be more problematic.)

I would recommend Anki only for learning languages. For other things I would recommend writing notes (text documents); although this advice may be too me-optimized. One computer directory called "knowledge", subdirectories per subject, files per topic -- that's a good starting structure; you can change it later, if you need. But making notes becomes really important at the university level.

I would stress the importance of other things than math. Gifted kids sometimes focus on their strong skills, and ignore their weak skills -- they put all their attention to where they receive praise. This is a big mistake. However, saying this without providing actionable advice does not help. For example, my weak spots were exercise and social skills. For social skills a list of recommended books could help; with emphasis that I should not only read the books, but also practice what I learned. For exercise, a simple routine plus HabitRPG could do the job. Maybe to emphasise that I should not focus on how I compare with others, but how I compare with yesterday's me.

Something about an importance of keeping contact with smart people, and insanity of the world in general. As a smart person, talking with other smart people increases your powers: both because you develop with them the ideas you understand, and because you can ask them about things you don't understand. (A stupid person will not understand what you are saying, and will give you harmful advice about things you asked.) In school you are supposed to work alone, but in real life a lot of success is achieved by teams; but the best teams are composed of good people, not of random people.

Another advice that is risky to give to other kids: Religion is bullshit and a waste of time. People will try to manipulate you, using lies and emotional pressure. Whatever other positive traits they have, try to find other people that have the same positive traits, but without the mental poison; even if it takes more time, it's worth it.

what blind spots did 12-14-year-old you have

I don't know how much of this falls under your remit, but I had quite a few educational blind-spots I inherited from my parents, who didn't come from a higher-educated background. If any of your students are in a similar position, it's worth checking they don't have any ludicrous expectations out of the next several years of education which no-one close to them is in a position to correct.

I think most of my blindspots before roughly the age of 18 involved not understanding that I'm personally responsible for my success and the extent of my knowledge and that "good enough" doesn't cut it. If I were to send a message back to 14-year-old!Me, I'd tell him that he has a lot of potential, but that he can't rely on others to fulfill that potential.

Speaking as a somewhat gifted seventeen year old, I'd really like to have known about AoPS, HPMOR and the Sequences.

Also, I'd like to have had in my mind the notion that my formal education is not optimised for me, and that I really need to optimise it myself. Speaking more concretely, I think that most teenagers in Britain pick their A Levels (if they do them at all) based on what classes the other people around them are doing, which isn't very useful. Speaking to a friend though, I realised that when he was picking his third A Level to study, there was no other A Level he needed to study to get into his main area of specialisation (jazz musician), and his time would be better spent not doing the A level at all; he needed to think more meta. He was just doing an A level because that's what everyone seems to think you should do. I'm about to give up a class because it's not going to help me get anywhere, I can use the time better and learn what I want to better alone anyway. So, really optimise.

Don't know if that helps. And AoPS is ridiculously useful.

I have been in such a program when I was 12-14 (run by the William Stern foudnation in Hamburg, Germany) and the curriculum consisted mostly of very diverse 'math' problems prepared in a way to make them accessible to us in a func way without introducing too much up-front terminology or notation. Examples I remember of the spot:

• turing machines (dresses as short-sighted busy beavers)

• generalized Nim really with lots of matches

• tilings of the plane

• conveys game of life (easy on paper)

More I just looked up in an older folder:

• distance metrics on a graph

• multi-way balances

• continuous fractions (cool for approximations; I still use this)

• logical derivations about beliefs of people whose dream are indistinuishable from reality

• generalized magical squares

• Fibinacci sequences and http://en.wikipedia.org/wiki/Missing_square_puzzle

• Drawing fractals (the iterated function ones; with printouts of some)

In general only an exposition was given and no task to solve. Or some introductory initial questions, The patterns to be detected were the primary reward.

We were not introduced to really practical applications but I'm unsure whether that had been helpful or rather whether it had been interesting. My interest at that time stemmed from the material being systematic patterns that I could approach abstractly and symbolically and 'solve'. I'm not clear whether the Sequences would have been interesting in that way. Their patterns are clear only in hindsight.

What should work is Bayes rule - at least in the form that can be visualized (tiling of the 1/1 grid) or symbolcally derived easily.

Also guessing and calibration games should work. You can also take standard games and add some layer of complexity on them (but please not arbitrary but helpful ones; a minimum example is: Play Uno but cards don't have to match color+number but some number theoretic identity e.g. +(2,5) modulo (4,10)).

Just curious -- are you teaching at a math camp? Which one? (I have a lot of friends from Canada/USA Mathcamp, although I didn't go myself.)

what blind spots did 12-14-year-old you have

• Social capital is important. Build it.
• Peer pressure is far more common and far more powerful than you think. Find an ingroup that puts it to constructive ends.
• Don't major in a non-STEM field. College is job training and a networking opportunity. Act accordingly.
• Something about time management, pattern-setting, and motivation management -- none of which I've managed to learn yet.

Update: They made it.

My guess is that to a human observer they would seem like a random noise. -- Which contradicts my initial words that the sequence would not be what you'd expect... but I guess the answer is that the generation process is trying to surprise the prediction machine, not me as a human.

"What is the specific pattern of bits?" and "Give a vague description that applies to both this pattern and asymptotically 100% of possible patterns of bits" are very different questions. You're asking the machine the first question and the human the second question, so I'm not surprised the answers are different.

In order to capture your intuition that a random sequence is "unsurprising", you want the predictor to output a distribution over {0,1} — or equivalently, a subjective probability p of the next bit being 1. The predictor tries to maximize the expectation of a proper scoring rule. In that case, the maximally unexpected sequence will be random, and the probability of the sequence will approach 2^{-n}.

Allowing the predictor to output {0, 1, ?} is kind of like restricting its outputs to {0%, 50%, 100%}.

I think adding a little meta-probability will help.

Since there's some probability of the sequence being "the most surprising", this would basically mean that several of the most surprising end up with basically the same probability. For example, if it takes n bits of data to define "the most surprising m-bit sequence", then there must be a 2^-n chance of that happening. Since there are 2^m sequences, and the most surprising sequence must have a probability of at most 2^-m, there must be at least 2^(m-n) most surprising sequences.

or whether anyone has explored this before

I have briefly thought about this idea in the context of password selection and password crackers: the "most unexpected" string (of some maximum length) is a good password. No deep reasoning here though.

Just in case anyone wants pointers to existing mathematical work on "unpredictable" sequences: Algorithmically random sequences (wikipedia)

My prediction machine can maximize it's expected minimum score by outputting random guesses. Then your bitstring is precisely the complement of my random string, and therefore drawn from the random distribution.

Does "most unexpected" differ from "least predictable" in any way? Seems like a random number generator would match any algorithm around 50% of the time so making an algorithm less predictable than that is impossible no?

The latter. The former is already decreasing at an incredible speed but I see no trend for the latter.

The formed is a major existential risk, while the latter is probably going to be solved soon(er), so the former.

The former. Stopping ageing without giving us time to prepare for it would cause all sorts of problems in terms of increasing population. Whereas stopping violence would accelerate progress no end (if only for the resources it freed up).

I'm much more likely to die of aging than of violence; so I'd rather stop aging.

This seems to generalize well to the rest of humanity. I am surprised that most others who replied disagrees. ISTM that most existential risks are not due to deliberate violence, but rather unintended consequences.

Ending aging would almost certainly greatly diminish human-on-human violence, since increasing expected lifespans would lower time preference. Right?

My problem with these questions is that it sorta gets difficult quickly. If you stopped aging today, I imagine there would very quickly be overpopulation issues and many old patients in hospitals wouldn't die etc. and yet I am finding it difficult to think of major issues with the ending of violence (boxing champions would be out of a job). And even now, I'm sure someone's thought of a counter example, and then the discussion would be harder. And so even though I think that aging is more important than violence as a focus, the question asks a hypothetical that is never going to occur (being able to just make that decision, I mean) and takes us away from reality into the nitty/gritty of a literal non-problem.

Why did you ask?

Edit: I didn't mean to make a case for either side, I was trying to suggest that the question itself seems unhelpful. We'll end up with a complicated technical discussion which is unlikely to have any practical value.

Not an expert, but Euclid made some mistakes, like using superposition to prove some theorems. I'm curious how they handle those. (e.g. I think Euclid attempted to prove side-angle-side congruence, but Hilbert had to include it as an axiom.)

(Are you suggesting that Hofstadter deserves a similar status to e.g. Dostoyevsky? Are you insane or what? Look at the size of your faction, our faction, and think again.)

I'm not sure about this - the "Yay Hofstadter" team looks about as big as the "Yay Dostoyevsky" team, at least especially in the anglophone internet.

I think you're overcomplicating it. People like Eliezer Yudkowsky and Paul Graham are certainly not cult leaders, but they have many strong opinions that are well outside the mainstream; they don't believe in, and in fact actively scorn, hedging/softening their expression of these opinions; and they have many readers, a visible subset of whom uncritically pattern all their opinions, mainstream or not, after them.

And pushback against excitement over Hofstadter can stem from legitimate disagreement about the importance/interestingness of his work. The pushback is proportional to the excitement that incites it.

I was talking to the loved one about this last night. She is going for ministry in the Church of England. (Yes, I remain a skeptical atheist.)

She is very charismatic (despite her introversion) and has the superpower of convincing people. I can just picture her standing up in front of a crowd and explaining to them how black is white, and the crowd each nodding their heads and saying "you know, when you think about it, black really is white ..." She often leads her Bible study group (the sort with several translations to hand and at least one person who can quote the original Greek) and all sorts of people - of all sorts of intelligence levels and all sorts of actual depths of thinking - get really convinced of her viewpoint on whatever the matter is.

The thing is, you can form a cult by accident. Something that looks very like one from the outside, anyway. If you have a string of odd ideas, and you're charismatic and convincing, you can explain your odd ideas to people and they'll take on your chain of logic, approximately cut'n'pasting them into their minds and then thinking of them as their own thoughts. This can result in a pile of people who have a shared set of odd beliefs, which looks pretty damn cultish from the outside. Note this requires no intention.

As I said to her, "The only thing stopping you from being L. Ron Hubbard is that you don't want to. You better hope that's enough."

(Phygs look like regular pigs, but with yellow wings.)

There is nothing bad per se about suggesting that e.g. Douglas R. Hofstadter should be the king of the nonconformist tribe.

Disagreed. IMO, there should only be kings if there's a good reason... among other things, I suspect that status differences are epistemologically harmful. See Stanley Milgram's research and the Asch conformity experiment.

I also disagree with the rest of your analysis. I anticipate a different sense of internal revulsion when someone starts talking to me about why Sun Myung Moon is super great vs why Mike Huckabee is so great or why LeBron James is so great. In the case of LW, I think people whose intuitions say "cult" are correct to a small degree... LW does seem a tad insular, groupthink-ish, and cultish to me, though it's still one of my favorite websites. And FWIW, I would prefer that people who think LW seems cultish help us improve (by contributing intelligent dissent and exposing us to novel outside thinking) instead of writing us off.

(The most charitable interpretation of the flaws I see in LW is that they are characteristics that trade off against some other things we value. E.g. if we downvoted sloppy criticism of LW canon less, that would mean we'd get more criticism of LW canon, both sloppy and high-quality... not clear whether this would be good or not, though I'm leaning towards it being good. A less charitable interpretation is that the length of the sequences produces some kind of hazing effect. Personally, I haven't finished the sequences, don't intend to, think they're needlessly verbose, and would like to see them compressed.)

I assumed it was because it motivated people into becoming much more productive.

How in the world did that work?

It didn't. This is the argument in image form, and you can find similar ones for employment (basically, when you conscript people, unemployment goes down. Shocking!). There are lots of libertarian articles on the subject--this might be an alright introduction--but the basic argument is that standards of living dropped (that's what happens when food is rationed and metal is used for tanks instead of cars or household appliances) but the government spending on bombs and soldiers made the GDP numbers go up, and then the post-war boost in standards of living was mostly due to deferred spending.

One simple model which seems to fit the "WWII ending the depression" piece of data (and which might have some overlap with the truth) is that it's relatively difficult to put idle resources into use, and significantly easier to repurpose resources that have been in use for other uses.

During the depression, a bunch of people were unemployed, factories were not running, storefronts were empty, etc. According to this model, under those economic conditions there were significant barriers to taking those idle resources and putting them to productive use.

Then WWII came and forced the country to mobilize and put those resources to use (even if that use was just to make stuff which would be shipped off to Europe and the Pacific to be destroyed). Once the war was over, those resources which had been devoted to war could be repurposed (with relatively little friction) to uses with a much more positive effect on people's standard of living. So things became good according to meaningful metrics like living standards, not merely according to metrics like unemployment rate or total output which ignore the fact that building a tank to send to war isn't valuable in the same way as building a car for local consumers.

The glaring open question here is why there might be this asymmetry between putting idle resources to use and repurposing in-use resources. Which is closely related to the question of why recessions/depressions exist at all (as more than momentary blips): once a recession hits and bunch of people become unemployed (and other resources go idle), why doesn't the market immediately jump in to snap up those idle resources? This article gets into some of the attempts to answer those questions.

(Bonus answer: World War One did not happen during a depression, so mobilizing for war mostly involved repurposing resources which had served other uses in peacetime rather than bringing idle resources into use.)

Part of it is that deflation in the early 1930s meant that workers were overpaid relative to the value of goods they produced (wages being harder to cut than prices). That caused wasteful amounts of unemployment. WWII triggered inflation, and combined with wage controls caused wages to become low relative to goods, shifting the labor supply and demand to the opposite extreme.

The people who were employed pre-war presumably had their standard of living lowered in the war (after having it increased a good deal during the deflation).

I won't try to explain here why deflation and inflation happened when they did, or why wages are hard to cut (look for "sticky wages" for info about the latter).

I'm not sure how much it influenced the overall picture, but there was quite a brain drain to the US before and during WWII (mostly Jewish refugees) as well as after (Wernher von Braun and the like). Migrating away from the Nazi and Stalinist spheres of influence demonstrates intelligence, and the ability to enter the US despite the complex “national origins quota system” that went into effect in 1929 demonstrates persistence, affluence and/or marketable skills, so I estimate these immigrants gave a significant boost to the US economy.

The labor force of the 1930s was sapped by over-allocation in unproductive industries. Specifically, much of the labor share was occupied in the sitting around feeling depressed and wishing you had a job industry. Economic conditions improved as workers shifted out of that industry and into more productive ones, such as all of them.

However this goes, this seems to be a huge argument in favor of big-government spending (if we get this much utility from the government building things that literally explode themselves without providing non-military utility, then in a time of peace, we should be able to get even more by having the government build things like high-tech infrastructure, places of beauty, peaceful scientific research, large-scale engineering projects, etc.). So should we be spending 20-40% of our GDP on peace-time government mega-projects? It's either that or this piece of common knowledge is wrong (and we all know how reliable common knowledge is!).

I'm surprised no one has explained this yet, but this is wrong according to standard economic theory as I understand it.

1. The United States suffered from terrible monetary policy during the Great Depression.
2. Due to "animal spirits" and "sticky wages" this caused large scale unemployment and output well below our production possibilities frontier.
3. World War II caused the government to kickstart production for the war effort.
4. Living standards actually didn't rise, although GDP did (GDP per capita is NOT the same as living standards). Consumption was dramatically deferred during the war. People had fewer babies, bought fewer consumer products (and fewer were produced) and shifted toward home production for some necessities.
5. There was a short recession as the end of the war lowered demand, but pent-up consumer demand quickly re-stabilized the economy.

The point is WWII helped the economy because we were well under our production possibilities frontier during the depression. Peace-time mega projects would only be helpful under recessed/depressed conditions, and fortunately, we now can use monetary policy to produce similar effects.

Anyway, the argument you were making seems pretty common among people who don't follow economics debates, and in fact is one of the major policy recommendations of the oddball Lyndon LaRouche cult.

I had the notion a while ago to try to write a linter to aid in tasks beyond code correctness by automatically detecting the desired features in a plethora of objects. Kudos on actually doing it and in a not hare-brained fashion.

As a former Natural Language Processing researcher, the technology definitely exists. Using general vocabulary combined with many (semi-manually generated) regexes to figure out argumentative or weaselly sentences with decent accuracy should be doable. It could improve over time if you input exemplar sentences you came across.

"[...]may be the case[...]"

Sometimes this phrase is harmless, but sometimes it is part of an important enumeration of possible outcomes/counterarguments/whatever. If "the case" does not come with either a solid plan/argument or an explanation why it is unlikely or not important, then it is often there to make the author and/or the audience feel like all the bases have been covered. E.g.,

We should implement plan X. It may be the case that [important weak point of X], but [unrelated benefit of X].

I think this is a good idea, and I'm curious to see how it goes. I'll be watching, and as I complete some of my other writing duties I think this has a good chance of becoming one.

Something else that might be interesting: this comment and the idea it's a response to in the OP.

Try posting some problems in the open threads here. MathOverflow has also worked really well for me.

Sure, lay it on us. If nothing else, writing it up clearly should help you.

Without citation: minimal "organic" labeling standards often aren't a very high or impressive barrier to clear.

"Not having a word for it"? Or in the technical vocabulary of linguistics, the concept is not "lexicalised".

Not quite what you're looking for I think, but if someone is having that problem they might have anomic aphasia.

I've heard "anomia" and "being able to talk all around the idea of an [X] but not the word [X] itself".

Sounds kind of like the Tip of the Tongue Effect

Had a look at your link, but couldn't make sense of it. Consider writing a proper summary upfront.

I want to build a better formalization of naturalized induction than Solomonoff's

This seems an ambitious task. Can you start with something simpler?

If a government produces goods, the results tend to be low quality (education may be an exception in some places).

The cost of a guaranteed minimum income may not be quite as high as you think-- it would replace a lot of more complicated government support. Also, it might be possible to build in some social rewards for not taking it if you don't need it.

A sharp divide between basic, subsidized, no-frills good and services and other ones didn't work in the socialist German Democratic Republic (long story, reply if you need it). What does seem to be for various countries is different rates of value-added tax depending on the good or service - the greater the difference in taxation, the closer you get to the system you've described, but it is more gradual and can be fine-tuned. Maybe that could work for sales tax, too?

If the government would give each person say 30 % of GDP per capita to each person (not a very high standard of living), then that would force them to raise 30 % of GDP in taxes to cover for that.

In 2002, total U.S. social welfare expenditure constitutes over 35% of GDP

I think that would be too high anyway. Since anyone who bothers to work can make more than that, and the reduction in labor supply would increase pay, and any money you save will last you longer, there's little reason to make it enough for people to be well off, as opposed to getting just enough to scrape by.

It's also worth noting that most people will get a significant portion of that money back. If you make below the mean income (which most people do, since it's positively skewed) you will end up getting all of it back.

It seems unfair to charge people the entire price to get slightly better goods. Thus, if you want to get slightly better goods, the government should still reimburse you for the price of the cheap goods. At this point, it's just unconditional basic income with the government selling cheap goods.

As a minor point, Soylent as it is now can't be considered no-frills food. If you buy it ready-made, it costs around $10 a day.

"Those who want to distinguish themselves from the masses - who want to consume conspiciously - will also be affected, since they will have to spend less to stand out from the crowd" - maybe I've misunderstood this, but surely it would have the opposite result? Let's say rents are ~$20/sqm (adjust for your own city; the principle stays the same). If I want my apartment to be 50 sqm rather than 40 sqm, that's an extra $200. But if 40 sqm apartments were free, the price difference would be the full $1000/month price of the bigger apartment. You've still got a cliff, just like in the means-tested welfare case; it's just that now it's on the consumption side.

In practice this would probably destroy the market for mid-priced goods - who wants to pay $1000/month just for an extra 10 square meters? Non-subsidized goods will only start being attractive when they get much better than the stuff the government provides, not just slightly better.

Also, if you give out goods rather than money, you're going to have to provide a huge range of different goods/services, because otherwise there will be whole categories of products that people who legitimately can't work (elderly, disabled etc) won't have access to. And if you do that, the efficiency of your economy is going to go way down - not just because the government is generally less efficient than the free market, but also because people can't use money to allocate resources according to their own preferences.

The advantage over the basic income system is that it would be considerably cheaper, since these goods and services would only be used by a part of the population. The advantage over the means-tested system is that people will still be allowed to use these goods and services if their income goes up, so it doesn't create perverse incentives.

The universal basic income schemes that seem the most reasonable to me adjust the taxation so that, while the UBI itself is never taxed, if you make a lot of money then your non-UBI earnings get an extra tax so that the whole reform ends up having very little direct effect on you. In effect, that ends up covering the "only used by a part of the population" criteria. The perverse incentives can't be avoided entirely, but they can be mitigated somewhat if the tax system is set up so that you're always better off working than not working.

For a concrete example, there's e.g. this 2007 proposal by the Finnish Green party. Your working wage (in euros per month) is on the X-axis, your total income after is on the Y-axis. Light green is the basic income, dark green is your after-tax wage, red is paid in tax. According to their calculations, this scheme would have been approximately cost neutral (compared to what the Finnish state normally gets in tax income and pays out in welfare).

One middle route I've been toying with is that the government could provide people with cheap goods and services.

This is a popular practice in the third world.

See e.g. this or this.

how is this better than Walmart and Mcdonalds?

That makes sense. Even cooperating across short time frames might be problematic - "I'll stay in bed for 10 more minutes, even if it means that me-in-10-minutes will be stressed out and might be late for work"

I prefer to see long-term thinking as increased integration among different time-selves rather than a sacrifice, though - it's not a sacrifice to take actions with a delayed payoff if your utility function puts a high weight on your future-selves' wellbeing.

See https://www.google.com/search?q=picoeconomics

I don't think Tetlock talks about that much.

Imagine a better forecast about whether invading Iraq reduces terrorism, or about whether Saddam would survive the invasion. Wouldn't both sides make wiser decisions?

Improved forecasting might mean that both sides do fewer stupid (negative sum) things.

improved political forecasting is good. .... Why is improved forecasting not zero sum?

For the simple reason that politics is not zero-sum, foreign policy included.

A greatly simplified example: two countries are having a dispute and the tension keeps rising. They both believe that they can win against the other in a war, meaning neither side is willing to back down in the face of military threats. Improved forecasting would indicate who would be the likely winner in such a conflict, and thus the weaker side will preemptively back down.

International politics is zero-sum once you've already reached the Pareto frontier and can only move along it, but if forecasting is sufficiently bad you might not even be close to the Pareto frontier.

This might be counter-intuitive and impractical for self-teaching, but for me it was an assembly language course that made it 'click' for how things work behind the scenes. It doesn't have to be much and you'll probably never use it again, but the concepts will help your broader understanding.

If you can be more specific about which parts baffle you, I might be able to recommend something more useful.

Could you give a couple examples of specific things that you'd like to understand?

Without that, a classic that might match what you're interested in is Structure and Interpretation of Computer Programs. It starts as an introduction to general programming concepts and ends as an introduction to writing interpreters.

The square brackets are greedy. What you want to do is this:

\[Link\]: [Why do people persist in believing things that just aren't true?](http://www.newyorker.com/online/blogs/mariakonnikova/2014/05/why-do-people-persist-in-believing-things-that-just-arent-true.html?utm_source=www&utm_medium=tw&utm_campaign=20140519&mobify=0)

which looks like:

[Link]: Why do people persist in believing things that just aren't true?

but the seller gets less/none, and the rest goes to charity. Seller has no right to complain.

Why would there be any sellers under this system?