Yes, this is a really nice open thread that seems to be working well.

Will have you ever had an encounter with the divine?

I don't quite have the mathematical background and sophistication to grok those posts as well, but I did get their purpose - to hook mathematicians into thinking about the open problems that Eliezer and MIRI have identified as being relevant.

The most apt description I've found is something along the lines of "consciousness is what information-processing feels like from the inside."

It's not just about the what a brain does, because a simulated brain would still be conscious, despite not being made of neurons. It's about certain kinds of patterns of thought (not the physical neural action, but thought as in operation performed on data). Human brains have it, insects don't, anything in between is something for actual specialists to discuss. But what it is - the pattern of data processing - isn't all that mysterious.

Depends on what you mean by new: I elaborated on some of my core ideas about the field in the blog posts Why edugames don't have to suck, Videogames will revolutionize school (not necessarily the way you think), and also touched upon their role in society in Doing Good in the Addiction Economy. My thoughts have gotten somewhat more precise, but off-hand I can't think of any major recent insights that I wouldn't have mentioned in those posts.

On the topic of the educational game that I'm doing for my Master's Thesis, I'm making slow but sure progress.

What is the intended audience for this game? Why, do you think, people will play it?

Hmm. I haven't really looked into any actual funding agencies or the "getting money for this" side at this point, so I don't know much about that, but I can think of some researchers who might either have an interest in collaborating, or who could know more direct sources of funding. Two groups that come to mind who might be worth contacting in this regard are GAPS and Institute of Play. I'll let you know if I think of any others. (If you do contact them, I'd be curious to hear about the response.)

Ethics, I suppose. Most of my other interests are either probably too mindkilling for LW or are written about in the Sequences already, more clearly than I could write about them.

Is this a fist-fight or can blacktrance use weapons?

Depends on the situation. Do I have to kill whatever I'm fighting, or do I just have to defend myself? If it's the former, the horse-sized duck, because duck-sized horses would be too good at running away and hiding. If it's the latter, then the duck- horses, because they'd be easier to scatter.

What do you feel are the most pressing unsolved problems in AGI?

In AGI? If you mean "what problems in AI do we need to solve before we can get to the human level", then I would say:

• Ability to solve currently intractable statistical inference problems (probably not just by scaling up computational resources, since many of these problems have exponentially large search spaces).
• Ways to cope with domain adaptation and model mis-specification.
• Robust and modular statistical procedures that can be fruitfully fit together.
• Large amounts of data, in formats helpful for learning (potentially including provisions for high-throughput interaction, perhaps with a virtual environment).

To some extent this reflects my own biases, and I don't mean to say "if we solve these problems then we'll basically have AI", but I do think it will either get us much closer or else expose new challenges that are not currently apparent.

Do you believe AGI can "FOOM" (you may have to qualify what you interpret FOOM as)?

I think it is possible that a human-level AI would very quickly acquire a lot of resources / power. I am more skeptical that an AI would become qualitatively more intelligent than a human, but even if it was no more intelligent than a human but had the ability to easily copy and transmit itself, that would already make it powerful enough to be a serious threat (note that it is also quite possible that it would have many more cycles of computation per second than a biological brain).

In general I think this is one of many possible scenarios, e.g. it's also possible that sub-human AI would already have control of much of the world's resources and we would have built systems in place to deal with this fact. So I think it can be useful to imagine such a scenario but I wouldn't stake my decisions on the assumption that something like it will occur. I think this report does a decent job of elucidating the role of such narratives (not necessarily AI-related) in making projections about the future.

How viable is the scenario of someone creating a AGI in their basement, thereby changing the course of history in unpredictable ways?

Not viable.

We brainstormed things that we know now that we wished we had known in high school. During the first year, we just made courses out of those (also borrowing from CFAR workshops) and rolled with that, because we didn't really know what we were doing and just wanted to get something off the ground.

Over time we've asked ourselves what the common thread is in our various courses, in an attempt to develop a more coherent curriculum. Three major themes are statistics, programming, and life skills. The thing these have in common is that they are some of the key skills that extremely sharp quantitative minds need to apply their skills to a qualitative world. Of course, it will always be the case that most of the value of SPARC comes from informal discussions rather than formal lectures, and I think one of the best things about SPARC is the amount of time that we don't spend teaching.

I don't personally work on AGI and I don't think the majority of "AGI progress" comes from people who label themselves as working on AGI. I think much of the progress comes from improved tools due to research and usage in machine learning and statistics. There are also of course people in these fields who are more concerned with pushing in the direction of human-level capabilities. And progress everywhere is so inter-woven that I don't even know if thinking in terms of "number of AI researchers" is the right framing. That said, I'll try to answer your question.

I'm worried that I may just be anchoring off of your two numbers, but I think 10^3 is a decent estimate. There are upwards of a thousand people at NIPS and ICML (two of the main machine learning conferences), only a fraction of those people are necessarily interested in the "human-level" AI vision, but also there are many people who are in the field who don't go to these conferences in any given year. Also many people in natural language processing and computer vision may be interested in these problems, and I recently found out that the program analysis community cares about at least some questions that 40 years ago would have been classified under AI. So the number is hard to estimate but 10^3 might be a rough order of magnitude. I expect to find more communities in the future that I either wasn't aware of or didn't think of as being AI-relevant, and who turn out to be working on problems that are important to me.

I work on computationally bounded statistical inference. Most theoretical paradigms don't have a clean way of handling computational constraints, and I think it's important to address this since the computationally complexity of exact statistical inference scales extremely rapidly with model complexity. I also have recently starting working on applications in program analysis, both because I think it provides a good source of computationally challenging problems, and because it seems like a domain that will force us into using models with high complexity.

Singularity considerations were a factor when choosing to work on AI, although I went into the field because AI seems like a robustly game-changing technology across a wide variety of scenarios, whether or not a singularity occurs. I certainly think that software safety is an important issue more broadly, and this partially influences my choice of problems, although I am more guided by the problems that seem technically important (and indeed, I think this is mostly the right strategy even if you care about safety to a fair degree).

Learning more about mainstream AI has greatly shaped my beliefs regarding AGI, so it's something that I would certainly recommend. Going to grad school shaped my beliefs even further, even though I had already read many AI papers prior to arriving at Stanford.

What does that question mean?

Like Mark, I'm not sure I was able to parse your question, can you please clarify?

I wouldn't presume to know what the field as a whole thinks, as I think views vary a lot from place to place and I've only spent serious time at a few universities. However, I can speculate based on the data I do have.

I think a sizable number (25%?) of AI graduate students I know are aware of LessWrong's existence. Also a sizeable (although probably smaller) number have read at least a few chapters of HPMOR; for the latter I'm mostly going off of demographics, I don't know that many who have told me they read HPMOR.

There is very little actual discussion of MIRI or LessWrong. From what I would gather most people silently disagree with MIRI, a few people probably silently agree. I would guess almost no one knows what MIRI is, although more would have heard of the Singularity Institute (but might confuse it with Singularity University). People do occasionally wonder whether we're going to end up killing everyone, although not for too long.

To address your comment in the grandchild, I certainly don't speak for Norvig but I would guess that "Norvig takes these [MIRI] ideas seriously" is probably false. He does talk at the Singularity Summit, but the tone when I attended his talk sounded more like "Hey you guys just said a bunch of stuff, based on what people in AI actually do, here's the parts that seem true and here's the part that seem false." It's also important to note that the notion of the singularity is much more widespread as a concept than MIRI in particular. "Norvig takes the singularity seriously" seems much more likely to be true to me, though again, I'm far from being in a position to make informed statements about his views.

Along with the views of EY (which such readers would already know) I present the singularity views of Robin Hanson and Ray Kurzweil, and discuss the intelligence enhancing potential of brain training, smart drugs, and eugenics. My thesis is that there are so many possible paths to super-human intelligence and such incredible military and economic benefits to develop super-human intelligence that unless we destroy our high-tech civilization we will almost certainly develop it.

Hard to say about the time because I worked on both books while also doing other projects. I suspect I could have done the Singularity book in about 1.5 years of full time effort. I don't have a good estimate for the textbook. Alas, I have lost money on the singularity book because the advance wasn't all that big, and I had personal expenses such as hiring a research assistant and paying a publicist. The textbook had a decent advance, still I probably earned roughly minimum wage for it. Surprisingly, I've done fairly well with my first book, Game Theory at Work, in part because of translation rights. With Game Theory at Work I've probably earned several times the minimum wage. Of course, I'm a professor and part of my salary from my college is to write, and I'm not including this.

I wanted to write a free market microeconomics textbook, and there are very few of these. I was recruited to write the textbook by the people who published Game Theory at Work. Had the textbook done very well, I could have made a huge amount of money (roughly equal to my salary as a professor) indefinitely. Alas, this didn't happen but the odds of it happening were well under 50%. Since teaching microeconomics is a big part of my job as a college professor, there was a large overlap between writing the textbook and becoming a better teacher. My textbook publisher sent all of my chapters to other teachers of microeconomics to get their feedback, and so I basically got a vast amount of feedback from experts on how I teach microeconomics.

Not really. Someone (I forgot who) wrote that I helped them see the race to create AI as a potential existential risk. I promoted the book on numerous radio shows and I hope I convinced at least a few people to do further research and perhaps donate money to MIRI, but this is just a hope.

No. I ran as a Republican in one of the most Democratic districts in Massachusetts, my opponent was the second most powerful person in the Massachusetts State Senate, and even Republicans in my district had a high opinion of him.

Greater economic growth means more money for AI research from companies and governments and if you think that AI will probably go wrong then this is a source of trouble. But there are benefits as well including increased charitable contributions for organizations that reduce existential risk and better educational systems in India and China which might produce people who end up helping MIRI. Overall, I'm not sure how this nets out.

Catch up growth is not necessarily safe because it will increase the demand for products that use AI and so increase the amount of resources companies such as Google devote to AI.

The only safe path is someone developing a mathematically sound theory of friendly AI, but this will be easier if we get (probably via China) intelligence enhancement with eugenics.

I am not up on my philosophical literature (trying to change this), but I think most analytic philosophers have heard of Pearl et al. by now. Not every analytic philosopher is as mathematically sophisticated as e.g. people at the CMU department. But I think that's ok!

I don't think it's a wise social move for LW to beat on philosophers.

Which academic disciplines care about causality? (I'm guessing statistics, CS, philosophy... anything else?)

On some level any empirical science cares, because the empirical sciences all care about cause-effect relationships. In practice, the 'penetration rate' is path-dependent (that is, depends on the history of the field, personalities involved, etc.)

To add to your list, there are people in public health (epidemiology, biostatistics), social science, psychology, political science, economics/econometrics, computational bio/omics that care quite a bit. Very few philosophers (excepting the CMU gang, and a few other places) think about causal inference at the level of detail a statistician would. CS/ML do not care very much (even though Pearl is CS).

Is there anything like a mainstream agreement on how to model/establish causality? E.g. does more or less everyone agree that Pearl's book, which I haven't read, is the right approach? If not, is it possible to list the main competing approaches?

I think there is as much agreement as there can reasonably be for a concept such as causality (that is, a philosophically laden concept that's fun to argue about). People model it in lots of ways, I will try to give a rough taxonomy, and will tell you where Pearl lies

Interventionist vs non-interventionist

Most modern causal inference folks are interventionists (including Pearl, Rubin, Robins, etc.). The 'Nicene crede' for interventionists is: (a) an intervention (forced assignment) is key for representing cause/effect, (b) interventions and conditioning are not the same thing, (c) you express interventions in terms of ordinary probabilities using the g-formula/truncated factorization/manipulated distribution (different names for the same thing). The concept of an intervention is old (goes back to Neyman (1920s), I think, possibly even earlier).

To me, non-interventionists fall into three categories: 'naive,' 'abstract', and 'indifferent.' Naive non-interventionists are not using interventions because they haven't thought about things hard enough, and will thus get things wrong. Some EDT folks are in this category. People who ask 'but why can't we just use conditional probabilities' are often in this set. Abstract non-interventionists are not using interventions because they have in mind some formalism that has interventions as a special case, and they have no particular need for the special case. I think David Lewis was in this camp. Joe Halpern might be in this set, I will ask him sometime. Indifferent non-interventionists operate in a field where there is little difference between conditioning and interventions (due to lack of interesting confounding), so there is no need to model interventions explicitly. Reinforcement learning people, and people who only work with RCT data are in this set.

Counterfactualists vs non-counterfactualists

Most modern causal inference folks are counterfactualist (including Pearl, Rubin, Robins, etc.). To a counterfactualist it is important to think about a hypothetical outcome under a hypothetical intervention. Obviously all counterfactualists are interventionist. A noted non-counterfactualist interventionist is Phil Dawid. Counterfactuals are also due to Neyman, but were revived and extended by Rubin in the 70s.

Graphical vs non-graphical

Whether you like using graphs or not. Modern causal inference is split on this point. Folks in the Rubin camp do not like graphs (for reasons that are not entirely clear -- what I heard is they find them distracting from important statistical modeling issues (??)). Folks in the Pearl/SGS/Robins/Dawid/etc. camp like graphs. You don't have to have a particular commitment to any earlier point to have an opinion on graphs (indeed lots of graphical models are not about causality at all). In the context of causality, graphs were first used by Sewall Wright for pedigree analysis (1920s). Lauritzen, Pearl, etc. gave a modern synthesis of graphical models. Spirtes/Glymour/Scheines and Pearl revived a causal interpretation of graphs in the 90s.

"Popperians" vs "non-Popperians"

Whether you restrict yourself to testable assumptions. Pearl is non-Popperian, his models make assumptions that can only be tested via a time machine or an Everett branch jumping algorithm. Rubin is also non-Popperian because of "principal stratification." People that do "mediation analysis" are generally non-Popperian. Dawid, Robins, and Richardson are Popperians -- they try to stick to testable assumptions only. I think even for Popperians, some of their assumptions must be untestable (but I think this is probably necessary for statistical inference in general). I think Dawid might claim all counterfactualists are non-Popperian in some sense.

I am "a graphical non-Popperian counterfactualist" (and thus interventionist).

Does there exist a reasonably neutral high-level summary of the field?

We are working on it.

There are certainly areas of mathematics where it seems like there is an underlying causality structure (frequently orthogonal or even inverse to the proof structure)

Can you give a simple example of/pointer to what you mean?

I think that the estimates cannot be undertaken independently. FAI and UFAI would each pre-empt the other. So I'll rephrase a little.

I estimate the chances that some AGI (in the sense of "roughly human-level AI") will be built within the next 100 years as 85%, which is shorthand for "very high, but I know that probability estimates near 100% are often overconfident; and something unexpected can come up."

And "100 years" here is shorthand for "as far off as we can make reasonable estimates/guesses about the future of humanity"; perhaps "50 years" should be used instead.

Conditional on some AGI being built, I estimate the chances that it will be unfriendly as 80%, which is shorthand for "by default it will be unfriendly, but people are working on avoiding that and they have some small chance of succeeding; or there might be some other unexpected reason that it will turn out friendly."

what probability estimate would be low enough for you to drop the AI business from your consideration?

Even a fairly low probability estimate would justify effort on an existential risk.

And I have to admit, a secondary, personal, reason for being involved is that the topic is fascinating and there are smart people here, though that of course does not shift the estimates of risk and of the possibilities of mitigating it.

Median estimate for when they'll start working on a serious code project (i.e., not just toy code to illustrate theorems) is 2017.

This will not necessarily be development of friendly AI -- maybe a component of friendly AI, maybe something else. (I have no strong estimates for what that other thing would be, but just as an example--a simulated-world sandbox).

Everything I say above (and elsewhere), is my opinion, not MIRIs. Median estimate for when they'll start working on friendly AI, if they get started with that before the Singularity, and if their direction doesn't shift away from their apparent current long-term plans to do so: 2025.

Sure, I'll send it to you. If anyone else wants it, please contact me. I always knew that Semitic Noun Patterns would be a best seller :-)

I don't think I have anything to say that hasn't been said better by others in MIRI and FHI, but I think that AI boxing is impossible because (1) it can convince any gatekeepers to let it out and (2) any AI is "embodied" and not separate from the outside world if only in that its circuits pass electrons, and (3) I doubt you could convince all AGI reseachers to keep their projects isolated.

Still, I think that AI boxing could be a good stopgap measure, one of a number of techniques that are ultimately ineffectual, but could still be used to slightly hold back the danger.

It's still functioning to some extent-- I'll be at Arisia next weekend. As far as I can tell, I'm neglecting the website because of depression and inertia.

I'm not-quite completely blind; what little vision I have tends to fluctuate between effectively nonexistent and good enough to notice vague details maybe once or twice a year. I could see better up until I was 14, but my vision was still too poor to get out of using braille and a cane (given thick glasses and enough time, I could possibly have read size 20 font; even with the much larger font used in movie subtitles, I had to pause the video and put my face against the screen to read them).

I don't know my official acuity/diagnoses (It's been a few years since I saw an eye doctor), but I appear to have started out with retinal detachment and scarring, and later developed uveitis. The latter seems to be the primary cause for the dramatic decline starting from age 14.

1. Issues in my first two years of college interfered in my Math/Physics/Computer Science courses, and I never got back into those. So my skills in each has remained only that which I've used most (for example, I've made some games, but the required qualifications for most programming jobs I've come across exceed what I can do without additional training. I think that, even had I not dropped the ball on those, competing with sighted programmers/scientists/mathematicians would require a decent amount of exceptionalism and/or luck. Mathematic notation is also tricky in terms of accessibility; there exist codes such as Nemeth that make math in braille relatively powerful, but on the computer side of things, graphs and LeTX take some doing to use, which also makes trying to study anything with math online difficult (I once downloaded a web page and edited its source so I could read the equations).
2. It's hard to say. For roughly four years after my vision went from poor to useless, I think I was still fairly driven and ambitious (I did a lot of writing, half taught myself Japanese and Javascript, self-published a terrible science fiction novel, learned to use a music composition program whose accessibility was poor, improvised some crude techniques for making simple images, got into and graduated from the state Math and Science school, and was taking plenty of notes on numerous other things I was hoping to do sooner than later). It all went to hell when I got to college, and has gone back and fourth since, but I'm not sure if any of this compares favorable/unfavorably to the average person. There may be some contributing factors to the negative aspects that go back to my vision (I can't safely get up and go running, or do the all-important eye-contact thing, as examples), but I don't think the affect in the ambition/motivation area has been majorly significant.
3. I'm not sure what you mean, specifically? My exposure to CFAR consists primarily of LesssWrong; I've been attempting to apply LW-style rationality to the situation, but the timing has made this difficult (I found LW a few months after returning home from college, at which point my options in general were reduced to "things I can do over the internet" and "things for which I would need to go through my parents", and have mostly stayed there until half a week ago. I've not been able to avoid antisocial death spirals; I find myself wanting to deal with the family members I live with less, which makes dealing with them more annoying, repeat ad hermitdom.) If the results of last week's meeting go as planned, I should have a better answer by the end of February.

Can you talk about your specific field in linguistics/philology?

I've mucked about here and there including in language classification (did those two extinct tribes speak related languages?), stemmatics (what is the relationship between all those manuscripts containing the same text?), non-traditional authorship attribution (who wrote this crap anyway?) and phonology (how and why do the sounds of a word "change" when it is inflected?). To preserve some anonymity (though I am not famous) I'd rather not get too specific.

what are the main challenges?

There are lots of little problems I'm interested in for their own sake but perhaps the meta-problems are of more interest here. Those would include getting people to accept that we can actually solve problems and that we should try our best to do so, Many scholars seem to have this fatalistic view of the humanities as doomed to walk in circles and never really settle anything. And for good reason - if someone manages to establish "p" then all the nice speculation based on assuming "not p" is worthless. But many would prefer to be as free as possible to speculate about as much as possible.

Do you have a stake/an opinion in the debates about the Chomskian strain in syntax/linguistics in general?

Yes. I think the Chomskyan approach is based on a fundamentally mistaken view of cognition, akin to "good old fashioned artificial intelligence". I hope to write a top-level post on this at some point. But I'll say this for Chomsky: He's not a walk-around-in-circles obscurantist. He's a resolutely-march-ahead kind of guy. A lot of the marching was in the wrong direction, but still, I respect that.

Well, it is sometimes difficult to be me, but I'm not sure how much of that is caused by being smart, how much by lack of some skills, and how much is simply the standard difficulty of human life. :D

Seems to me that most people around me don't care about truth or rationality. Usually they just don't comment on things outside of their kitchens; unless they are parrotting some opinion from a newspaper or a TV. That's actualy the less annoying part; I am not disappointed because I didn't expect more from them. More annoying are people who try to appear smart and do so basicly by optimizing for signalling: they repeat every conspiracy theory, share on facebook every "amazing" story without bothering to google for hoaxes or just use some basic common sense. When I am at Mensa and listen to people discussing some latest conspiracy theory, I feel like I might strangle them. Especially when they start throwing around some fully general arguments, such as: You can't actualy know anything. They use their intelligence to defeat themselves. Also, I hate religion. That's a poison of the mind; an emotional electric fence in a mind that otherwise might have a chance to become sane. -- But I suspect all countries are like this, in general. And I am lucky to live in one where people won't try to hurt me just because I say something blasphemous. Still, as is obvious from this paragraph, I feel greatly frustrated about the sanity waterline here.

Okay, specifically for Slovakia: This country used to be mostly Catholic, then it was Communist for a few decades, now it's going back to catholicism again. During the communism, the Catholics were pretty successful in recruiting many contrarians to their ranks; they pretty much told them that the search for truth is the search for God, and they associated atheism with communism (which wasn't difficult at all, since Communists used it as an applause light). I was frustrated by seeing people around me look for the truth in the supernatural, and dismissing the reality almost as a propaganda. Then there was a higher level of contrarians who dismissed also the local religion, and instead embraced buddhism or whatever. Believing in "mere reality" does not work as a signal for intelligence here.

I actually don't have a good way for dealing with it. Some time I was alone. Some time I was friendly with religious people, politely participating in their rituals, believing none of that, but enjoying the company of smart contrarians. Once or twice I tried to find some reason in Mensa, always horribly disappointed.

As a child, I was a member of the mathematical club; elementary-school students who loved math and did the mathematical olympiad. That was the best part of my life; smart activities, and no bullshit. But as we grew older, the club dissolved. -- Skip almost two frustrating decades and I found LessWrong. And I was like: "Smart and sane people again!" and "Oh shit, why do they have to be on the other side of the planet?" And since then I am trying to build a local rationalist movement, progressing very very slowly.

One thing that keeps me sane is my current girlfriend, who also reads LessWrong, and attended a CFAR minicamp with me. But she is not as enthusiastic about it as I am; and she seems to prefer good relationships with other people to being right. Maybe I am just a horrible person unable to deal with people, but the thing is I am unable to unsee the bullshit; when someone speaks bullshit, it's like a painful shrieking sound in my ears, I just can't ignore it; I can keep quiet but it still feels unpleasant.

I suspect most rational people around me cope by focusing their energy into their favorite project, and ignoring the insanity of the rest of the world. (But I may be wrong at modelling other people.) They probably can be rational in their work, and social in the rest of their lives. Maybe they are happy like that. Maybe they just don't know they could expect more (if I didn't have the unique experience of the mathematical club and of LW, probably neither would I). So this year I try to make a list of smart and sane people around me, get in contact with each of them, invite them to a local LW meetup, and give them a copy of my translation of Sequences. -- I am not sure how much should I push the LW; whether having a club of smart and sane people couldn't be enough. For me, LW is simply one level more meta: before LW I approximately knew what was and what wasn't rational, but I didn't have any arguments to win a debate. It was like a matter of feeling: this seems like a correct way to approach truth, and this feels like a way to madness. I just had the general idea that the reality is out there, and that the proper way to grasp it is to adjust my map to the territory, not the other way round. (Because that's what worked for me in mathematics.) -- Maybe my role here is to join the local smart people together. But maybe I am just projecting my desires onto them, and they are actually quite happy as they are now. This will be resolved experimentally.

I was born here, and I never lived anywhere else (longer than two weeks). I dislike travelling, and I feel uncomfortable speaking another language (it has a cognitive cost, so I feel I sound more stupid than I would in my language). Generally, I dislike changes -- I should probably work on that, but this is where I am now.

I could also provide some rationalization... uhh, I have friends here, I am familiar with how the society works here, maybe I prefer being a fish in a smaller pond -- okay the last one is probably honest, too.

I think the main reason is that basically nobody in the wider world talks about LW, and RW is the only place that talks about LW even that much. And RW can't reasonably be called very interested in LW either (though many RW regulars find LW annoying when it comes to their attention). Also, we use the word "rational", which LW thinks of as its own - I think that's a big factor.

From my own perspective: RW has many problems. The name is a historical accident (and SkepticWiki.com/org is in the hands of a domainer). Mostly it hasn't enough people who can actually write. It's literally not run by anyone (same way Wikipedia isn't), so is not going to be fixed other than organically. Its good stuff is excellent and informative, but a lot of it isn't quite fit for referring outside fresh readers to.

It surprises me how popular it is (as in, I keep tripping over people using a particular page they like - Alexa 21,000 worldwide, 8800 US - and Snopes uses us a bit) - it turns out there's demand for something that can set out "no, actually, that's BS and here's why, point for point". Raising the sanity waterline does in fact also involve dredging the swamps and cleaning up toxic waste spills. Every time we have a fundraiser it finishes ridiculously quickly ('cos our expenses are literally a couple thousand dollars a year). We have readers who just love us.

On balance, though, I do think RW makes the world a better place rather than a worse one. (Or, of course, I wouldn't bother.)

FWIW, there's a current active discussion on What RW Is For, which I expect not to go anywhere much.

I'm not sure I could reasonably steelman LW opposition to RW as if either were a monolith and there were no crossover (which simply isn't the case). I will note that RW is piss-insignificant, and if you're spending any time whatsoever worrying what RW thinks of LW then you're wasting precious seconds.

(The discussion of RW on LW actually came up on the LW and RW Facebook groups this morning too.)

Because RW sucks at actually being rational. Rather they seem to have confused being "rational" with supporting whatever they perceive to be the official scientific position. Whereas LW has a number of contrarian positions, most notably cryonics and the Singularity, where it is widely believed the mainstream position is likely wrong and their argument for it is just silly.

My answer may be a bit generic :-)

Re: Wikipedia - This is pretty well-trodden ground, in terms of (a) people coming up with explanations (b) having little evidence as to which of them hold. There's all manner of obvious systemic problems with Wikipedia (maybe the easy stuff's been written, the community is frequently toxic, the community is particularly harsh to newbies, etc) but the odd thing is that the decline in editing observed since 2007 has also held for wikis that are much younger than English Wikipedia - which suggests an outside effect. We're hoping the Visual Editor helps, once it works well enough (at present it's at about the stage of quality I'd have expected; I can assure you that everyone involved fully understands that the Google+-like attempt to push everyone into using it was an utter disaster on almost every level). The Wikimedia Foundation is seriously interested in getting people involved, insofar as it can make that happen.

As for LessWrong ... it's interesting reading through every post on the site (not just the Sequences) from the beginning in chronological order - because then you get the comments. You can see some of the effect you describe. Basically, no-one had read the whole thing yet, 'cos it was just being written.

I'm not sure it was easier for beginners at all. Remember there was only "main" for the longest time - and it was very scary to write for (and still is). Right now you can write stuff in discussion, or in various open threads in discussion.

In general online commenting is something I do out of habit. Higher return on time than completely passive media consumption such as watching TV but not that I book under time spent with maximum returns.

I generally think that a shift to massive information consumption of content via TV/radio in the 20st century was something that's bad for the general discourse of ideas in society. Active engagment helps learning.

I also prefer it over chatting in venues such as IRC, because it provides it provides deeper engagement with ideas and leaves more of a footprint. Created content is findable afterwards.

Lesswrong is also a choice to keep me intellecutally grounded. These days I do spent plenty of time thinking in mental frameworks that are not based on reductionist materalism. I do see value of being pretty flexible about changing the map I use to navigate the world and I don't want to lose access to the intellectual way of thinking.

In total I however spent more time than optimal on LW and frequently use it to procrastinate on some other task.

Yes

Wandering Son (Hōrō Musuko)

Personal reasons: the story's relevant to my own and in a genre I don't normally pay much attention to, which might be why it stands out over other possible candidates (e.g. Puella Magi Madoka☆Magica). Also, by choosing an artsy show that tackles a serious dramatic subject, full of tragedy (and qbrfa'g erfbyir rirelguvat arngyl at the end), I sound more intellectual.

Psuedo-objective reasons: I feel it accurately captures the feelings of childhood and growing up. I particularly liked the portrayal of the sibling relationship, where you hate each other on a level that's superficial but no less genuine for that, but will stand by each other when you discover things the other really cares about. The conclusion also felt very true-to-life. I liked the visual style; the character designs are much more realistic than the animé norm (and for viewers who find it hard to tell them apart, serve as a demonstration of the valid reasons for the animé norm), and the whole setting and story feels like something you could do in live action. But at the same time this would be completely impossible to produce in live action, for a different reason than normal (child actors and ethical issues), so it shows off the ability of animé to do what other media can't. The slightly washed-out, watercolour visual style is distinctive, even among animé - but it's like that for a reason, the uncertain, blurry visuals aligning perfectly with the emotions this series is trying to convey. Likewise the light, childish-sounding soundtrack is distinctive - but it's not just style for the sake of style, it fits with the show as a whole.

Practical notes: I prefer the 11-episode (rather than 12-episode) release. I've avoided describing the premise because it's an episode 1 spoiler; if you think you'd like the show from this description I recommend watching it (or at least watching episode 1) rather than seeking out more information.

Do you think any part of what MIRI does is at all useful?

It now seems like a somewhat valuable research organisation / think tank. Valuable because they now seem to output technical research that is receiving attention outside of this community. I also expect that they will force certain people to rethink their work in a positive way and raise awareness of existential risks. But there are enough caveats that I am not confident about this assessment (see below).

I never disagreed with the basic idea that research related to existential risk is underfunded. The issue is that MIRI's position is extreme.

Consider the following fictive and actual positions people take with respect to AI risks in ascending order of perceived importance:

1. Someone should actively think about the issue in their spare time.

2. It wouldn’t be a waste of money if someone was paid to think about the issue.

3. It would be good to have a periodic conference to evaluate the issue and reassess the risk every year.

4. There should be a study group whose sole purpose is to think about the issue. All relevant researchers should be made aware of the issue.

5. Relevant researchers should be actively cautious and think about the issue.

6. There should be an academic task force that actively tries to tackle the issue.

7. It should be actively tried to raise money to finance an academic task force to solve the issue.

8. The general public should be made aware of the issue to gain public support.

9. The issue is of utmost importance. Everyone should consider to contribute money to a group trying to solve the issue.

10. Relevant researchers that continue to work in their field, irrespective of any warnings, are actively endangering humanity.

11. This is crunch time. This is crunch time for the entire human species. And it’s crunch time not just for us, it’s crunch time for the intergalactic civilization whose existence depends on us. Everyone should contribute all but their minimal living expenses in support of the issue.

Personally, most of the time, I alternate between position 3 and 4.

Some people associated with MIRI take positions that are even more extreme than position 11 and go as far as banning the discussion of outlandish thought experiments related to AI. I believe that to be crazy.

Extensive and baseless fear-mongering might very well cause MIRI's value to be overall negative.

How should I fight a basilisk?

Every basilisk is different. My current personal basilisk pertains measuring my blood pressure. I have recently been hospitalized as a result of dangerously high blood pressure (220 systolic, mmHg / 120 diastolic, mmHg). Since I left the hospital I am advised to measure my blood pressure.

The problem I have is that measuring causes panic about the expected result, which increases the blood pressure. Then if the result turns out to be very high, as expected, the panic increases and the next measurement turns out even higher.

Should I stop measuring my blood pressure because the knowledge hurts me or should I measure anyway because knowing it means that I know when it reaches a dangerous level and thus requires me to visit the hospital?

Yes, but incompletely. I'll track things precisely until a habit is established, at which point I stop tracking everything and check-in every once in a while to make sure I'm still on track. Some things I keep track of consistently, such as my budget, weight lifting numbers, bodyweight, etc.

The process is different for different things. I usually start with a Google Drive spreadsheet, and then experiment with other more specific apps if they're better than spreadsheets (they rarely are). If you have any more specific questions, I'd be glad to answer them.

What's your motivation for veganism?

Moral reasons. All else equal, I think that inflicting pain or death is bad, and that the ability to feel pain and the desire to not die is very widespread. I also think that the intensity of pain in simpler animals is still very strong (I think humans did not evolve large brains because otherwise the pain was not strong enough). I also think that our ability to manage pain slighly reduces the impact of our having the ability to suffer more strongly and with more variety. But I give, for sanity check reasons, priority to the desires of "more complex" animals, like humans.

Due to our technical ability we can now produce supplements for micronutrient which are missing or insufficently available in plants[1], and so I see health concerns resolved. So all the pain and death that I would inflict would only be there for greated enjoyment of food. Although I love the taste of meat and animal products, the comparative enjoyment is not big enough that I would kill for it. That I can enjoy plant-based foods is partly based upon my not being afraid of using my kitchen, and having a good vegan/vegetarian self-service restaurant 100m from my apartment.

And than there are the environmental reasons, and the antibiotic use, etc. etc. They count, and might be even sufficent on their own, but I'll only investigate those in case my other concerns/reasons were invalidated.

[1] There is vegan vit B12, vit D3, EPA/DHA (omega3), and creatin powder.

What do you enjoy most in software development, and why are you going to be looking for a job again soon? What's your dream SW dev job?

Cannot really answer what I enjoy most; I like almost every job that comes up, with only a few exceptions. I hate repeating myself, and I hate having to do things in a ... ... ... way against my better judgement. I prefer to work more time (as in effort and calender time) doing the architecture/design/coding parts, but I also prefer doing other stuff once in a while more than being purely a lonely coder.

I will give my notice in a few hours, so I'll than search for a new job. I will have two months time for that, though, and maybe I take some time off before starting in a new company. I'll end this job because neither one of money, project nor team is good enough to make me happy, and the job market for software developers allows for searching for improved conditions.

My dream SW job would involve writing open source software which somehow tangibly improves the lives of some people (think better medical DAq and analysis instead of the newest photo sharing app), working with a team where competence and respect is wide-spread, as is friendlyness, and pay which is not worse than I what I got when I was still failing to drop out of college. Sadly, I do not think such a job exists, especially not for people like me (who do not have the necessary skills for anything fancy).

Thanks!

This was surprisingly simple: I got myself to want to run, started running, and patted myself on the back everytime I did it.

The want part was a bit of luck: I always thought I "should" do some sports, for physical and more importantly mental health reasons, and think that being able to do stuff is better than not being able, ceteris paribus. So I was thinking what kind of activity I might prefer.

I like my alone time (so team- or pair-sports are out), I dislike spending money when I expect it to be wasted (like Gym memberships, bikes, et al.). And I feel easily embarassed and ashamed, and like to get myself at least somewhat up to speed on my own.

Running fits those side requirements. Out of chance I got hold of "Born to Run", and even after the first quarter of the book I thought that it would be great if I could just go out on a bad day and spend an hour free of shit, or how it would be great that I could just reach some location a few kilometers away without any prep or machines or services.

I then decided that I will start running, and that my primary goal shall be that I like it and be able to do it even in old age if such would happen. With the '*' that I give myself an easy way out in case of physical pain or unexpected hatred against the activity, but not for any weasel reasons.

I didn't start running for another one and a half years, because Schweinehund, subtype Innerer. When my mood was getting slightly better (I was again able to do productive work), I started, with the "habit formation" mind-set. Also didn't tell anyone in the beginning. I think it helped that I already had some knowledge on how to train and run correctly, which especially in the beginning meant that I always felt like I could run further than I was "allowed" to.

And for good feedback: However it went, when I finished my training, I "said" to myself: I did good. I feel good. I feel better than before I started. I wrote every single run down on RunKeeper and Fitocracy, and always smiled at the "I'm awesome!" button of the latter one. I'm also quite sure that having at least one new personal best once a week helped. (Also, when you run barefoot, you get the "crazy badass" card for free, however slow you run. I like this.)

Once started, such a feedback loop is quite powerful. When I once barely trained for month, I was also surprised that getting back into regular running after that down-phase was so much easier. Now, after only seven months of training, I went from doing walk/run for 15 minutes to running 75 minutes, and having no problem with a cold-start 6% incline for the first two kilometers. I'm proud. Feels good (is quite new to me).

I'm not sure, but one of the techniques that seems most salient to me is breadth-first search. Partly this is to hold off on proposing solutions. Take just a little bit longer to look at the problem and gather data before generating hypotheses. The second part is to find cheap tests to disprove your hypotheses instead of going farther down the path that an early hypothesis leads. Folks who use depth-first search, building up a large tree of hypotheses first or going down a long path of possible tests and fixes, seem more likely to get stuck.

I also really like troubleshooting out loud with colleagues who aren't afraid to contradict each other. Generating lots of hypotheses and quickly disconfirming most of them can quickly narrow down on the problem. "Okay, maybe the cause is a bad data push. But if that were so, it would be on all the servers, not just the ones in New York, because the data push logs say the push succeeded everywhere. But the problem's just in New York. So it's not the data push."