Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: capybaralet 07 April 2016 05:52:00PM 4 points [-]

Comparing with articles from a year ago, e.g. http://www.popsci.com/bill-gates-fears-ai-ai-researchers-know-better, this represents significant progress.

I'm a PhD student in Yoshua's lab. I've spoken with him about this issue several times, and he has moved on this issue, as have Yann and Andrew. From my perspective following this issue, there was tremendous progress in the ML community's attitude towards Xrisk.

I'm quite optimistic that such progress with continue, although pessimistic that it will be fast enough and that the ML community's attitude will be anything like sufficient for a positive outcome.

Comment author: satt 16 January 2017 02:08:30AM 0 points [-]

I'm a PhD student in Yoshua's lab. I've spoken with him about this issue several times, and he has moved on this issue,

Thank you!

Comment author: ChristianKl 10 January 2017 09:44:28AM 1 point [-]

For instance, he says that a teacher in the top quartile increases their class' performance "by over 10 percent in a single year". I can believe that, but maybe all it means is that a top-quartile teacher increases their class' scores by 11% a year, while a bottom-quartile teacher increases their class' scores by 9% a year. That would hardly refute the idea that teacher effects are small on average!

I understand him to be speaking about them increasing 10% more than non-top quartile teachers.

Eg. enough to circumvent the US-Asia difference in two years and also enough to circumvent the Black-White difference in four years as suggested in the answer to the Stackexchange question.

It's worth noting that Medical Error is the third leading cause of death in the US http://www.bmj.com/content/353/bmj.i2139

It'd be immediately possible in the US: eliminate de facto immigration barriers for foreign doctors.

The article doesn't only describe immigration barriers but also barriers of credentialism. The non-US degree often isn't enough to work in the US as a doctor because it's quality is in doubt. If there would be hard evidence that those doctors perform as well as US doctors the ability to seek rents via credentialism will be reduced.

Maybe. I did a bit more searching on Google Scholar but didn't uncover a more recent review article with relevant statistics. (I did find one study of a thousand students in HSGI schools in "school year 2013-2014", which found that 12% of the variance in GPA was at the teacher level.)

I looked at that study. It seems their best predictors were e (i) Fall algebra EOC (End Of Course) scores, (ii) English language learner (ELL) status, (iii) Black student status, and (iv) Hispanic student status.

Of course you do a better prediction of the student performance for a standardized test when you look at the last standardized test they took than when you look at whether or not they had a very good teacher for a single school year. The 12% variance under that setting might be compatible with the claims that Gates makes. 12% variance per year might compound over multiple years to bridge the gap between the US and Asia in two years and US Black White gap in 4 years.

Comment author: satt 16 January 2017 02:04:45AM 0 points [-]

I understand him to be speaking about them increasing 10% more than non-top quartile teachers.

OK, thanks for clarifying. That sounds like a more impressive effect. At the same time, it's probably still consistent with teacher quality explaining only 10% of the variance in student performance.

I'll do back-of-envelope arithmetic to demonstrate. The median top-quartile teacher is at the 88th percentile. The median non-top quartile teacher is at the 38th. Suppose, just to allow me to arrive at concrete numbers, teacher quality has a normal distribution. Then the median top-quartile teacher is 1.48 standard deviations better than the median non-top quartile teacher. Now, an R^2 of 10% implies a correlation of sqrt(10%) = 0.23 between teacher quality and pupil performance, so the difference in pupil performance between the median non-top quartile teacher and the median top-quartile teacher is 1.48 * 0.23 = 0.34 standard deviations. That's a statistically detectable effect, and one that could well translate into 10% higher test scores after a year with the better teachers.

Eg. enough to circumvent the US-Asia difference in two years and also enough to circumvent the Black-White difference in four years as suggested in the answer to the Stackexchange question.

Plausible. If I remember correctly the black/white difference is about 1 standard deviation, so if my estimated effect size of 0.34 SD for good vs. less good teachers is accurate and can be built on year by year, it's enough to close the black/white difference in 3 years. I don't know the US-Asia difference but probably the same kind of logic applies.

It's worth noting that Medical Error is the third leading cause of death in the US http://www.bmj.com/content/353/bmj.i2139

Agreed, medical error is a real & substantial issue. I am just dubious about the ability of some proposals to inexpensively reduce fatal medical error. (But I am optimistic about others. Checklists seem promising.)

The article doesn't only describe immigration barriers but also barriers of credentialism.

The way I would put it is that the credentialism barriers are the immigration barriers. AFAIK the explicit immigration barriers for foreign doctors looking to enter the US and practice in the US aren't the bottleneck; the requirement that the doctor do a US residence programme, or a degree from a US school, is a much stronger de facto bar to immigrating.

I agree with your last paragraph.

Comment author: luminosity 08 January 2017 05:04:07AM 3 points [-]

Dinner meetups

We have a monthly meetup at an RSL in Sydney -- we've found this is a good venue, as people who want to can get food and drinks, but not everyone needs to. At a cafe there is more pressure to have to make purchases over time if you're taking up their floor space.

This was the first meetup we ran when the Less Wrong Sydney group resumed meeting -- at first we ran them with a focus on group discussions, and talking about a particular topic, but we found that often a topic was more of a reason for people to stay away, than for people to come. With the advent of the dojos, there wasn't as much pressure for this to deliver on applied rationality, and so it has turned into more of a group discussion space.

We're currently in the middle of an experiment of trying to drive people to these meetups from a meetup.com group -- if people seem interested, we can then invite them to come along to the dojos also.

Comment author: satt 08 January 2017 09:26:58PM 1 point [-]

an RSL in Sydney

To save others from Googling, the RSL appears to be Australia's Returned and Services League, which licenses clubs as meeting venues, originally for veterans but now also for guests.

Comment author: ChristianKl 30 December 2016 08:32:17PM 0 points [-]

Education, like healthcare, is very expensive, mostly carried out for laypeople by trained specialists, and is generally considered (excepting people promoting their own one-size-fits-all solutions) a really knotty & complex thing to do

Let's look at schools. Having a masters degree in teaching doesn't result in the teachers students getting higher grades on standardized tests. Unions still press schools to pay people who have a masters degree in education more money and mostly succeed at opposing pay-for-performance.

Education is a good example of a field where teachers are payed for useless training instead of being payed for producing outcomes for students.

Bill Gates suggest in his TED talk:

So, how do you make education better? Now, our foundation, for the last nine years, has invested in this. There's many people working on it. We've worked on small schools, we've funded scholarships, we've done things in libraries. A lot of these things had a good effect. But the more we looked at it, the more we realized that having great teachers was the very key thing. And we hooked up with some people studying how much variation is there between teachers, between, say, the top quartile -- the very best -- and the bottom quartile. How much variation is there within a school or between schools? And the answer is that these variations are absolutely unbelievable. A top quartile teacher will increase the performance of their class -- based on test scores -- by over 10 percent in a single year. What does that mean? That means that if the entire U.S., for two years, had top quartile teachers, the entire difference between us and Asia would go away.

Given that citing a TED talk as a response to a scientific paper is a bit bad form, I added a question on skeptics to verify the claim.

I think part of the problem of the cited study is likely that it's done on data from before no-child-left-behind and the efforts of the Gates Foundation. It's also simply possible to find groups of teachers where there's little variance in teaching skill, that doesn't mean that differences don't exist on a larger scale.

Thought experiment: suppose every doctor were replaced by identical computers all running the same treatment-recommending software. How much does the variance in patient outcomes decrease?

There are treatment that require high skill to administer like Brain surgery and treatments that require less skill like handing over a pill. For the high skill tasks I do think that variance in patient outcomes would decrease.

But even for simply taking a pill a doctor can spend five minutes to hand over the pill or he can spend an hour to talk through the issue with the patient and make sure that the patient has TAP's to actually take the pills according to the schedule.

Currently there's no economic reason to spend that hour. It can't be billed to the insurance company. Even if currently everybody spends five minutes for such a patient you would suddenly get variance if you would start to pay some doctors by the outcome instead of simply paying them per visit.

This matters because it means individual practitioners are going to have a hard time beating the EBM approach at estimating treatment effects, because the statistical win of assessing treatments at the finer-grained level of the practitioner is going to be more than cancelled out by the statistical loss of each practitioner having a smaller sample to refer to.

I don't think that large sample sizes are everything. I think you can learn a lot by looking very carefully at the details of single cases.

Additionally the person who's the best in city X at treating disease Y for class of patients Z might get more than 40 patients of class Z with Y because everybody wants to be treated by the best.

If all those expensively trained doctors achieve the same outcomes there's also a question of why we limit their supply as strongly as we are doing it at the present by forcing to them to have the expensive training. In that case the way to go would be to get people with cheaper training to compete in the market of the highly trained doctors. In the absence of performance tracking this won't be possible because the expensively trained doctors have more prestige.

Comment author: satt 07 January 2017 06:03:47PM *  0 points [-]

Education is a good example of a field where teachers are payed for useless training instead of being payed for producing outcomes for students.

That doesn't surprise me. Training and degrees are easily observable; outcomes (or, rather, how much of an outcome is attributable to each teacher) are not. It's harder to pay people based on something less observable.

Given that citing a TED talk as a response to a scientific paper is a bit bad form, I added a question on skeptics to verify the claim.

Thanks!

But I'm not sure it matters. What Bill Gates says in that quotation might well be consistent with what I wrote — it's hard to be sure because he's quite vague.

For instance, he says that a teacher in the top quartile increases their class' performance "by over 10 percent in a single year". I can believe that, but maybe all it means is that a top-quartile teacher increases their class' scores by 11% a year, while a bottom-quartile teacher increases their class' scores by 9% a year. That would hardly refute the idea that teacher effects are small on average!

I think part of the problem of the cited study is likely that it's done on data from before no-child-left-behind and the efforts of the Gates Foundation.

Maybe. I did a bit more searching on Google Scholar but didn't uncover a more recent review article with relevant statistics. (I did find one study of a thousand students in HSGI schools in "school year 2013-2014", which found that 12% of the variance in GPA was at the teacher level.)

However, while the study I cited uses only pre-2004 data, I don't see much reason to think a newer study would reveal a big increase in teacher-level variance in outcomes. The low proportion of variance attributable to teachers has been true, as far as I know, for as long as people have investigated it (at least 46 years). I'm doubtful that an act which demands high-stakes testing, improvements in average state-wide test scores, and state-wide standards for teachers has changed that, or that a charity has changed that.

It's also simply possible to find groups of teachers where there's little variance in teaching skill, that doesn't mean that differences don't exist on a larger scale.

It's possible, but I don't see evidence that the paper I cited has this flaw. Its new analysis was based on about 100 Tennessee schools included in Project STAR, and the earlier analyses it summarized (table 1) used "samples of poor or minority students", "nationally representative samples of students", and "a large sample of public school students in Texas".

There are treatment that require high skill to administer like Brain surgery and treatments that require less skill like handing over a pill. For the high skill tasks I do think that variance in patient outcomes would decrease.

Fair point — my thought experiment only addresses diagnosis and treatment recommendation, not treatment administration. I think there would be more doctor-level variation among the latter...although not much more in absolute terms.

But even for simply taking a pill a doctor can spend five minutes to hand over the pill or he can spend an hour to talk through the issue with the patient and make sure that the patient has TAP's to actually take the pills according to the schedule.

Currently there's no economic reason to spend that hour.

Probably true in most places.

Even if currently everybody spends five minutes for such a patient you would suddenly get variance if you would start to pay some doctors by the outcome instead of simply paying them per visit.

Agreed that you'd get more variance, but I suspect it wouldn't be much more (subject to this hypothetical's exact details).

I don't think that large sample sizes are everything. I think you can learn a lot by looking very carefully at the details of single cases.

This is certainly true. I've seen too many stories of people with rare genetic conditions and other diseases successfully working out aetiologies to think otherwise. But those are unusual cases where people tended to invest lots of effort into figuring their cases out. I don't see them as signs that we can improve the quality-to-cost ratio of healthcare in general by looking very carefully at the details of each case.

Additionally the person who's the best in city X at treating disease Y for class of patients Z might get more than 40 patients of class Z with Y because everybody wants to be treated by the best.

Sure. But the ratios in my example are more important than the exact numbers.

If all those expensively trained doctors achieve the same outcomes there's also a question of why we limit their supply as strongly as we are doing it at the present by forcing to them to have the expensive training. In that case the way to go would be to get people with cheaper training to compete in the market of the highly trained doctors.

A good question, and a good answer to the question!

In the absence of performance tracking this won't be possible because the expensively trained doctors have more prestige.

It'd be immediately possible in the US: eliminate de facto immigration barriers for foreign doctors. Those barriers are, I'd guess, lower in other developed countries (hence why doctors in the UK, Germany, Canada, etc. earn less than US doctors) but I expect doctors' salaries there could also be reduced a bit by further relaxing immigration restrictions for foreign doctors.

Another option is for patients to go to the cheaper doctors: medical tourism.

Comment author: Anders_H 05 January 2017 10:52:14PM *  3 points [-]

The rational choice depends on your utility function. Your utility function is unlikely to be linear with money. For example, if your utility function is log (X), then you will accept the first bet, be indifferent to the second bet, and reject the third bet. Any risk-averse utility function (i.e. any monotonically increasing function with negative second derivative) reaches a point where the agent stops playing the game.

A VNM-rational agent with a linear utility function over money will indeed always take this bet. From this, we can infer that linear utility functions do not represent the utility of humans.

(EDIT: The comments by Satt and AlexMennen are both correct, and I thank them for the corrections. I note that they do not affect the main point, which is that rational agents with standard utility functions over money will eventually stop playing this game)

Comment author: satt 07 January 2017 03:19:45PM 2 points [-]

For example, if your utility function is log (X), then you will accept the first bet

Not even that. You start with $1 (utility = 0) and can choose between

  1. walking away with $1 (utility = 0), and

  2. accepting a lottery with a 50% chance of leaving you with $0 (utility = −∞) and a 50% chance of having $3 (utility = log(3)).

The first bet's expected utility is then −∞, and you walk away with the $1.

Comment author: satt 30 December 2016 06:25:01PM *  1 point [-]

Why don't we pay doctors in the present system based on their skills? We can't measure their skills in the present paradigm, because we can't easily compare the outcomes of different doctors. Hard patients get send to doctors with good reputations and as a result every doctor has an excuse for getting bad outcomes. In the status quo he can just assert that his patients were hard.

That is one difficulty, but I expect a bigger and more fundamental difficulty is just that there's lots of random noise in how patients respond to medical treatments.

Thought experiment: suppose every doctor were replaced by identical computers all running the same treatment-recommending software. How much does the variance in patient outcomes decrease? My gut says not very much. If it's right, most variance isn't doctor-level, it's going to be higher-level (at the level of a disease or a hospital/clinic) or lower-level (patient-level).

To me the most obvious analogy is teaching. A standard finding in education research is that classroom/teacher-level variation is only a small part of the variation in educational outcomes. (Doing a quick Google...tables 1 & 5 of this highly-cited paper suggest it's typically ~ 10% of the variance.) Education, like healthcare, is very expensive, mostly carried out for laypeople by trained specialists, and is generally considered (excepting people promoting their own one-size-fits-all solutions) a really knotty & complex thing to do, so I take the analogy seriously.

This matters because it means individual practitioners are going to have a hard time beating the EBM approach at estimating treatment effects, because the statistical win of assessing treatments at the finer-grained level of the practitioner is going to be more than cancelled out by the statistical loss of each practitioner having a smaller sample to refer to.

Imagine going from a multi-centre study of 40 specialists treating 1,600 people, to each specialist knowing about only their 40 patients. Each specialist then has only 1/40th the information they would've had, and that's going to negate the slight gain of eliminating the effect of different specialists. (The specialists could of course tell each other about their results, but then one's basically back to the large-scale, expensive EBM-style approach, and the agile, startuppy USP is lost.)

Allowing for doctor-level effects in analysis of treatments could help things, but I predict it would be a small improvement, and an improvement produced by extending the EBM approach, rather than building a parallel track to it.

Comment author: Mac 29 December 2016 05:32:30PM 1 point [-]

We all basically know that complex systems are unpredictable...I'm interested in how others identify complex systems...

Following from your quotes above, you could focus your search on systems for which the accuracy of predictions has been poor.

We are able to personally develop heuristics for evaluating predictions and complex systems, but sharing them with others is really tough.

FYI: This is basically the subject of the book Blink by Malcolm Gladwell. It's by no means a rigorous examination, a little too anecdotal, but you might find it useful.

Comment author: satt 29 December 2016 08:45:16PM 0 points [-]

Following from your quotes above, you could focus your search on systems for which the accuracy of predictions has been poor.

Mmm, reading the post it seems like it's driving towards "unpredictable" as an operational definition of "complex". And, reflecting a bit, I reckon that's not too far from how people actually tend to use the idea of a "complex system".

Comment author: satt 29 December 2016 07:50:40PM *  4 points [-]

The title primed me to upvote the OP, because I think novel, potentially overlooked arguments against promising policies are valuable. However, reading the post from start to finish, I don't think the core critique is communicated clearly. It seems to go from

  1. the UBI making the "assumption that the state needs its citizens" "more wrong", to

  2. the UBI intensifying the divide between "contributors and noncontributors", to

  3. the UBI worsening, "in the long run", the difficulties of integrating "the economically unnecessary parts of the population into society", to, in the end

  4. the UBI triggering "unforeseen consequences", "a whole class of problems that arise out of the changed relationships between citizens, states and economies"

and these four critiques are not the same, and not interchangeable. Moreover, the point where the post ends up (critique 4) is not novel or different to those I've seen before. It's the generic warning of unintended consequences that gets levelled against every public policy proposal ever, basic income included.

While I'm here, I'm uneasy with the handling of the side points as well. Paragraph 2 conflates Saudi Arabia's population with its citizenry, and these aren't the same thing. Yes, that sounds like pedantry, and for a lot of countries it would be pedantry, but Saudi Arabia might be literally the worst big country for which to elide the population-citizenry distinction. And the treatment of existing UBI critiques in the last paragraph is unduly generous. The claim that a UBI would "be impossible to get rid of if it is a failure" is very strong, and the notion that a UBI is mutually exclusive of "gradual approaches" puzzles me, since a UBI could certainly be introduced gradually (whether by gradually increasing the UBI allowance from zero, or by introducing the "U" in "UBI" slowly by incrementally expanding the range of people included).

I haven't downvoted the post either, because I do like the idea of using Discussion as a forum for polishing half-baked arguments about topics popular on LW, and don't (at the moment) want to discourage that activity.

Comment author: siIver 18 December 2016 12:10:30PM 5 points [-]

I would like downvotes and upvotes to be both shown rather than mathed out against each other, and also them not be anonymous. I also endorse restricting downvotes.

Comment author: satt 18 December 2016 03:21:51PM 10 points [-]

I'd also like to see downvotes & upvotes shown separately, but want to keep their anonymity.

The big upside of a downvote is that it lets you quickly signal that a comment's bad without having its poster follow you around afterwards or draw you into an unproductively time-consuming argument. This can of course be abused, but in LW's one big case of downvote abuse (Eugine_Nier) it didn't take long to see who was behind it anyway.

Comment author: NatashaRostova 17 December 2016 11:56:16PM 1 point [-]

I wonder what the statistical power of the study was.

With n = ~2000, and dementia rates being relatively low, and there either being no controls or some lame half-missing linear controls (even worse than no control, because it makes you think the control worked), and the treatment being seemingly arbitrary ,I basically am going to assume this is meaningless information.

It's turning an uncontrolled correlation in a low power sample into a causal story of protection.

Anyway, I didn't actually read the paper so maybe I'm being unfair. I somehow doubt that's the case though.

Comment author: satt 18 December 2016 01:35:52PM 0 points [-]

Thinking along basically the same lines, I tried to access the actual paper via its DOI link and got redirected to a "Production in progress" page. So we have what looks suspiciously like an embargo!

View more: Next