Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: Douglas_Knight 20 December 2016 05:31:59PM 3 points [-]

this likely understates the magnitudes of differences in underlying traits across cities, owing to people anchoring on the people who they know when answering the questions

Sure, if there really are differences, this method probably underestimates them. But there are other phenomena that could create false differences, such as varying social desirability bias.

Comment author: JonahSinick 21 December 2016 04:45:07AM 1 point [-]

Yes, this is something that I've wondered about quite a bit specifically in connection with the variation in conscientiousness and agreeableness by religion. I plan on partially addressing this issue by discussing some objective behavioral proxies to the personality traits in later posts.

Comment author: taygetea 20 December 2016 11:53:18PM 2 points [-]

I think you have the causality flipped around. Jonah is suggesting that something about Berkeley contributes to the prevalence of low conscientiousness among rationalists.

Comment author: JonahSinick 21 December 2016 04:32:49AM 4 points [-]

What I had in mind was that the apparent low average conscientiousness in the Bay Area might have been one of the cultural factors that drew rationalists who are involved in the in-person community to the location. But of course the interpretation that you raise is also a possibility.

Comment author: Qiaochu_Yuan 20 December 2016 09:08:15PM *  2 points [-]

Thanks for writing this! I really think people should be doing this (applying well-known algorithms to interesting datasets and seeing what happens) a lot more often overall, and it's on my list of skills I'd really like to learn personally. So I'd be interested to hear a little more info on methodology - what programming language(s) you used, how you generated the graphs, etc.

I'm pretty skeptical of making any connections to the Bay Area rationalist community based on Berkeley's conscientiousness score (which I think is interesting but not for this reason). There are 100,000 people living in Berkeley, and most of them aren't rationalists. And depending on how far back most of this data was collected, plausibly most of the Berkeley respondents were high school or college students (UC Berkeley alone has over 35,000 students), since for awhile that was the main demographic of Facebook users, and probably for awhile longer that was the main demographic of Facebook users willing to take personality tests. (Edit: But see Douglas_Knight's comment below.) In general I'd think more about selection effects like this before drawing any conclusions.

Comment author: JonahSinick 21 December 2016 04:26:16AM *  3 points [-]

Glad you liked it :-).

So I'd be interested to hear a little more info on methodology - what programming language(s) you used, how you generated the graphs, etc.

I used R for this analysis. Some resources that you might find relevant:

And depending on how far back most of this data was collected, plausibly most of the Berkeley respondents were high school or college students (UC Berkeley alone has over 35,000 students), since for awhile that was the main demographic of Facebook users, and probably for awhile longer that was the main demographic of Facebook users willing to take personality tests.

Douglas_Knight is correct – the average age of users is quite low, at ~26 years old both for the high conscientiousness cities and the low conscientiousness cities.

How does personality vary across US cities?

30 JonahSinick 20 December 2016 08:00AM

In 2007, psychology researchers Michal Kosinski and David Stillwell released a personality testing app on Facebook app called myPersonality. The app ended up being used by 4 million Facebook users, most of whom consented to their personality question answers and some information from their Facebook profiles to be used for research purposes.

The very large sample size and matching data from Facebook profiles make it possible to investigate many questions about personality differences that were previously inaccessible. Koskinski and Stillwell have used it in a number of interesting publications, which I highly recommend (e.g. [1], [2] [3]).

In this post, I focus on what the dataset tells us about how big five personality traits vary by geographic region in the United States

continue reading »
Comment author: JonahSinick 10 December 2016 08:05:54PM 1 point [-]

Physics is established, so one can defer to existing authorities and get right answers about physics. Starting a well-run laundromat is also established, so ditto. Physics and laundromat-running both have well-established feedback loops that have validated their basic processes in ways third parties can see are valid.

Depending on which parts of physics one has in mind, this seems possibly almost exactly backwards (!!). Quoting from Vladimir_M's post Some Heuristics for Evaluating the Soundness of the Academic Mainstream in Unfamiliar Fields:

If a research area has reached a dead end and further progress is impossible except perhaps if some extraordinary path-breaking genius shows the way, or in an area that has never even had a viable and sound approach to begin with, it’s unrealistic to expect that members of the academic establishment will openly admit this situation and decide it’s time for a career change. What will likely happen instead is that they’ll continue producing output that will have all the superficial trappings of science and sound scholarship, but will in fact be increasingly pointless and detached from reality.

Arguably, some areas of theoretical physics have reached this state, if we are to trust the critics like Lee Smolin. I am not a physicist, and I cannot judge directly if Smolin and the other similar critics are right, but some powerful evidence for this came several years ago in the form of the Bogdanoff affair, which demonstrated that highly credentialed physicists in some areas can find it difficult, perhaps even impossible, to distinguish sound work from a well-contrived nonsensical imitation.

The reference to Smolin is presumably to The Trouble With Physics: The Rise of String Theory, the Fall of a Science, and What Comes Next . Penrose's recent book Fashion, Faith, and Fantasy in the New Physics of the Universe also seems relevant.

Comment author: JonahSinick 03 December 2016 03:46:00AM *  22 points [-]

A few nitpicks on choice of "Brier-boosting" as a description of CFAR's approach:

Predictive power is maximized when Brier score is minimized

Brier score is the sum of differences between probabilities assigned to events and indicator variables that are are 1 or 0 according to whether the event did or did not occur. Good calibration therefore corresponds to minimizing Brier score rather than maximizing it, and "Brier-boosting" suggests maximization.

What's referred to as "quadratic score" is essentially the same as the negative of Brier score, and so maximizing quadratic score corresponds to maximizing predictive power.

Brier score fails to capture our intuitions about assignment of small probabilities

A more substantive point is that even though the Brier score is minimized by being well-calibrated, the way in which it varies with the probability assigned to an event does not correspond to our intuitions about how good a probabilistic prediction is. For example, suppose four observers A, B, C and D assigned probabilities 0.5, 0.4, 0.01 and 0.000001 (respectively) to an event E occurring and the event turns out to occur. Intuitively, B's prediction is only slightly worse than A's prediction, whereas D's prediction is much worse than C's prediction. But the difference between the increase in B's Brier score and A's Brier score is 0.36 - 0.25 = 0.11, which is much larger than corresponding difference for D and C, which is approximately 0.02.

Brier score is not constant across mathematically equivalent formulations of the same prediction

Suppose that a basketball player is to make three free throws, observer A predicts that the player makes each one with probability p and suppose that observer B accepts observer A's estimate and notes that this implies that the probability that the player makes all three free throws is p^3, and so makes that prediction.

Then if the player makes all three free throws, observer A's Brier score increases by

3*(1 - p)^2

while observer B's Brier score increases by

(1 - p^3)^2

But these two expressions are not equal in general, e.g. for p = 0.9 the first is 0.03 and the second is 0.073441. So changes to Brier score depend on the formulation of a prediction as opposed to the prediction itself.

======

The logarithmic scoring rule handles small probabilities well, and is invariant under changing the representation of a prediction, and so is preferred. I first learned of this from Eliezer's essay A Technical Explanation of a Technical Explanation.

Minimizing logarithmic score is equivalent to maximizing the likelihood function for logistic regression / binary classification. Unfortunately, the phrase "likelihood boosting" has one more syllable than "Brier boosting" and doesn't have same alliterative ring to it, so I don't have an actionable alternative suggestion :P.

Comment author: JonahSinick 27 November 2016 08:20:23PM 10 points [-]

Brian Tomasik's article Why I Prefer Public Conversations is relevant to

I suspect that most of the value generation from having a single shared conversational locus is not captured by the individual generating the value (I suspect there is much distributed value from having "a conversation" with better structural integrity / more coherence, but that the value created thereby is pretty distributed). Insofar as there are "externalized benefits" to be had by blogging/commenting/reading from a common platform, it may make sense to regard oneself as exercising civic virtue by doing so, and to deliberately do so as one of the uses of one's "make the world better" effort. (At least if we can build up toward in fact having a single locus.)

Comment author: gjm 22 August 2016 10:13:27PM -1 points [-]

Wait, your category (ii) is surely exactly what we care about here. We want to know: For someone whose background would lead you not to expect high-paying data science jobs, is Signal effective in getting them a better chance of a high-paying data science job?

Comment author: JonahSinick 23 August 2016 12:30:06AM *  1 point [-]

Wait, your category (ii) is surely exactly what we care about here.

Yes, I see how my last message was ambiguous.

What I had in mind in bringing up category (ii) is that we've had some students who had a priori worse near term employment prospects relative to the usual range of bootcamp attendees, who are better positions than they had been and who got what they were looking to get from the program, while not yet having $100k+ paying jobs. And most students who would have gotten $100k+ paying jobs even if they hadn't attended appear to have benefited from attending the program.

The nature of the value that we have to add is very much specific to the student.

Comment author: ThisSpaceAvailable 21 August 2016 01:31:02AM *  4 points [-]

I suppose this might be better place to ask than trying to resurrect a previous thread:

What kind of statistics can Signal offer on prior cohorts? E.g. percentage with jobs, percentage with jobs in data science field, percentage with incomes over $100k, median income of graduates, mean income of graduates, mean income of employed graduates, etc.? And how do the different cohorts compare? (Those are just examples; I don't necessarily expect to get those exact answers, but it would be good to have some data and have it be presented in a manner that is at least partially resistant to cherry picking/massaging, etc.) Basically, what sort of evidence E does Signal have to offer, such that I should update towards it being effective, given both E, and "E has been selected by Signal, and Signal has an interest in choosing E to be as flattering rather than as informative as possible" are true?

Also, the last I heard, there was a deposit requirement. What's the refund policy on that?

Comment author: JonahSinick 22 August 2016 09:33:02PM 1 point [-]

Hello! I'm a cofounder of Signal Data Science.

Because our students have come into the program from very heterogeneous backgrounds (ranging from high school dropout to math PhD with years of experience as a software engineer), summary statistics along the lines that you're looking for are less informative than might seem to be the case prima facie. In particular, we don't yet have meaningfully large sample of students who don't fall into one of the categories of (i) people who would have gotten high paying jobs anyway and (ii) people who one wouldn't expect to have gotten high paying jobs by now, based on their backgrounds.

If you're interested in the possibility of attending the program, we encourage you to fill out our short application form. If it seems like it might be a good fit for you, we'd be happy to provide detailed answers to any questions that you might have about job placement.

Comment author: ThisSpaceAvailable 16 June 2016 02:21:42AM 0 points [-]

"We're planning another one in Berkeley from May 2nd – July 24th."

Is that June 24th?

Comment author: JonahSinick 17 June 2016 10:48:35PM 0 points [-]

Yes, that was supposed to be June 24th! We have a third one from July 5th – August 24th. There are still spaces in the program if you're interested in attending.

View more: Next