DavidLS comments on Fixing Moral Hazards In Business Science - Less Wrong

33 Post author: DavidLS 18 October 2014 09:10PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (96)

Comments (96)

You are viewing a single comment's thread. Show more comments above.

Comment author: MaximumLiberty 18 October 2014 11:58:52PM 11 points [-]

Thanks for the example. It leads me to questions:

  1. For more complicated propositions, who does the math and statistics? The application apparently gathers the data, but it is still subject to interpretation.
  2. Is the data (presumably anonymized) made publicly available, so that others can dispute the meaning?
  3. If the sponsoring company does its own math and stats, must it publicly post its working papers before making claims based on the data? Does anyone review that to make sure it passes some light smell test, and isn't just pictures of cats?
  4. What action does the organization behind the app take if a sponsor publicly misrepresents the data or, more likely, its meaning? If the organization would take action, does it take the same action if the statement is merely misleading, rather than factually incorrect?
  5. What do the participants get? Is that simply up to the sponsor? If so, who reviews it to assure that the incentive does not distort the data? If no one, will you at least require that the incentive be reported as part of the trial?
  6. Does a sponsor have any recourse if it designed the trial badly, leading to misleading results? Or is its remedy really to design a better trial and publicize that one?
  7. Can sponsors do a private mini-trial to test its trial design before going full bore (presumably, with their promise not to publicize the results)?
  8. Have you considered some form of reputation system, allowing commenters to build a reputation for debunking badly supported claims and affirming well-supported claims? (Or perhaps some other goodie?) I can imagine it becoming a pastime for grad students, which would be a Good Thing (TM).

I imagine these might all be very basic questions that arise out of my ignorance of such studies. If so, please spend your time on people with more to contribute than ignorance!

Max L.

Comment author: DavidLS 19 October 2014 01:03:28AM *  4 points [-]

2 - Is the data (presumably anonymized) made publicly available, so that others can dispute the meaning?

That was the initial plan, yes! Beltran (my co-founder at GB) is worried that will result in either HIPPA issues or something like this, so I'm ultimately unsure. Putting structures in place so the science is right the first time seems better.

Comment author: sbenthall 20 October 2014 04:52:41AM 4 points [-]

The privacy issue here is interesting.

It makes sense to guarantee anonymity. Participants recruited personally by company founders may be otherwise unwilling to report honestly (for example). For health related studies, privacy is an issue for insurance reasons, etc.

However, for follow-up studies, it seems important to keep earlier records including personally identifiable information so as to prevent repeatedly sampling from the same population.

That would imply that your organization/system needs to have a data management system for securely storing the personal data while making it available in an anonymized form.

However, there are privacy risks associated with 'anonymized' data as well, since this data can sometimes be linked with other data sources to make inferences about participants. (For example, if participants provide a zip code and certain demographic information, that may be enough to narrow it down to a very few people.) You may want to consider differential privacy solutions or other kinds of data perturbation.

http://en.wikipedia.org/wiki/Differential_privacy

Powered by Reddit Powered by Reddit