Lumifer comments on Fixing Moral Hazards In Business Science - Less Wrong

33 Post author: DavidLS 18 October 2014 09:10PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (96)

Comments (96)

You are viewing a single comment's thread. Show more comments above.

Comment author: Lumifer 20 October 2014 08:21:24PM *  1 point [-]

Well, I still don't understand how it's supposed to work.

Let's take a specific example. For example there is a kitchen gadget, Sansaire sous vide circulator, that was successfully funded on Kickstarter. Let's pretend that I'm one of the people behind the project and I'm interested in (1) whether people find that product useful; (2) whether the product is safe.

How would the product study go in this particular example? Who does what in which sequence?

Comment author: DavidLS 21 October 2014 08:45:02AM *  0 points [-]

I described an overview in a different thread, but that was before a lot of discussion happened.

I'll use this as an opportunity to update the script based on what has been discussed. This is of course still non-final.

  1. The creator of the sous vide machine (Tester) would register his study and agree to the terms.
  2. The Tester would register this as a food-related study, automatically adding required safety questions.
  3. The Tester would perform a search of our questions database and locate customer satisfaction related questions.
  4. The Tester would click "start the experiment".
  5. Our application would post an ad seeking participants.
  6. The participants would register for the study, once a critical mass was reached our app would create a new instance of the data collection webapp.
  7. Once the study period is complete, the data collection app signs and transfers the participant data back to our main app. The analysis is performed, the study is posted publicly, and the Tester is notified of the results via email.

Okay, hope (and the link to my earlier user story) helps make things more clear. If you see issues with this please do bring it up -- finding and fixing issues early is the reason I started this thread.

Comment author: Lumifer 21 October 2014 02:50:10PM 1 point [-]

You're describing just the scaffolding, but what actually happens? All the important stuff is between points 6 and 7 and you don't even allocate space for it :-/

Comment author: DavidLS 21 October 2014 07:34:01PM *  0 points [-]

The link I gave to the data collection webapp describes the data collection more depth, which I believe is what you are asking about between 6 and 7.

From that url:

Core function:

  • Every day an SMS/email is sent to participants with a securely generated one time URL.
  • The participant visits this URL and is greeted with a list of questions to answer.

Potential changes to this story:

  • If the URL is not used within 16 hours, it expires forever.
  • If a participant does not enter data for more than 3 days, they are automatically removed from the study.
  • If a participant feels that they need to removed from the study, they may do so at any time. They will be prompted to provide details on their reasons for doing so. These reasons will be communicated to the study organizer.
  • The study organizer may halt the study for logistical or ethical reasons at any time
Comment author: Lumifer 21 October 2014 08:12:11PM *  0 points [-]

which I believe is what you are asking

No, not really. Recall the setting -- I am about to produce a sous vide circulator and am interested (1) whether people find that product useful; and (2) whether the product is safe. I see nothing in your post which indicates how the process of answering my questions will work.

By the way, shipping a product to random people and asking them "Is it useful?" and "Did you kill yourself at any point during the last 24 hours?" is not likely to produce anything useful at all, never mind a proper scientific study.

Comment author: DavidLS 21 October 2014 11:32:38PM *  0 points [-]

"Did you kill yourself at any point during the last 24 hours?" is not likely to produce anything useful at all.

I see. Right now the system doesn't have any defined questions. I believe that suitable questions will be found so I'm focusing on the areas I have a solid background in.

If a project is unsafe in a literal way, shipping the product to consumers (or offering it for sale) is of course illegal. However, when considering a sous vide cooker in the past I have always worried about the dangers of potentially eating undercooked food (eg. diarrhea, nausea, and light headedness), which was how I took your meaning previously. "Product is safe for use, but accidental use might lead to undesirable outcomes". As I mentioned in our discussion here this project is not intended to be a replacement for the FDA.

shipping a product to random people and asking them "Is it useful?" ... is not likely to produce anything useful

I agree that "is it useful" is not a particularly useful question to ask, but I don't see any harm in supporting it. If you are looking for a better question, "80% of users used the product twice a week or more three months after receiving it" sounds like information that would personally help me make a buying decision. (Have you used the product today?)

So perhaps frequency of use might be a better question? I wasn't haggling over what questions to ask because it was your example.

never mind a proper scientific study

I think rigor in data collection and data processing are what make something scientific. For an example, you could do a rigorous study on "do you think the word turtle is funny?".

Comment author: Lumifer 22 October 2014 03:03:52PM 1 point [-]

Sorry, I don't find this idea, at least in its present form, useful. However I've certainly been wrong before and will be wrong in the future so it's quite possible I"m wrong now as well :-) There doesn't seem to be much point for me to play the "Yes, but" game and I'll just tap out.

Powered by Reddit Powered by Reddit