You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

DavidLS comments on Fixing Moral Hazards In Business Science - Less Wrong Discussion

33 Post author: DavidLS 18 October 2014 09:10PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (96)

You are viewing a single comment's thread. Show more comments above.

Comment author: DavidLS 19 October 2014 11:05:57PM 0 points [-]

The participant has no way to decide between the two packages or know which one is the placebo and which one is the real thing so he doesn't need to go through the process of flipping a real coin.

I am worried that standardized shipping will come with standardized package layout, and I'm guessing "preference of left vs right identical thing" correlates with something the system will eventually test. Having thought about it more, this is the real issue with allowing customers to choose which product they'll use: that decision has to be purely random if you want the math to be simple / understandable. I agree people are unlikely to actually flip a coin :/

Thankfully the fix is easy: you have the testing webapp decide for the participant. They receive the product, enter the numbers online, and are told which to use.

For non-crossover trials I agree this needlessly increases the cost. It's almost surely better to use a trusted third party.

That means the participants has to do the work of going to the post office and remailing packages. Some of them will require additional time to remail and it might produce complications.

Agree. I think having a trusted third party handle the shipments is cleaner at the moment. I'm still curious what blogospheroid's thread comes to. It seems like the paranoia of cryptoland is helping us see some more holes in modern experiment design (ie your thread on poisonous placebos).

Maybe you can win a company such as Amazon as a partner for distribution. Amazon warehouses can store both products and placebos and ship randomly.

Thanks for saying this. Looking closer, I actually think their existing Fulfillment APIs would just work for this (ie the webapp controls an Amazon fulfillment account, the person seeking a test ships two pallets of physical product there, the webapp says where to send them).

Comment author: ChristianKl 20 October 2014 12:02:43AM 0 points [-]

Looking closer, I actually think their existing Fulfillment APIs would just work for this (ie the webapp controls an Amazon fulfillment account, the person seeking a test ships two pallets of physical product there, the webapp says where to send them).

You are right, if we already have hosted our webapp at trust-place we should be able to use the existing Amazon API.

If the company whose product is tested simply ships additional copies to the Amazon warehouse, those copies could by achieved by the trusted organisation. If anybody doubts that the products are real the trusted organisation has copies that they can analyse. If the whole things scales the trusted organisation also can randomly inspect products to see if they contain what they should contain.

Agree. I think having a trusted third party handle the shipments is cleaner at the moment. I'm still curious what blogospheroid's thread comes to. It seems like the paranoia of cryptoland is helping us see some more holes in modern experiment design

Yes cryptoparanoia is always fun ;) The web app could regularly publish hashes of the data of specific studies to a public block chain. That way any tempering that happens afterwards can be detected and you only need to trust that the web app is temper proof the moment the data gets transmitted.

Comment author: DavidLS 20 October 2014 02:12:24AM 0 points [-]

If anybody doubts that the products are real the trusted organisation has copies that they can analyse.

This is a great point. Maybe community members could bet karma on the outcome of a tox screening? This could create a prioritized list.

One problem with my earlier suggestion is that some companies will want narrowly selected participant pools. These will necessarily differ from the population at large, and might create data that looks like a poison placebo is being used. I see two possible solutions to this problem:

  1. Log baseline data before the treatment is used. If people do worse on the placebo, that would be very suspicious.
  2. Include an additional group of testers that do something different not related to the placebo/product. "Eat an apple every day for the next week". If the placebo group did worse than the apple group, that would be very suspicious.

I feel like #2 from above is unsatisfying though, if we think it works then why are we using normal placebos?

The web app could regularly publish hashes of the data of specific studies to a public block chain. That way any tempering that happens afterwards can be detected and you only need to trust that the web app is temper proof the moment the data gets transmitted.

This would actually be really easy to implement. (Not the block chain portion, the per-study rolling checksums).