Recently, I have been learning about industry norms, legal discovery proceedings, and incentive structures related to companies building risky systems. I wanted to share some findings in this post because they may be important for the frontier AI community to understand well.
TL;DR
Documented communications of risks (especially by employees) make companies much more likely to be held liable in court when bad things happen. The resulting Duty to Due Diligence from Discoverable Documentation of Dangers (the 6D effect) can make companies much more cautious if even a single email is sent to them communicating a risk.
Companies tend to avoid talking about risk through documented media.
Companies often intentionally avoid discussing the risks of what they are doing through permanent media such as email. For example, this article gives some very shady advice on how companies can avoid liability by using “safe communication” practices to avoid the creation of incriminating “bad documents”.
Often the drafters of these documents tend to believe that they are providing the company with some value to the business. For example, an engineer notices a potential liability in a design so he informs his supervisor through an email. However, the engineer’s lack of legal knowledge and misuse of legal vocabulary in the communication may later implicate the company with notice of the problem when a lawsuit arises.
I personally enjoyed the use of “when” and not “if” in the excerpt.
This is a perverse consequence of how it is relatively hard for companies to be held liable for risks when it cannot be proven they knew about them, even if they did. When an incident happens and a company is sued, evidence about its role in the problem is gathered during what is known as the “discovery” phase of a lawsuit (emails are usually discoverable). When records showing that a company had knowledge of the problem are found in discovery, they are much more likely to be found liable.
One email can have a lot of power.
The unfortunate consequence of how discovery works is that companies strategically avoid communicating risks via documented media. But there is a silver lining. The threat of liability due to documented communications of risks can have a lot of influence over how cautious a company is. One discoverable record of a risk can be very impactful.
I like to call this the 6D effect – the Duty to Due Diligence from Discoverable Documentation of Dangers.
A few examples
Here are some notable examples of companies being held liable for damages because they ignored documented communication of risks (but there are many throughout legal history).
- In Grimshaw v. Ford Motor Company, 1981, Ford was held liable for damages involving a fatal crash with a Ford Pinto because it was shown that leadership within the company ignored warnings about problems with the vehicle’s fuel system.
- In April of this year, a large settlement was reached after the 2017 Grenfell Tower fire in London, which killed 72 people. A big factor in the lawsuit was that the company managing the tower had ignored numerous fire safety warnings which were found in discovery.
- Last year, the Hardwick v. 3M case ended. It was a class action lawsuit from 2018 about the presence of harmful “forever chemicals” (PFAS) in consumer products. The company behind these chemicals was found to have known about risks since the 1970s but was knowingly negligent, which led to a ruling against them.
Miscellaneous notes
- The 6D effect can result from any discoverable communication, but it is especially powerful when the warning comes from an employee of the company itself.
- If you communicate a risk, it is important to speak up and bring documentation of it to the attention of a court during the discovery phase of a lawsuit.
- If you are aware that something a company has done is hazardous, it is your ethical obligation to inform the company, but it is NOT your ethical obligation to help them fix it without compensation. Make sure not to let a company take advantage of you.
Three takeaways
- If you work at a company doing potentially risky things, insist on discussing dangers through documented media. If you are retaliated against for documenting communication of risks, you may have grounds for legal recourse. #notlegaladvice
- If you notice something risky, say something. If the thing you predicted happens, point out the fact that you communicated it.
- Safety-focused companies (such as those working on frontier AI systems) should have explicit policies about documenting all discussions of risk.
It doesn't matter. Plaintiff wants to prove that an engineer told the CEO that the widgets were dangerous. So he introduces testimony from the engineer that the engineer told the CEO that the widgets were dangerous. Defendant does not dispute this. How much more weight could you possibly want? The only other thing you could do was to ask defendant to stipulate that the engineer told the CEO about the widgets. I think most lawyers wouldn't bother.
Because it makes it look like they're trying to conceal evidence, which is much worse for them than simply maybe being negligent. This could it be grounds for punitive damages or an adverse inference ruling or both. It would also be so easy for plaintiff to score off of that the court might not even bother with an adverse inference
If I say I want you to turn over your email records to me in discovery to establish that an engineer had told you that your widgets were dangerous, but you instead destroy those records, the court will instruct the jury to assume that those records did contain that evidence. This is an adverse inference.
Even if there's no adverse inference, just think about what happens. Defendant attempts to counter the testimony by saying that if this meeting took place there would be records, but there aren't, so you must be lying. Plaintiff responds by showing that defendant had a policy designed to prevent such records from being created, so defendant knows that records would not exist whether the meeting took place or not, and thus his argument is disingenuous. Would you follow defendant's strategy here? I wouldn't.
Remember, they're not conceding the whole case, just the fact that the engineer told them his opinion. What they're going to do instead is admit that the first engineer told them, but that they asked some other other engineers to weigh in on the point, and those engineers disagreed. They decided to trust the other engineers and thus the resulting injury wasn't negligent, it was just a mistake.
This is what I was saying before. And moreover, if the actual main effect of these policies was to prevent creation of discoverable records of real bad stuff the companies deserved to burn for, they'd probably risk trouble just for having them in the first place.
In my own case, I was working on a project where I thought it would be useful for us to have a copy of the entire WHOIS database. Under ICANN's rules (IIRC), a registrar who got the WHOIS database in the first place was required to sell copies to organizations which meet certain criteria, which we definitely did. I was told that regardless of ICANN's rules, registrars all just ignore that rule and refuse to sell the database. I said that in that case we should just flagrantly violate their ToS and scrape ourselves a full copy from their free lookup interface, and if they didn't like it they could sell me a copy like they were required too. It's not like they'd be able to stop me. Legal got in touch and said basically "Look, we're not saying you're wrong, but keep that off of discoverable channels. Proving that you're right in court costs money." Which of course was correct, and exactly what I was missing. As an engineer, I could establish that it was both feasible and legally winnable, but it takes a lawyer to know about the expected legal costs in the event of a conflict and how to minimize those. Anyway, we never ended up actually doing it for unrelated reasons.