Perhaps it would be best to have as many teams as possible working on different pieces independently, with some form of arithmetic coding operating over their output.
Could you clarify what you mean by "arithmetic coding operating over their output"?
The point of having teams work independently on the same project is that they're unlikely to make exactly the same mistakes. Publishers do this for proofreading: have two proofreaders return error-sets A and B, and estimate the number of uncaught errors as a function of |A\B|, |B\A| and |A∩B|. If A=B, that would be strong evidence that there are no errors left. (But depending on priors, it might not bring P(no errors left) close to 1.) If two proofreaders worked on different parts of the book, you couldn't use the same technique.
Could the arithmetic coding make checks like this unnecessary?
One possible answer to the argument "attempting to build FAI based on Eliezer's ideas seems infeasible and increases the risk of UFAI without helping much to increase the probability of a good outcome, and therefore we should try to achieve a positive Singularity by other means" is that it's too early to decide this. Even if our best current estimate is that trying to build such an FAI increases risk, there is still a reasonable chance that this estimate will turn out to be wrong after further investigation. Therefore, the counter-argument goes, we ought to mount a serious investigation into the feasibility and safety of Eliezer's design (as well as other possible FAI approaches), before deciding to either move forward or give up.
(I've been given to understand that this is a standard belief within SI, except possibly for Eliezer, which makes me wonder why nobody gave this counter-argument in response to my post linked above. ETA: Carl Shulman did subsequently give me a version of this argument here.)
This answer makes sense to me, except for the concern that even seriously investigating the feasibility of FAI is risky, if the team doing so isn't fully rational. For example they may be overconfident about their abilities and thereby overestimate the feasibility and safety, or commit sunken cost fallacy once they have developed lots of FAI-relevant theory in the attempt to study feasibility, or become too attached to their status and identity as FAI researchers, or some team members may disagree with a consensus of "give up" and leave to form their own AGI teams and take the dangerous knowledge developed with them.
So the question comes down to, how rational is such an FAI feasibility team likely to be, and is that enough for the benefits to exceed the costs? I don't have a lot of good ideas about how to answer this, but the question seems really important to bring up. I'm hoping this post this will trigger SI people to tell us their thoughts, and maybe other LWers have ideas they can share.