To make sure I'm understanding you correctly, do you think the largest problem comes from (1) thinking of AGI development as a sequence of Bernoulli trials, or (2) each Bernoulli trial having constant probability, or (3) both?
It's not obvious to me that (1) is hugely problematic - isn't Laplace's rule of succession commonly applied to forecasting previously unseen events? Are you perhaps arguing that there's something particular to AGI development such that thinking of it as a series of Bernoulli trials is completely invalid?
(1) seems reasonable as a model at this level of abstraction, absent quibbles about whether some outcome really is AGI or not, instead of some degree of AGI-ness.
(2) seems utterly wrong, and I don't think it even makes sense to talk about a "first trial" as being a clear-cut thing, let alone having a sensible probability, and definitely not as something related to success of all future trials. I contend that it is not even "semi-informative", it is useless.
To make sure I'm understanding you correctly, do you think the largest problem comes from (1) thinking of AGI development as a sequence of Bernoulli trials, or (2) each Bernoulli trial having constant probability, or (3) both?
It's not obvious to me that (1) is hugely problematic - isn't Laplace's rule of succession commonly applied to forecasting previously unseen events? Are you perhaps arguing that there's something particular to AGI development such that thinking of it as a series of Bernoulli trials is completely invalid?
I'm more sympathetic to your cr... (read more)