All of Roger Scott's Comments + Replies

How is it that some tiny number of man made mirror life forms would be such a threat to the millions of naturally occurring life forms, but those millions of naturally occurring life forms would not be an absolutely overwhelming symmetrical threat to those few man made mirror forms? This whole analysis seems to be assuming that the man made mirror forms will have some grossly asymmetrical advantage.

0azergante
I have a similar intuition that if mirror-life is dangerous to Earth-life, then the mirror version of mirror-life (that is, Earth-life) should be about equally as dangerous to mirror-life as mirror-life is to Earth-life. Having only read this post and in the absence of any evidence either way this default intuition seems reasonable. I find the post alarming and I really wish it had some numbers instead of words like "might" to back up the claims of threat. At the moment my uneducated mental model is that for mirror-life to be a danger it has to: * find enough food that fit its chirality to survive * not get killed by other life-forms * be able to survive Earth temperature, atmosphere etc etc * enter our body * bypass our immune system * be a danger to us Hmm, 6 ifs seems like a lot, so is it unlikely? in the absence of any odds it is hard to say. The post would be more convincing and useful if it included a more detailed threat model, or some probabilities, or a simulation, or anything quantified. A last question: how many mirror molecules does an organism need to be mirror-life? is one enough? does it make any difference to its threat-level?
7Carl Feynman
The advantage is that they would have neither predators nor parasites, and their prey would not have adapted defenses to them.  This would be true of any organism with a sufficiently unearthly biochemistry.  Mirror life is the only such organism we are likely to create in the near term.

How is it that some tiny number of man made mirror life forms would be such a threat to the millions of naturally occurring life forms, but those millions of naturally occurring life forms would not be an absolutely overwhelming symmetrical threat to those few man made mirror forms?

Can’t you ask the same question for any invasive species? Yet invasive species exist. “How is it that some people putting a few Nile perch into Lake Victoria in the 1950s would cause ‘the extinction or near-extinction of several hundred native species’, but the native species of... (read more)

2jasoncrawford
The asymmetric advantage of bacteria is that they can invade your body but not vice versa.

Given only finite time, isn't one always omitting nearly everything? If you believe in dishonesty by omission is everyone not dishonest, in that sense, nearly all the time? You can argue that only "relevant" information is subject to non-omission, but since relevance is a subjective, and continuous, property this doesn't seem like very useful guidance. Wherever you choose to draw the line someone can reasonably claim you've omitted relevant (by some other standard) information just on the other side of that line.

It seems like this discussion might cover power imbalances between speaker and listener more. For example, in the border agent example, a border control agent has vastly more power than someone trying to enter the country. This power gives them the "right" (read: authority) to ask all sorts of question, the legitimacy of which might be debatable. Does deep honesty compel you to provide detailed, non-evasive answer to questions you personally don't believe the interlocutor has any business asking you? The objective of such interactions is not improving the ... (read more)

In software engineering things often become "accidentally load bearing" when people don't respect interfaces. If they go digging around in a component's implementation they learn things that happen to be true but are not intended to be guaranteed to be true. When they start relying on these things it limits the ability of the maintainer of the component to make future changes. This problem is exacerbated by under-specified interfaces, either when formal specification mechanisms are underutilized or, more often, due to the limits of most formal interfaces specification mechanisms, when important behavioral aspects of an interface are not documented.

I don't think you even need to go as far as you do here to undermine the "emergent convergence (on anti-human goals)" argument. Even if we allow that AIs, by whatever means, develop anti-human goals, what reason is there to believe that the goals (anti-human, or otherwise) of one AI would be aligned with the goals of other AIs? Although infighting among different AIs probably wouldn't be good for humans, it is definitely not going to help AIs, as a group, in subduing humans.

Now let's bring in something which, while left out of the primary argument, repeate... (read more)

Not only that, but if your goals is to create a powerful army of AIs the last thing you'd want to do is make them all identical. Any reason you're going to choose for why there are a huge number of AI instances in the first place -- as assumed by this argument -- would want those AIs to be diverse, not identical, and that very diversity would argue against "emergent convergence". You then have to revert to the "independently emerging common sub-goals" argument, which is a significantly bigger stretch because of the many additional assumptions it makes.

Isn't multi-epoch training most likely to lead to overfitting, making the models less useful/powerful?

If it were possible to write an algorithm to generate this synthetic training data how would the resulting training data have any more information content than the algorithm that  produced it? Sure, you'd get an enormous increase in training text volume, but large volumes of training data containing small amounts of information seems counterproductive for training purposes -- it will just bias the model disproportionately toward that small amount of information.

Why wouldn't people (and maybe even AIs, at least up to a point) be applying these ever-advancing AI capabilities to developing better and better interpretability tools as well? I.e., what reason is there to expect an "interpretability gap" to develop (unless you believe interpretability is a fundamentally unsolvable problem, in which case no amount of AI power is going to help)?