Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

taryneast comments on That Alien Message - Less Wrong

111 Post author: Eliezer_Yudkowsky 22 May 2008 05:55AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (164)

Sort By: Old

You are viewing a single comment's thread. Show more comments above.

Comment author: taryneast 30 December 2010 10:51:05PM *  6 points [-]

plz specifically disprove the defenses I've written.

Expense.

People will not pay for the extensive defenses you have suggested... at least not until it's been proven necessary... ie it's already too late.

Even then they'll bitch and moan about the inconvenience, and why wouldn't you? hair-trigger bomb on every computer on the planet? ready to go off the moment it "detects an anomaly"?

Have you any idea how many bugs there are in computer applications? Would you trust your life (you'll die in the bomb too) to your computer not crashing due to some dodgy malware your kid downloaded while surfing for pron?

Even if it's just on the computers that are running the AGI (and AGI programmers are almost as susceptible to malware), it would still be nigh-on-impossble to "detect an anomaly".

What's an anomaly? How do we determine it? Any program that tried to examine its own code looking for an anomaly would have to simulate the running of the very code it was testing... thus causing the potentiality for it to actually become the anomalous program itself. ...it's not actually possible to determine what will happen in a program any other way (and even then I'd be highly dubious).

So... nice try, but sadly not really feasible to implement. :)