JGWeissman comments on What I would like the SIAI to publish - Less Wrong

27 Post author: XiXiDu 01 November 2010 02:07PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (218)

Comments (218)

Sort By: Best

Show:

You are viewing a single comment's thread. Show more comments above.

Comment author: JGWeissman 02 November 2010 07:34:13PM 0 points [-]

It occurs to me, as I listen to this debate, that a certain amount of sanity can be imposed on a utility-maximizing agent simply by specifying decreasing returns to scale and increasing costs to scale over the short term with the long term curves being somewhat flatter. That will tend to guide the agent away from explosive growth pathways.

Such an AI would still be motivated to FOOM to consolidate its future ability to achieve large utility against the threat of being deactivated before then.

Comment author: XiXiDu 02 November 2010 08:27:59PM 1 point [-]

Such an AI would still be motivated to FOOM to consolidate its future ability to achieve large utility against the threat of being deactivated before then.

It doesn't know about any threat. You implicitly assume that it has something equivalent to fear, that it perceives threats. You allow for the human ingenuity to implement this and yet you believe that they are unable to limit its scope. I just don't see that it would be easy to make an AI that would go FOOM because it doesn't care to go FOOM. If you tell it to optimize some process then you'll have to tell it what optimization means. If you can specify all that, how is it then still likely that it somehow comes up with its own idea that optimization might be to consume the universe if you told it to optimize its software running on a certain supercomputer? Why would it do that, where does the incentive come from? If I tell a human to optimize he might muse to turn the planets into computronium but if I tell a AI to optimize it doesn't know what it means until I tell it what it means and then it still won't care because it isn't equipped with all the evolutionary baggage that humans are equipped with.

Comment author: wedrifid 02 November 2010 09:08:39PM 4 points [-]

It doesn't know about any threat.

It is a general intelligence that we are considering. It can deduce the threat better than we can.

If you can specify all that, how is it then still likely that it somehow comes up with its own idea that optimization might be to consume the universe if you told it to optimize its software running on a certain supercomputer?

Because it is a general intelligence. It is smart. It is not limited to getting its ideas from you, it can come up with its own. And if the AI has been given the task of optimising its software for performance on a certain computer then it will do whatever it can to do that. This means harnessing external resources to do research on computation theory.

You implicitly assume that it has something equivalent to fear, that it perceives threats.

No he doesn't. He assumes only that it is a general intelligence with an objective. Potentially negative consequences are just part of possible universes that it models like everything else.

I'm not sure what can be done to make this clear:

SELF IMPROVEMENT IS AN INSTRUMENTAL GOAL THAT IS USEFUL FOR ACHIEVING MOST TERMINAL VALUES.

If I tell a human to optimize he might muse to turn the planets into computronium but if I tell a AI to optimize it doesn't know what it means until I tell it what it means and then it still won't care because it isn't equipped with all the evolutionary baggage that humans are equipped with.

You have this approximately backwards. A human knows that if you tell her to create 10 paperclips every day you don't mean take over the world so she can be sure that nobody will interfere with her steady production of paperclips in the future. The AI doesn't.

Comment deleted 03 November 2010 09:07:02AM * [+] (32 children)

Comment author: PeterisP 03 November 2010 09:51:56AM 3 points [-]

Assuming that a general, powerful intelligence has a goal 'do x', say - win chess games, optimize traffic flow or find cure for cancer, then it has implicit dangerous incentives if we don't figure out a reasonable Friendly framework to prevent them.

A self-improving intelligence that does changes to it's code to become better at doing it's task may easily find out that, for example, a simple subroutine that launches a botnet in the internet (as many human teenagers have done), might get it an x % improvement in processing power that helps it to obtain more wins chess games, better traffic optimizations or faster protein-folding for the cure of cancer.

A self-improving general intelligence that has human-or-better capabilities may easily deduce that a functioning off-button would increase the chances of it being turned off, and that it being turned off would increase the expected time of finding cure for cancer. This puts this off-button in the same class as any other bug that hinders its performance. Unless it understands and desires the off-button to be usable in a friendly way, it would remove it; or if it's hard-coded as nonremovable, then invent workarounds for this perceived bug - for example, develop a near-copy of itself that the button doesn't apply to, or spend some time (less than the expected delay due to the turning-off-risk existing, thus rational spending of time) to study human psychology/NLP/whatever to better be able to convince everyone that it shouldn't be turned off ever, or surround the button with steel walls - these are all natural extensions of it following it's original goal.

If an self-improving AI has a goal, then it cares. REALLY cares for it in a stronger way than you care for air, life, sex, money, love and everything else combined.

Humans don't go FOOM because they a)can't at the moment and b) don't care about such targeted goals. But for AI, at the moment all we know is how to define such supergoals which work in this unfriendly manner. At the moment we don't know how to make these 'humanity friendly' goals, and we don't know how to make an AI that's self-improving in general but 'limited to certain contraints'. You seem to imply these constraints as trivial - well, they aren't, the friendliness problem actually may as hard or harder than general AI itself.

Comment author: XiXiDu 03 November 2010 09:56:37AM * 0 points [-]

Assuming that a general, powerful intelligence has a goal 'do x', say - win chess games, optimize traffic flow or find cure for cancer, then it has implicit dangerous incentives if we don't figure out a reasonable Friendly framework to prevent them.

I think you misunderstand what I'm arguing about. I claim that general intelligence is not powerful naturally but mainly does possess the potential to become powerful and that it is not equipped with some goal naturally. Further I claim that if a goal can be defined to be specific enough that it is suitable to self-improve against it, it is doubtful that it is also unspecific enough not to include scope boundaries. My main point is that it is not as dangerous to work on AGI toddlers as some make it look like. I believe that there is a real danger but that to overcome it we have to work on AGI and not avoid it altogether because any step into that direction will kill us all.

Comment author: PeterisP 03 November 2010 04:55:16PM 4 points [-]

OK, well these are the exact points which need some discussion.

1) Your comment "general intelligence is [..] is not equipped with some goal naturally" - I'd say that it's most likely that any organization investing the expected huge manpower and resources in creating a GAI would create it with some specific goal defined for it.

However, in absence of an intentional goal given by the 'creators', it would have some kind of goals, otherwise it wouldn't do absolutely anything at all, so it wouldn't be showing any signs of it's (potential?) intelligence.

2) In response to "If a goal can be defined to be specific enough that it is suitable to self-improve against it, it is doubtful that it is also unspecific enough not to include scope boundaries" - I'd say that defining specific goals is simple, too simple. From any learning-machine design a stupid goal 'maximize number of paperclips in universe' would be very simple to implement, but a goal like 'maximize welfare of humanity without doing anything "bad" in the process' is an extremely complex goal, and the boundary setting is the really complicated part, which we aren't able to even describe properly.

So in my opinion is quite viable to define a specific goal that is suitable to self-improve against, and that includes some scope boundaries - but where the defined scope boundaries has some unintentional loophole which causes disaster.

3) I can agree that working on AGI research is essential, instead of avoiding it. But taking the step from research through prototyping to actually launching/betatesting a planned powerful self-improving system is dangerous if the world hasn't yet finished an acceptable solution to Friendliness or the boundary-setting problem. If having any bugs in the scope boundaries is 'unlikely' (95-95% confidence?) then it's not safe enough, because 1-5% chance of an extinction event after launching the system is not acceptable, it's quite a significant chance - not the astronomical chances involved in Pascal's wager or asteroid hitting the earth tomorrow or LHC ending the universe.

And given the current software history and published research on goal systems, if anyone would show up today and demonstrate that they've solved self-improving GAI obstacles and can turn it on right now, then I can't imagine how they could realistically claim a larger than 95-99% confidence in their goal system working properly. At the moment we can't check any better, but such a confidence level simply is not enough.

Comment author: XiXiDu 03 November 2010 05:46:19PM * 2 points [-]

Yes, I agree with everything. I'm not trying to argue that there exist no considerable risk. I'm just trying to identify some antipredictions against AI going FOOM that should be incorporated into any risk estimations as it might weaken the risk posed by AGI or increase the risk posed by impeding AGI research.

I was insufficiently clear that what I wanted to argue about is the claim that virtually all pathways lead to destructive results. I have an insufficient understanding of why the concept of general intelligence is inevitably connected with dangerous self-improvement. Learning is self-improvement in a sense but I do not see how this must imply unbounded improvement in most cases given any goal whatsoever. One argument is that the only general intelligence we know, humans, would want to improve if they could tinker with their source code. But why is it so hard to make people learn then? Why don't we see much more people interested in how to change their mind? I don't think you can draw any conclusions here. So we are back at the abstract concept of a constructed general intelligence (as I understand it right now), that is an intelligence with the potential to reach at least human standards (same as a human toddler). Another argument is based on this very difference between humans and AI's, namely that there is nothing to distract them, that they will possess an autistic focus on one mandatory goal and follow up on it. But in my opinion the difference here also implies that while nothing will distract them, there will also be no incentive not to hold. Why would it do more than necessary to reach a goal? The further argument here is that it will misunderstand its goals. But the problem I see in this case is firstly that the more unspecific the goal the less it is able to measure its self-improvement against the goal to quantify the efficiency of its output. Secondly, the more vague a goal the larger has to be its general knowledge, previous to any self-improvement, to make sense of it in the first place? Shouldn't those problems outweigh each other to some extent?

For example, if you told the AGI to become as good as possible in Formula 1, so that it was faster than any human race driver. How is it that the AGI is yet smart enough to learn this all by itself but fails to notice that there are rules to follow. Secondly, why would it keep improving once it is faster than any human rather than just hold and become impassive? This argument could be extended to many other goals which have scope bounded solutions.

Of course, if you told it to learn as much about the universe as possible, that is something completely different. Yet I don't see how this risk does raise against other existential risks like grey goo since it should be easier to create advanced replicators to destroy the world than creating AGI that then creates advanced replicators that then fails hold and then destroys the world?

Comment author: PeterisP 05 November 2010 10:14:55AM 5 points [-]

" How is it that the AGI is yet smart enough to learn this all by itself but fails to notice that there are rules to follow" - because there is no reason for an AGI automagically creating arbitrary restrictions if they aren't part of the goal or superior to the goal. For example, I'm quite sure that F1 rules prohibit interfering with drivers during the game; but if somehow a silicon-reaction-speed AGI can't win F1 by default, then it may find it simpler/quicker to harm the opponents in one of the infinity ways that the F1 rules don't cover - say, getting some funds in financial arbitrage, buying out the other teams, and firing any good drivers or engineering a virus that halves the reaction speed of all homo-sapiens - and then it would be happy as the goal is achieved within the rules.

Comment author: XiXiDu 05 November 2010 03:05:51PM * 1 point [-]

...because there is no reason for an AGI automagically creating arbitrary restrictions if they aren't part of the goal or superior to the goal.

That's clear. But let me again state what I'd like to inquire. Given the large amount of restrictions that are inevitably part of any advanced general intelligence (AGI), isn't the nonhazardous subset of all possible outcomes much larger than that where the AGI works perfectly yet fails to hold before it could wreak havoc? Here is where this question stems from. Given my current knowledge about AGI I believe that any AGI capable of dangerous self-improvement will be very sophisticated, including a lot of restrictions. For example, I believe that any self-improvement can only be as efficient as the specifications of its output are detailed. If for example the AGI is build with the goal in mind to produce paperclips, the design specifications of what a paperclip is will be used as leveling rule by which to measure and quantify any improvement of the AGI's output. This means that to be able to effectively self-improve up to a superhuman level, the design specifications will have to be highly detailed and by definition include sophisticated restrictions. Therefore to claim that any work on AGI will almost certainly lead to dangerous outcomes is to assert that any given AGI is likely to work perfectly well, subject to all restrictions except one that makes it hold (spatiotemporal scope boundaries). I'm unable to arrive at that conclusion as I believe that most AGI's will fail extensive self-improvement as that is where failure is most likely for that it is the largest and most complicated part of the AGI's design parameters. To put it bluntly, why is it more likely that contemporary AGI research will succeed at superhuman self-improvement (beyond learning), yet fail to limit the AGI, rather than vice versa? As I see it, it is more likely, given the larger amount of parameters to be able to self-improve in the first place, that most AGI research will result in incremental steps towards human-level intelligence rather than one huge step towards superhuman intelligence that fails on its scope boundary rather than self-improvement.

continue this thread »

Comment author: WrongBot 03 November 2010 07:30:09PM * 10 points [-]

One argument is that the only general intelligence we know, humans, would want to improve if they could tinker with their source code. But why is it so hard to make people learn then? Why don't we see much more people interested in how to change their mind?

Humans are (roughly) the stupidest possible general intelligences. If it were possible for even a slightly less intelligent species to have dominated the earth, they would have done so (and would now be debating AI development in a slightly less sophisticated way). We are so amazingly stupid we don't even know what our own preferences are! We (currently) can't improve or modify our hardware. We can modify our own software, but only to a very limited extent and within narrow constraints. Our entire cognitive architecture was built by piling barely-good-enough hacks on top of each other, with no foresight, no architecture, and no comments in the code.

And despite all that, we humans have reshaped the world to our whims, causing great devastation and wiping out many species that are only marginally dumber than we are. And no human who has ever lived has known their own utility function. That alone would make us massively more powerful optimizers; it's a standard feature for every AI. AIs have no physical, emotional, or social needs. They do not sleep, or rest, or get bored or distracted. On current hardware, they can perform more serial operations per second than a human by a factor of 10,000,000.

An AI that gets even a little bit smarter than a human will out-optimize us, recursive self-improvement or not. It will get whatever it has been programmed to want, and it will devote every possible resource it can acquire to doing so.

But in my opinion the difference here also implies that while nothing will distract them, there will also be no incentive not to hold. Why would it do more than necessary to reach a goal?

Clippy's cousin, Clip, is a paperclip satisficer. Clip has been programmed to create 100 paperclips. Unfortunately, the code for his utility function is approximately "ensure that there are 100 more paperclips in the universe than there were when I began running."

Soon, our solar system is replaced with n+100 paperclips surrounded by the most sophisticated defenses Clip can devise. Probes are sent out to destroy any entity that could ever have even the slightest chance of leading to the destruction of a single paperclip.

The further argument here is that it will misunderstand its goals. But the problem I see in this case is firstly that the more unspecific the goal the less it is able to measure its self-improvement against the goal to quantify the efficiency of its output.

The Hidden Complexity of Wishes and Failed Utopia #4-2 may be worth a look. The problem isn't a lack of specificity, because an AI without a well-defined goal function won't function. Rather, the danger is that the goal system we specify will have unintended consequences.

Secondly, the more vague a goal the larger has to be its general knowledge, previous to any self-improvement, to make sense of it in the first place? Shouldn't those problems outweigh each other to some extent?

Of course, if you told it to learn as much about the universe as possible, that is something completely different.

Acquiring information is useful for just about every goal. When there aren't bigger expected marginal gains elsewhere, information gathering is better than nothing. "Learn as much about the universe as possible" is another standard feature for expected utility maximizers.

And this is all before taking into account self-improvement, utility functions that are unstable under self-modification, and our dear friend FOOM.

TL;DR:

Agents that aren't made of meat will actually maximize utility.
Writing a utility function that actually says what you think it does is much harder than it looks.
Be afraid.

Comment author: XiXiDu 03 November 2010 08:17:51PM * 4 points [-]

Upvoted, thanks! Very concise and clearly put. This is so far the best scary reply I got in my opinion. It reminds me strongly of the resurrected vampires in Peter Watts novel Blindsight. They are depicted as natural human predators, a superhuman psychopathic Homo genus with minimal consciousness (more raw processing power instead) that can for example hold both aspects of a Necker cube in their heads at the same time. Humans resurrected them with a deficit that was supposed to make them controllable and dependent on their human masters. But of course that's like a mouse trying to hold a cat as pet. I think that novel shows more than any other literature how dangerous just a little more intelligence can be. It quickly becomes clear that humans are just like little Jewish girls facing a Waffen SS squadron believing they go away if they only close their eyes.

continue this thread »

Comment author: Nick_Tarleton 03 November 2010 09:46:05AM 1 point [-]

Again, I recommend The Basic AI Drives.

Comment author: XiXiDu 03 November 2010 11:03:04AM * 1 point [-]

To say that a system of any design is an “artificial intelligence”, we mean that it has goals which it tries to accomplish by acting in the world.

I cannot disagree with the paper based on that definition of what an “artificial intelligence” is. If you've all of this, goals, planning and foresight then you're already at the end of a very long and hard journey peppered with failures. I'm aware of the risks associated with such agents and support the SIAI, including donations. The intention of this thread was that I wanted to show that contemporary AGI research is much more likely to lead to other outcomes, not that there will be no danger if you already have an AGI with the ability for unbounded self-improvement. But I believe there are many AGI designs who fail this characteristic and therefore I concluded that it is more likely than not that it won't be a danger. I see now that my definition of AGI is considerable weaker than yours. So of course, if you take your definition what I said is not compelling. I believe that we'll arrive at your definition only after a long chain of previous weak AGI's who are impotent of considerable self-improvement and that once we figure out how to create the seed for this kind of potential we are also much more knowledgeable about associated risks and challenges such advanced AGI's might pose.

Comment author: Vladimir_Nesov 03 November 2010 11:20:07AM * 2 points [-]

Yes, and weak AGIs are dangerous in the same sense as Moore's law is: by probably making the construction of strong AGI a little bit closer, and thus a development contributing to the eventual existential risk, while being probably not directly dangerous in itself.

Comment author: XiXiDu 03 November 2010 11:39:28AM 0 points [-]

Yes, but each step into that direction does also provide insights into the nature of AI and therefore can help to design friendly AI. My idea was that such uncertainties are incorporated into any estimations of the dangers posed by contemporary AI research. How much does the increased understanding outweigh its dangers?

Comment author: Vladimir_Nesov 03 November 2010 11:58:11AM 5 points [-]

Yes, but each step into that direction does also provide insights into the nature of AI and therefore can help to design friendly AI.

This was my guess for the first 1.5 years or so. The problem is, FAI is necessarily a strong AGI, but if you learn how to build a strong AGI, you are in trouble. You don't want to have that knowledge around, unless you know where to get the goals from, and studying efficient AGIs doesn't help with that. The harm is greater than the benefit, and it's entirely plausible that one can succeed in building a strong AGI without getting the slightest clue about how to define Friendly goal, so it's not a given that there is any benefit whatsoever.

Comment author: Vladimir_Nesov 03 November 2010 09:23:28AM 0 points [-]

An abstract general intelligence simply does not care about values enough to take that pathway naturally. It will just do what it is told, not more.

Why do you believe that? What privileges "doing what it's told"?

Comment author: XiXiDu 03 November 2010 09:49:38AM * 1 point [-]

The question is not what privileges doing what it is told but why it would do what it is not told? A crude mechanical machine has almost no freedom, often it can only follow one pathway. An intelligent machine on the other hand has much freedom, it can follow infinitely many pathways. With freedom comes choice and the necessity to decide, to follow one pathway but not others. Here you assume that a general intelligence will follow a pathway of self-improvement. But I do not think that intelligence implies self-improvement and further that following a pathway that leads an intelligence to optimize will be taken without it being a explicit specified goal. And that is where I conclude that from a certain number of AGI projects not all will follow the pathway of unbounded, dangerous self-improvement as there are more pathways to follow which lead any given general intelligence to be impassive or hold.

If you've read the thread above you'll see that my incentive is not to propose that there is no serious risk but that it is not inevitable that any AGI will turn out to be an existential risk. I want to propose that working on AGI carefully can help us better understand and define friendliness. I propose that the risk to carefull work on AGI is justified and does not imply our demise in any case.

Comment author: Vladimir_Nesov 03 November 2010 09:53:06AM * 2 points [-]

The question is not what privileges doing what it is told but why it would do what it is not told?

Because planning consists in figuring out instrumental steps on your own.

Comment author: XiXiDu 03 November 2010 10:11:31AM 0 points [-]

If we are talking about a full-fledged general intelligence here (Skynet), there's no arguing against any risk. I believe all we disagree about are definitions. That there are risks from advanced real-world (fictional) nanotechnology is indisputable. I'm merely saying that what researchers are working on is nanotechnology with the potential to lead to grey goo scenarios but that there is no inherent risk that any work on it will lead down the same pathway.

It is incredible hard to come up with an intelligence that knows what planning conists of and to know and care to be able to judge what step is instrumental. This won't just happen accidently and will likely necessitate knowledge sufficient to be able to set scope boundaries as well. Again, this is not an argument that there is no risk but that it is not as strong as some people believe it to be.

Comment author: Vladimir_Nesov 03 November 2010 10:25:27AM 2 points [-]

If we are talking about a full-fledged general intelligence here (Skynet), there's no arguing against any risk. I believe all we disagree about are definitions. That there are risks from advanced real-world (fictional) nanotechnology is indisputable. I'm merely saying that what researchers are working on is nanotechnology with the potential to lead to grey goo scenarios but that there is no inherent risk that any work on it will lead down the same pathway.

Please keep focus, which is one of the most important tools. The above paragraph is unrelated to what I addressed in this conversation.

It is incredible hard to come up with an intelligence that knows what planning consists of and to know and care to be able to judge what step is instrumental. This won't just happen accidentally and will likely necessitate knowledge sufficient to be able to set scope boundaries as well.

Review the above paragraph: what you are saying is that AIs are hard to build. But of course chess AIs do plan, to give an example. They don't perform only the moves they are "told" to perform.

Comment author: XiXiDu 03 November 2010 10:51:13AM * 1 point [-]

What I am talking about is that full-fledged AGI is incredible hard to achieve and that therefore most of all AGI projects will fail on something other than limiting the AGI's scope. Therefore it is not likely that work on AGI is as dangerous as proposed.

That is, it is much more likely that any given chess AI will fail to beat a human player than that it will win. Still the researchers are working on chess AI's and the chess AI's will suit the definition of a general chess AI. Yet to get everything about a chess AI exactly right to beat any human but fail to implement certain performance boundaries (e.g. strength of its play or that it will overheat its CPU's etc.) is an unlikely outcome. It is more likely that it will be good at chess but not superhuman, that it will fail to improve, slow or biased than that it will succeed on all of the previous and additionally leave its scope boundaries.

So the discussion is about if the idea that any work on AGI is incredible dangerous is strong or if it can be weakened.

Comment author: jimrandomh 02 November 2010 08:39:40PM 3 points [-]

It doesn't know about any threat. You implicitly assume that it has something equivalent to fear, that it perceives threats.

It has the ability to model and to investigate hypothetical possibilities that might negatively impact the utility function it is optimizing. If it doesn't, it is far below human intelligence and is non-threatening for the same reason a narrow AI is non-threatening (but it isn't very useful either).

The difficulty of detecting these threats is spread out around the range of difficulties the AI is capable of handling, so it can infer that there are probably more threats which it could only detect if it were smarter. Therefore, making itself smarter will enable it to detect more threats and thereby increase utility.

Comment author: XiXiDu 03 November 2010 09:36:51AM * 1 point [-]

It has the ability to model and to investigate hypothetical possibilities that might negatively impact the utility function it is optimizing.

To be able to optimize it will have to know what it is supposed to optimize. You've to carefully specify what it output (utility function) is supposed to be or it won't be able to tell how good it is at optimizing. If you just tell it to produce paperclips, it won't be able to self-improve because it doesn't know how paperclips look like etc., therefore it cannot judge its own success or that extreme heat would be a negative impact giving paperclips made out of plastic. You further assume that it has a detailed incentive, that it is given a detailed pathway that it tells to look for threats and eliminate them.

If it doesn't, it is far below human intelligence and is non-threatening for the same reason a narrow AI is non-threatening (but it isn't very useful either).

If it doesn't it is what most researchers are working on, an intelligence with the potential to learn and make use of what it learnt, with the potential to become intelligent (educated). I'm getting the impression that people here assume that researchers are not working on an AGI but to hardcode a FOOM machine. If FOOM is simply part of your definition then there's no arguing against it going FOOM. But what researchers like Goertzel are working on are systems with the potential to reach human level intelligence, that does not mean that they will by definition jailbreak their nursery school. Although I never tried to argue against the possibility but that there are many pathways where this won't happen rather than the way it is portrayed by the SIAI, that any implementation of AGI will most likely consume humnanity.

Comment author: jimrandomh 03 November 2010 04:43:57PM 5 points [-]

The sorts of intelligences you are talking about are narrow AIs, not general intelligences. If you told a general intelligence to produce paperclips but it didn't know what a paperclip was, then its first subgoal would be to find out. The sort of mind that would give up on a minor obstacle like that wouldn't foom, but it wouldn't be much of an AGI either.

And yes, most researchers today are working on narrow AIs, not on AGI. That means they're less likely to successfully make a general intelligence, but it has no bearing on the question of what will happen if they do make one.

JGWeissman comments on What I would like the SIAI to publish - Less Wrong

Article Navigation

Comments (218)

Nearest Meetups

Virtual Study Room

Recent Comments

Recent Posts

Recent Rationality Quotes

Recent Wiki Edits

Top Contributors, 30 Days

Recent Karma Awards

Comments (218)

Top Contributors, 30 Days

You'll need to login or register to do that

(Don't worry, it only takes a few seconds)

Create

Login