I'm surprised that no one has addressed Karnofsky's main point: even if we accept all of the premises on which SIAI was founded, it doesn't look like SIAI is doing anything particularly impressive (from an Outside View). Jasen's reply seemed to consist mostly of "we're working on it." What does the community think about this?
I was also rather disappointed that SIAI doesn't have a definitive plan for what they would do with more money. This might end up being a self-fulfilling prophecy that works against them, because GiveWell may decrease SIAI's rating based on the fact that SIAI apparently haven't planned out how they would scale up their operation. (But I think this is definitely a smaller concern than the previous one.)
That's been one of the reasons I've not donated either (apart from one £10 donation made for odd reasons). Unfortunately, the SIAI seem to have made a decision that they need to keep what work they're doing secret. That's understandable, but it means we have absolutely no idea what the actual effect of donation is.
Holden Karnofsky of GiveWell.org interviewed Jasen Murray of SIAI and published his notes (Edit: PDF, thanks lukeprog!), with updates from later conversations. Lots of stuff to take an interest in there - thanks to jsalvatier for drawing our attention to it. One new bit of information stands out in particular: