I think it would be interesting if we weigh the benefits of human desire modification in all its forms (ranging from strategies like delayed gratification to brain pleasure centre stimulation: covered very well in this fun theory sequence article ) against the costs of continuous improvement.

Some of these costs :

• Resource exhaustion : There is always the risk of using up resources earlier for relatively unimportant things, and facing constraints for later, more important, purposes. This risk ends up materialising more often as we develop faster. Undoing material exhaustion is difficult, while energy is impossible.
• Environmental limits : Excessive global warming, pollution, etc. impose costs on humans
• Economic : Continuous uncoordinated development likely misallocates resources due to various market imperfections
• Social : Creating winners and losers is harmful to the happiness of people
• Psychological : If we cannot improve as fast as we adapt to the resulting happiness, then we get less happy

A lot of singularitarian thought tries to holds human desire to be exogenous and untouchable, which seems to be a rather odd blind-spot to have... we rightly discard the notion that death is desirable because it is natural, but not the notion that desire is sacred and hence should always be fulfilled, fighting against any and all limits?

Which is not something that the typical person, i.e. someone who barely understands the notion of "folders", can do. I despair at the thought of explaining that some of their data exists on the laptop and some of it on this box over there, let alone how and when to move data between them.

Like certain singulatarian futurists.

almost infinitely larger

Actually infinitely larger :-).

Then it will be just as "tractable" as linear models

Not necessarily. Some useful classes of models will not have the specific nice properties that linear models have.

When we define another subset of models suitable to the specific thing being modelled, then we will just as easily be able to come up with a set of explicit symbolic formulae.

Not necessarily. Closed-form solutions are not guaranteed to exist for your particular subset of models and, in fact, often do not, forcing you to use numeric methods with all the associated problems.

Why do you think "harmful for the environment" means "leading to global warming"? Lots of things are harmful for the environment. Drying swamps to make railroads harm it. Holidaying leads to decreased "old habitat" biodiversity. Building power plants on small mountain rivers leads to decreased biodiversity, too. Yes, these things are good for us. It just has no bearing on whether they are good for nature.

Wha...? Is that an argument by surface analogy? Does every increase in every value owing to human intervention lead to a catastrophe? How about internet connectivity? Land committed to agriculture? Air respired by humans? Shoes built? Radio waves transmitted?

How do you even measure the reference classes appropriately?

Although I don't have any references handy, I've seen people argue that Kyoto-like changes in our lifestyles are necessary on ethical grounds apart from global warming. More often they'll simply dismiss any sort of technological solution as a "quick fix" or even as the thing that caused the problem in the first place.

There are quite a few people who would like to abdicate control over the physical world.

Apologies for commenting almost a decade after most of the comments here, but this is the exact same reason why "using nonlinear models is harder but more realistic".

The way we were taught math led us to believe that linear models form this space of tractable math, and nonlinear models form this somewhat larger space of mostly intractable math. This is mostly right, but the space of nonlinear models is almost infinitely larger than that of linear models. And that is the reason linear models are mathematically tractable : they form such a small space of possible models. Of course nonlinear models don't have general formulae that always work : they're just defined as what is NOT linear. In other words, linear models are severely restricted in the form they can have. When we define another subset of models suitable to the specific thing being modelled, then we will just as easily be able to come up with a set of explicit symbolic formulae. Then it will be just as "tractable" as linear models, even though it's nonlinear : simply because it has different special properties belonging to its own class of models obeying something just like the law of linearity

The second point is something that really gets me. It seems to me that rather than feeling bad about donating to one charity rather than a more efficient or more "important" other charity, we should feel bad about spending money on frivolities rather than donating to charity. Nonprofit organizations are forced to compete against each other for slender resources in many ways, including donor dollars -- why can't they compete against things that have less moral value instead? It would be awesome if there were more social pressure to donate to charity rather than going to the movies or buying pretty clothes.

Interestingly, however, there is some social stigma against donating "too much". A few years ago, there was a New York gentleman who donated a much larger than "normal" percentage of his money to charity, as well as his kidney, plus some other stuff. (I'm sorry, I really wish I could remember his name, but I am very sure I have these details correct, because I read a lot about it at the time.) People speculated in the press about his mental status and other children mocked his kids at school, although his family was hardly left poor by the experience, and his health was not endangered.

In terms of the point in the OP about the lawyer who should be working overtime rather than volunteering ... I struggle with this so much. I spend most of my time doing activism, and I have friends who spend more time than I do (who do things like take very low-paying part-time jobs in order to finance spending most of their time doing activism), but most of us are sex-positive activists, and sex-positive activism is arguably an extremely "low priority" type of activism. If we are concerned about saving more lives, for example, then we should be dedicating our time to other types of activism, or we should be using our intelligence to get awesome jobs and then spending the money on charity. However, I (for one) have tried dedicating all my time to doing activism that seemed "more important" (HIV in Africa) rather than the activism that is most interesting to me (various types of sexuality stuff in America), and I was both less happy and less effective. I am also very sure that I would be unhappy if I dedicated my considerable IQ to becoming a corporate bitch and then donating lots of money, rather than working directly on the issues I care about.

Additionally, it is undeniable that someone has to work on the issues I care about, or else who would I donate money to even if I had a lot of it?