I think it would be interesting if we weigh the benefits of human desire modification in all its forms (ranging from strategies like delayed gratification to brain pleasure centre stimulation: covered very well in this fun theory sequence article ) against the costs of continuous improvement.

Some of these costs :

• Resource exhaustion : There is always the risk of using up resources earlier for relatively unimportant things, and facing constraints for later, more important, purposes. This risk ends up materialising more often as we develop faster. Undoing material exhaustion is difficult, while energy is impossible.
• Environmental limits : Excessive global warming, pollution, etc. impose costs on humans
• Economic : Continuous uncoordinated development likely misallocates resources due to various market imperfections
• Social : Creating winners and losers is harmful to the happiness of people
• Psychological : If we cannot improve as fast as we adapt to the resulting happiness, then we get less happy

A lot of singularitarian thought tries to holds human desire to be exogenous and untouchable, which seems to be a rather odd blind-spot to have... we rightly discard the notion that death is desirable because it is natural, but not the notion that desire is sacred and hence should always be fulfilled, fighting against any and all limits?

Which is not something that the typical person, i.e. someone who barely understands the notion of "folders", can do. I despair at the thought of explaining that some of their data exists on the laptop and some of it on this box over there, let alone how and when to move data between them.

Like certain singulatarian futurists.

almost infinitely larger

Actually infinitely larger :-).

Then it will be just as "tractable" as linear models

Not necessarily. Some useful classes of models will not have the specific nice properties that linear models have.

When we define another subset of models suitable to the specific thing being modelled, then we will just as easily be able to come up with a set of explicit symbolic formulae.

Not necessarily. Closed-form solutions are not guaranteed to exist for your particular subset of models and, in fact, often do not, forcing you to use numeric methods with all the associated problems.

Why do you think "harmful for the environment" means "leading to global warming"? Lots of things are harmful for the environment. Drying swamps to make railroads harm it. Holidaying leads to decreased "old habitat" biodiversity. Building power plants on small mountain rivers leads to decreased biodiversity, too. Yes, these things are good for us. It just has no bearing on whether they are good for nature.

Wha...? Is that an argument by surface analogy? Does every increase in every value owing to human intervention lead to a catastrophe? How about internet connectivity? Land committed to agriculture? Air respired by humans? Shoes built? Radio waves transmitted?

How do you even measure the reference classes appropriately?

Although I don't have any references handy, I've seen people argue that Kyoto-like changes in our lifestyles are necessary on ethical grounds apart from global warming. More often they'll simply dismiss any sort of technological solution as a "quick fix" or even as the thing that caused the problem in the first place.

There are quite a few people who would like to abdicate control over the physical world.

Apologies for commenting almost a decade after most of the comments here, but this is the exact same reason why "using nonlinear models is harder but more realistic".

The way we were taught math led us to believe that linear models form this space of tractable math, and nonlinear models form this somewhat larger space of mostly intractable math. This is mostly right, but the space of nonlinear models is almost infinitely larger than that of linear models. And that is the reason linear models are mathematically tractable : they form such a small space of possible models. Of course nonlinear models don't have general formulae that always work : they're just defined as what is NOT linear. In other words, linear models are severely restricted in the form they can have. When we define another subset of models suitable to the specific thing being modelled, then we will just as easily be able to come up with a set of explicit symbolic formulae. Then it will be just as "tractable" as linear models, even though it's nonlinear : simply because it has different special properties belonging to its own class of models obeying something just like the law of linearity