I think this depends a whole lot on the domain/product, the scalability vs locality question (cookies get worse if more are made in the same place and then distributed, most software doesn't), and the network effect (software that depends on many people using the same thing).

I love my Kindle (and have since the ugly angular V1).  It would be very hard to argue that Amazon is particularly good at "one thing well", though.  Almost none of it's other products are that focused, and it's only because serious readers spend so much on books that they've kept the Kindle fairly pure.  

More generally, focusing on a single thing is HARD.  You need to find a thing that people are willing to pay for (either a few paying a lot or a lot paying a bit), and that you can make more cheaply and better than your competitors.  For most enterpreneurs, the path to that is to do a lot of things, then cut the ones which aren't that successful.  This is a SEARCH strategy, not a long-term octopus vision.  They're trying to find the one (or few very related) things that they can do well (and do well for themselves).   Many of them don't actually find it, so they either just get used to "do lots of stuff pretty badly", or give up.   

In theory, competition should counteract a lot of those incentives. Since software generally has low marginal costs, the ones with better functionality for passing users should get more market share, and investing in becoming/staying best will be rewarded.

For a lot of it, noise and short-term metrics overwhelm the quality drive, unfortunately. That’s likely because most software is too cheap (because many customers prefer inexpensive crap, so good things don’t get made).

[ I don't consider myself EA, nor a member of the EA community, though I'm largely compatible in my preferences ]

I'm not sure it matters what the majority thinks, only what marginal employees (those who can choose whether or not to work at OpenAI) think.  And what you think, if you are considering whether to apply, or whether to use their products and give them money/status.

Personally, I just took a job in a related company (working on applications, rather than core modeling), and I have zero concerns that I'm doing the wrong thing.

[ in response to request to elaborate: I'm not going to at this time.  It's not secret, nor is my identity generally, but I do prefer not to make it too easy for 'bots or searchers to tie my online and real-world lives together. ]

Most of these kinds of posts should start with Woody Allen's 1979 quote:

More than any other time in history, mankind faces a crossroads. One path leads to despair and utter hopelessness. The other, to total extinction. Let us pray we have the wisdom to choose correctly.

Agreed, but it's not just software.  It's every complex system, anything which requires detailed coordination of more than a few dozen humans and has efficiency pressure put upon it.  Software is the clearest example, because there's so much of it and it feels like it should be easy.

I think this leans a lot on "get evidence uniformly over the next 10 years" and "Brownian motion in 1% steps".  By conservation of expected evidence, I can't predict the mean direction of future evidence, but I can have some probabilities over distributions which add up to 0.  

For long-term aggregate predictions of event-or-not (those which will be resolved at least a few years away, with many causal paths possible), the most likely updates are a steady reduction as the resolution date gets closer, AND random fairly large positive updates as we learn of things which make the event more likely.

I kind of see what you're saying, but I also rather think you're talking about specifying very different things in a way that I don't think is required.  The closer CS definition of math's "define a sorted list" is "determine if a list is sorted".  I'd argue it's very close to equivalent to the math formality of whether a list is sorted.  You can argue about the complexity behind the abstraction (Math's foundations on set theory and symbols vs CS library and silicon foundations on memory storage and "list" indexing), but I don't think that's the point you're making.

When used for different things, they're very different in complexity.  When used for the same things, they can be pretty similar.

It's fascinating (and a little disturbing and kind of unhelpful in understanding) how much steering and context adjustment that's very difficult in older/smaller/weaker LLMs becomes irrelevant in bigger/newer ones.  Here's ChatGPT4:

You

Please just give 100 digits of e * sqrt(3)

ChatGPT

Sure, here you go:

8.2761913499119 7879730592420 6406252514600 7593422317117 2432426801966 6316550192623 9564252000874 9569403709858

"Mathematical descriptions" is a little ambiguous.  Equations and models are terse.  The mapping of such equations to human-level system expectations (anticipated conditional experiences) can require quite a bit of verbosity.  

I think that's what you're saying with the "algorithms and data structures" part, but I'm unsure if you're claiming that the property specification of the math is sufficient as a description, and comparable in fidelity to the algorithmic implementation.