Some ideas...
Autonomous Management/Governance
The main function of managers of all stripes (according to me) is to solve coordination problems. However, there are also “autonomous” systems for solving some specific kinds of coordination problems - markets, in particular, are a major example. But these are limited to problems which fit their interface well - i.e. a high degree of standardization, in the transactions if not the products themselves. A requirement for large amounts of context, or difficulty recognizing success even in hindsight creates problems for markets.
I could imagine systems which solve (some) more context-heavy or alignment-limited coordination problems, with the scalability and decentralized “autonomy” of markets. Management/governance via prediction markets is one class of ideas in this vein, but it seems like modern information technology should open up many other possibilities. The way I picture it, it's not so much "build an AI to run a company" as "organize the company so its constituent people together implement the same algorithm the AI would". Just like competitive markets perform a variant of backprop and gradient descent, other AI algorithms should have equivalent structures in human organization.
Information Indexing
In recent years, availability of information/data has grown much faster than our ability to index it. Matching particular people with the highest-value information for them - i.e. indexing the information - is largely a social technology problem, though one which benefits a lot from computer science.
Suppose I'm researching mutations of mitochondrial DNA. Perhaps the topic has been studied by a particular group separate from the main field, with different aims and entirely different jargon. A text search for the normal jargon won't turn up their work, no matter how helpful and relevant it might be. In order to find something like that, we need some kind of jargon-independent indexing. For instance, if we had a detailed world-model, then we could link the information directly to structures in the world-model (i.e. modelled mitochondria). If we had extensive augmented reality, we use object recognition to recognize mitochondria visually, and then index information to that - so that even research using a different word for "mitochondria" would pop up when looking at a picture of a mitochondrion.
More generally, it would be useful to be able to index information based on the real-world patterns or mathematical structures they involve. This seems like something which could, in principle, be implemented purely socially, if we knew a good way to do it. (I'm sure people have tried things like this before, but usually projects like these try to use data structures which are convenient for programming, rather than data structures which represent the world well.)
More Efficient Notation
People joke that Einstein's greatest invention was tensor index notation. Good mathematical notation makes a surprisingly large difference, as anyone who's used both assembly language and python can attest.
It seems to me that the main trend in mathematical notation from ~1850 to ~1950 was to introduce really bad notation for things which previously had no formal notation at all. (This is part of the more general trend of introducing not-very-good foundations for things which previously had no foundations. Set theory and Turing machines and the like were better than nothing, but they are more like assembly than Lisp.) I expect in the future, progress will come more from improving notation, as a side effect of the general project of refactoring mathematical foundations.
Continuing improvement in programming is, of course, a similar story. In that department, I think progress will come from better understanding the structure of the world around us, and incorporating those insights into programming languages (abstraction is one potential example).
Model-Based Financial Markets
Similar to Abram's suggestions for model-based prediction markets, as automated trading gains share of the financial markets, there's potential for financial systems which take advantage of the legibility of automated models.
In particular: modern markets direct capital to existing projects. Traders probably have some views about what projects would be really good investments, but the market does not have an easy mechanism to share those views with business founders/managers during the planning phase. Instead, it's guess and check: a founder/manager tries a project, and then finds out how much the market likes that project. Part of the problem here is that project-space is very high dimensional, so it's not easy to compress all the information from the traders into a low-dimensional price signal.
But with transparent automated traders, we could in principle use a backprop-like mechanism to identify projects which the traders think are promising, without the slow guess-and-check process.
Multidimensional pricing
The benefit of a single scalar price is that it is super easy to compute, and lets you compare different types of value in a like-with-like fashion. The downside is that in order to compare different types of value, you have to sacrifice all of the information about it; there are probably arguments where this is considered a feature between two simple agents, but in meatspace there is a large problem with organizations not having good internal knowledge which means prices conceal information from yourself just as much. Secondly there is the eternal matter of externalities, which by long standing tradition are impossible to address outside of regulation. This is because we have no good way of assigning value to things not happening.
I think a multidimensional pricing model would help with this, because it would allow all the elements of value to be evaluated independently, including negative prices for things to be avoided. Then the whole thing could be computed as a single eigenprice, giving us the same overall benefit as regular dollars, but keeping all the relevant information. We are very good at automatic computations of this type now, so it seems feasible as long as something as sophisticated as a smartphone or engineering calculator is available. There is no reason a bank or transaction platform could not have this built in.
Generic strategy models
I've never been able to let go of Jaynes' notion of using the phase-space of macrophenomena for prediction, which gives me a strong feeling that there is a fully general and scalable method of generating strategies. If we think of strategies from an action perspective as being a combination of predictions+interventions, it seems like any desirable end-state could be approached.
This should help a million and one things, because at least the United States is pretty strongly anti-strategic. If it became more consistent, cheaper, and effective, then I expect adoption to go up and overall efficiency as a result.
Full spectrum group alignment methods
At some point in the relatively near future we will start putting together the pieces that allow the entire spectrum of human experience to be brought to bear in building group alignment. This will include tools traditionally shunned in developed nations, like violence and fear. Of the three ideas I suggest, this one feels the most inevitable and simultaneously the most double-edged; most of the legwork to date has been done by professional militaries and religions, but there is plenty of interest in investigating things like trauma, abuse, and political movements gone haywire. As far as I can tell consent is really just one more variable in this space, because otherwise conscription would not be viable as a military method, and people seldom set out with political or religious radicalization as their goal. As a consequence, once these methods are well established I expect them to work on whoever happens to be within reach, rather than requiring voluntary association.
Me either. I am lining up a crack at building gears for number 3, and the technique for number 2. We’ll see what can be made of it.