This paragraph also misses the possibility of constructing a LLM and/or training methodology such that it will learn certain functions, or can't learn certain functions. There is also a conflation of "reliable" with "provable" on top of that.

Perhaps there is some provision made elsewhere in the text that addresses these objections. Nonetheless, I am not going to search. I found that the abstract smells enough like bullshit to do something else.

I'll try to make it clearer:

Suppose b "knows" that Omega runs this experiment for all programs b. Then the optimal behaviour for a competent b (by a ridiculously small margin) is to 1-box.

Suppose b suspects that box-choosing programs are slightly less likely to be run if they 1-box on equal inputs. Then the optimal behaviour for b is to 2-box, because the average extra payoff for 1-boxing on equal inputs is utterly insignificant while the average penalty for not being chosen to run is very much greater. Anything that affects probability of being run as box-chooser with probability greater than 1000/|P| (which is on the order of 1/10^10^10^10^100) matters far more than what the program actually does.

In the original Newcombe problem, you know that you are going to get money based on your decision. In this problem, a running program does not know this. It doesn't know whether it's a or b or both, and every method for selecting a box-chooser is a different problem with different optimal strategies.

As a function of M, |P| is very likely to be exponential and so it will take O(M) symbols to specify a member of P. Under many encodings, there isn't one that can even check whether the inputs are equal before running out of time.

That aside, why are you assuming that program b "wants" anything? Essentially all of P won't be programs that have any sort of "want". If it is a precondition of the problem that b is such a program, what selection procedure is assumed between those that do "want" money from this scenario? Note that being selected for running is also a precondition for getting any money at all, so this selection procedure is critically important - far more so than anything the program might output!

It's interesting (and perhaps a bit sad) that a relatively lengthy post on representing sentences as logical statements doesn't make any reference to the constructed language Lojban in which the entire grammar and semantics is designed around expressing sentences as logical statements.

Going into all the ways in which civilization - and its markets - fails to be rational seems way beyond the scope of a few comments. I will just say that GDP does absolutely fail to capture a huge range of value.

However, to address "share prices are set by the latest trade" you need to consider why a trade is made. In principle, prices are based on the value to the participants, somewhere between the value to the buyer and value to the seller. A seller who needs cash soon (to meet some other obligation or opportunity) may accept a lower price to attract a buyer more quickly. In our hypothetical and simplified one-trade-per-day scenario, that seller may accept up to 20% less than the previous day's trade price, though they find a buyer at only 5% less. So the company's market cap drops 5% even though 99.9999% of the investors and potential investors still value it exactly the same as yesterday.

This scales up since there are many highly correlated and often very short-term factors that influence desirability of shares vs cash vs bonds vs commodities vs ...etc. It's not just "what do I think this is worth to me", but also "what do I think that other people think that the market price will be tomorrow" and so on, and this can result in self-fulfilling predictions over surprisingly long time spans.

Could it be generalizing from T E X T   L I K E   T H I S  and/or mojibake UTF-16 interpreted as UTF-8 with every second character being zero? It's still a bit more of a stretch from there to generalize to ignoring two intervening constant characters, though.

Market cap is a marginal measure of desirability of shares in the entity represented. It mostly measures the expectations of the most flighty investors over short timescales. If a company issues a billion shares but only one of those is traded in any given day, the price of that single share agreed between the single seller and the single buyer entirely determines the market capitalization of that company.

In practice there is usually a lot more volume, but the principle remains. Almost all shares of any given entity are not traded over the timescales that determine share (or index) prices and hence market capitalization. In addition, market price has very little to do with the value of an entity's assets. Such assets may be instrumental in generating profits, but the relation is weak and very far from linear.

GDP, too, does not measure what you appear to think it measures.

I was very interested to see the section "Posts by AI Agents", as the first policy I've seen anywhere acknowledging that AI agents may be both capable of reading the content of policy terms and acting based on them.