Stock prices represent the market's best guess at a stock's future time-discounted price. Stock prices go up even in the absence of technological advancement because stocks are tied to the bond market via arbitrage.
But that is a complicated abstract idea. Let's examine things in simple Marxist terms.
A business is a machine that extracts rents from the proletariat. A stock is partial ownership of a business. Traditionally, the business extracted money and then distributed that money to shareholders in the form of dividends. This is called a dividend stock. In principle, a dividend stock does not go up in value. You just extract rents until the Revolution.
However, publicly-traded companies often reinvest profits into themselves instead of distributing revenues to shareholders as dividends. This is called a growth stock. If a company reinvests profits into itself then the total value of the company goes up because additional capital has been invested into it. Since you own an undiluted share of the company, the value of your stock goes up in equal proportion to the growth of the company.
For growth stocks, why is the expected future growth not already priced in? If I know the company will be re-investing into future growth later, why not invest now?
There may be uncertainty, but if stocks on average trend upwards, doesn't it mean that the market continuously underestimates the amount that companies will re-invest?
Stock prices go up even in the absence of technological advancement because stocks are tied to the bond market via arbitrage.
If I'm understanding correctly, you're suggesting they should go up at least at the bond market nomi...
If I'm understanding correctly, you're suggesting they should go up at least at the bond market nominal rate…
Yes. It is necessary to disentangle two separate ideas. The first idea is time-discounting. Time-discounting plus arbitrage means that securities grow at no less than the bond market risk-free rate. The price of a stock right now doesn't reflected future growth. It represents future growth time-discounted by the bond market.
…but they tend to go up much faster than that?
The second idea is risk-adjustment. A 100% chance at $1 million is more valuable than a 1% chance at $100 million. If two securities have equal risk-neural expected value then the security with lower volatility has higher risk-adjusted expected value. If stocks tend to have higher volatility than bonds then an efficient risk-adjusted market ought to price stocks lower than bonds for a given expected (average) growth rate.
The value of capital invested in the bond market goes up via reverse time-discounting. The value of capital invested in the stock market goes up via reverse time-discounting and because you have purchased additional average growth by tolerating higher volatility.
The equity premium puzzle is still unsolved. The answer to your question is that nobody knows the answer. Stocks shouldn't have gone up historically, none of our current theories are capable of explaining why stocks did go up. Equivalently, stocks were massively underpriced over the last century or so and nobody knows why.
If you don't know why something was mispriced in the past, you should be very careful about asserting that it will or won't continue to be mispriced in the future.
The classic answer is risk. Stocks are riskier than bonds, so they should be underpriced (and therefore have higher returns) than bonds.
But we know how risky stocks have been, historically. We can calculate how much higher a return that level of risk should lead to, under plausible risk tolerances. The equity premium puzzle is that the observed returns on stocks is significantly higher than this.
Read through the wikipedia page on the equity premium puzzle. It's good.
Why shouldn't have stocks gone up historically? Isn't there more real wealth today than during the days of the East India Company? If a stock represents a piece of a businesses, and those businesses now have more real wealth today than 300 years ago, why shouldn't stock returns be quite positive? To be honest I'm bewildered by this perspective that stocks should never go up - I've never seen anyone in finance academically or professionally entertain this idea, so I'm surprised to hear you say "no one knows the answer," as if it's a common puzzle in the field. Why some stocks go up more than others is a good and open question, but that's one of relative valuation, not whether market returns on an absolute are greater than zero.
Suppose at some point there is an announcement that in ten years Free Hardware Foundation will release a magical nanofactory that turns dirt into most things currently in the basket of goods used to calculate inflation. There is no doubt about truth of the announcement. No company directly profits from the machine, as it's free (libre) hardware.
There's some upheaval in the market, that eventually settles down. Yet real value of stock is predictably going to sharply go up after ten years, not (just) immediately, as that's when the basket of goods actually becomes cheaper.
You may have to hold my hand on this one: I can agree the value of the stock (in the time-discounted future dividends sense) will go up after 10 years due to time-discounting -- the technology would enable value production that "comes into scope" as it gets closer in time.
But is there any reason other than time-discounting that the PRICE won't go up immediately? For instance, if I expect the time-discounted dividend value of the stock is $50 today and will be $5,000 ten years from now, and the rest of the market prices it at $50 today, then I could earn insane expected returns by investing at $50 today. Thus, I don't think the market would price it at $50 today.
One word : risk. There is uncertainty as to the company’s ability to provide that future revenue and as That uncertainty reduces, the stock price goes up.
Is this the same as positing "the market is continually surprised by the pace of technology"?
E.g., say I value company X's stock at $100. Then I learn a new fact that there's a 50% independent chance the company will discover a technology that doubles its value by 1 year from today. If I ignore all other factors, my estimate of the company's value 1 year from today should then be $150. If the company discovers the technology, I'll value it at $200, and if not, I'll value it at $100.
For the market to trend upward as it does, it seems like ...
Does this reduction come from seniority? Is the idea that older organizations are generally more reliable?
If I can summarize your question as something like "can I beat the returns on an index fund by only investing in companies with new/useful technologies", I think you'll find this question is similar to "which version of the EMH is true", and you'll also find a lot of good discussions about this, for example here: https://www.themoneyillusion.com/are-there-any-good-arguments-against-the-emh
For the two stories presented, I would say Story 1 is trivially true and Story 2 is probably false, although it's not phrased super well (for example, is "advances in technology" company specific or general economic growth?). Trying to read between the lines, it seems like you're wondering something like "if my choice to invest is between two companies, and Company 1 has current cash flow of $1-million and future cash flow of $1-million (no growth), and Company 2 has current cash flow of zero but future cash flow of $1-trillion (lots of growth), should I always invest in Company 2?" And my answer is "no, unless you think weak EMH is true and there's a specific inefficiency that you can uncover through your research" ( and even then, your research might determine you should sell rather than buy Company 2).
To improve the framework, I suggest the following distinctions/assumptions:
Using this framework, you could change your question to something like "assuming weak EMH is true, what sorts of public information about a company's new/useful technologies would allow me to value a company more accurately than the average investor". Then you could search for studies that try to answer this question or something similar.
The biggest reason stocks go up is pretty simple, in my view: a lot of very smart and hardworking people are working very hard to make stocks go up. In addition, lots of less smart and less hardworking people are also working to make stocks go up. In contrast, very few people are trying to make stocks go down.
While there are shortsellers, they are generally not trying to make stocks go down, i.e. by destroying value. Instead, shortsellers are simply saying that some companies are overvalued, and time and effort is better spent on other companies.
What do I mean by "make stocks go up"? I mean it in a fundamental sense: employees are trying to create value by making products better or adding new markets. Their rewards include stock compensation, bonuses, and promotions, so they are indeed incentivized to create value in whatever way possible. Meanwhile, financiers are trying to allocate capital (an abstract representation of time, effort, and value) in the best way, by optimizing society's effort. These people are also rewarded with capital gains, bonuses, and promotions.
Given that there is so much effort and incentive to fundamentally increase the value of stocks, why should the market stay flat or neutral? I'm sure someone can try to find some elaborate corner case why this isn't true, but mostly the reasons stocks wouldn't go up is if no one is trying (no incentive) or if someone is trying to destroy asset value (expropriation, violent revolution, etc.). If people are trying to build stuff that makes stock prices go up, and there isn't much effort to destroy them, it's pretty straightforward that they go up.
For what it's worth, this is basically Buffett's view. Also note, the comments about equity risk premium, valuation puzzles, and the efficient market hypothesis miss the point: that's only a matter of relative returns, not absolute returns. (If your question is about beating the market, then those points are more relevant, but it's not clear anyone can beat the market today in developed economies outside narrow pockets of inefficiency.) As long as you can build new stuff, you'll have a reason to invest, which demands a return, hence the positive returns we see. If there's less "stuff" to build, you can expect lower prospective returns because capital competes for each opportunity, resulting in lower rates and a huge premium on growth opportunities, which is what we observe today.
One last way to think about it is in real terms. If you have a machine that can build more of itself, you will have more of those machines over time, which is a real return.
"Stock prices represent the market's best guess at a stock's future price."
But they are not the same as the market's best guess at its future price. If you have a raffle ticket that will, 100% for definite, win $100 when the raffle happens in 10 years time, the the market's best guess of its future price is $100, but nobody is going to buy it for $100, because $100 now is better than $100 in 10 years.
Whatever it is that people think the stock will be worth in the future, they will pay less than that for it now. (Because $100 in the future isn't as good as just having the money now). So even if it was a cosmic law of the universe that all companies become more productive over time, and everyone knew this to be true, the stocks in those companies would still go up over time, like the raffle ticket approaching the pay day.
Toy example:
1990 - Stocks in C cost $10. Everyone thinks they will be worth $20 by the year 2000, but 10 years is a reasonably long time to wait to double your money so these two things (the expectation of 20 in the future, and the reality of 10 now) coexist without contradiction.
2000 - Stocks in C now cost $20, as expected. People now think that by 2010 they will be worth $40.
Don't forget the Greater Fool theory (https://www.investopedia.com/terms/g/greaterfooltheory.asp). Stocks go up because investors expect them to go up, and those investors' purchases actually drive the price up.
There is some underlying truth in terms of dividends and stock buybacks (effectively a dividend, but paid by reverse-dilution), but the vast (VAST) majority of invested dollars are fully disconnected from any actual business outcomes, except by the publicity and expectations channel.
I think things become simpler when you look at the sum of all stocks, versus particular ones. Then, you only need to consider the market cap of the entire stock market, and what makes it change over time.
The economy is much bigger than the stock market. Money flows from small companies to larger one as the economy consolidates -- since the former are more likely to be publicly traded than the latter, that makes the market become bigger.
It's easier to invest in the stock market now than in the past. Since it's accessible to more people, then more people's money can be put in it. So, the market cap goes up.
Finally, as inequality increases, more of a fraction of wealth is disposable, and therefore can be invested. That makes the market grow as well.
I'm sure there are many other reasons along these lines.
Here's some motions toward an answer. I'll consider an informally specified market model, as opposed to a real market. Whether my reasoning applies in real life depends on how much real life resembles the model.
In particular, consider as model an efficient market. Assume the price of any stock X is precisely its expected utility according to all evidence available to the market. Then the only way for the price to go up is if new evidence arrives. This evidence could be the observation that the company associated with the stock continues to exist and produce income, that it continues to produce income, or that the income it produces is is going up.
Those bits of evidence remind me of the risk perspective. Sure, investors may believe that a company, if it continues to exist, will one day be worth more money than they could ever invest today. But if they think there's a 5% chance each year that the company stops existing, then this can severely limit the expected utility of owning the stock right now. (You might ask, "What if I assign a nonzero probability to the class of futures where the company exists forever and the utility of holding its stock grows without bound?" which makes the expected utility infinite, and makes me suspect that defining utility over infinite spans of time is tricky.)
I feel like the time-discounting hypothesis makes a lot of sense and is probably part of the truth. To make sense of it within my toy model, I'd have to look at what time-discounting actually means in terms of utility. A reasonable assumption within this idealized model seems to be that the "utility" of anything you possess is equal to the maximum expected utility of anything you could do with it / exchange it for, including exchanges over time. (This is like assuming perfect knowledge of everything your could do with your possessions. Utility can never go up under this assumption, similar to how no legal move can improve a chess position in the eyes of a perfect player.) This means the utility of owning a stock is somewhere between the utility of its buy price and its sell price, as expected. And the utility of 1 dollar right now is no less than the expected utility of a dollar in 2030, given that you could just hold on to it. Then time-discounting is just the fact that the utility of 1 dollar right now is no less than the expected utility of buying one dollar's worth of stock X and waiting a couple years. Let's assume the stocks grow exponentially in price. That is to say, though neither the stocks nor the money increases in utility, stocks can be exchanged for more and more money over time. It seems converting money into stocks avoids futures where we lose utility. So how much should we pay for one stock? This is just determined by the ratio of the utility of the stock to the utility of a dollar. So the question becomes, why does money have any utility at all, if it is expected to fall in utility compared to stocks? And this must be because it can be exchanged for something of intrinsic utility, such as the enjoyment derived from eating a pizza. But why would someone sell you a pizza, knowing your money will decrease in utility? Because the pizza will decrease in utility even faster unless someone eats it, and the seller has too many pizzas to eat, or wants to buy other food for themself. (Plus, the money is backed by banks / governments / other systems.)
So if the average price of stocks tends to increase year by year, why are they not worth infinite money to begin with? Here's another perspective. How is the average price calculated? Presumably we're only averaging over stocks from active companies, ones that have not ceased to exist due to bankruptcy or the like. However, when evaluating the expected utility of a stock, we are averaging over all possibilities, including the possibilities where the company goes bankrupt. There is a nonzero chance that the company of a stock will cease to exist (unpredictably). So if you Kelly bet, you should not invest all your money into such a stock. As a consequence, successively lower prices are required to get you to buy each additional stock of the company.
Reflection: I imagine these concepts match with ideas from economic theory in quite a few places. I have a mathematical background myself, probably making the phrasing of this answer unusual. I'm not so sure about this whole informal, under-specified model I just made. It seems like the kind of thing that easily leads to pseudoscience, while at the same time playing around with inexact rules seems useful in early stages of getting less confused, as a sort of intuition pump. (Making the rules super strict immediately could get you stuck.)
I don't know the full answer to why stocks go up, but I have a partial answer based on risk. Imagine there are only 2 products available in the market:
(1) A US government bond that pays $100 in 1 year.
(2) Ownership in a company that will dissolve in 1 year, and at the end either return to the owner $95 or $105 with equal probability.
Note that both have the same expected return in 1 year, $100. But people will prefer to buy the first product compared to the second one, since the first one is risk free. Say the current 1-year interest rate for risk free returns is 1%, then people will pay $99 for the first product. But since the second product is less desirable, they might only pay $98 for it. So the second product has greater expected profit, since you paid $98 for $100 of expected returns. If you only invest in products like (2), then in the long run you'll make more money in expectation compared to investing in risk-free assets like (1).
Lots of interesting answers, and all of them correct (most of the time anyway). One I haven't seen mentioned, is the one described in this preprint titled "How to Increase Global Wealth Inequality for Fun and Profit".
In short:
Here are two stories about stocks that I find hard to reconcile:
But if stock prices tend to go up due to technology, why isn't that already priced in?
This seems relevant to investment strategy:
As an example: if you think AI is going to accomplish big thing X, and you think the market already knows this, should you buy and hold relevant AI company stock? Or would you expect the anticipated growth to already be priced in?
Some potential answers I can think of:
Question: What are good ways to think about this? What evidence do we have?
EDIT - Summary of things I got from answers/comments: