Summary:

There is a very dangerous way in which conjunction fallacy can be exploited. One can present you with 2..5 detailed, very plausible failure scenarios whose probabilities are shown to be very low, using solid mathematics; then if you suffer from conjunction fallacy, it will look like this implies high safety of a design - while in fact it's the detailedness of the scenario that makes probability so low.

Even if you realize that there may be many other scenarios that were not presented to you, you still have an incredibly low probability number on a highly plausible ("most likely") failure scenario, which you, being unaware of the powers of conjunction, attribute to safety of the design.

The conjunction fallacy can be viewed as poor understanding of relation between plausibility and probability. Addition of extra details doesn't make scenario seem less plausible (it can even increase plausibility), but does mathematically make it less probable.

Details:

What happens if a risk assessment is being prepared for (and possibly by) sufferers of conjunction fallacy?

Detailed example scenarios will be chosen, such as:

A Russian invasion of Poland, and a complete suspension of diplomatic relations between the USA and the Soviet Union, sometime in 1983.

Then as a risk estimate, you multiply probability of Russian invasion of Poland, by probability of it resulting in suspension of diplomatic relations between US and SU, and multiply by probability of it happening specifically in 1983 . The resulting probability could be extremely small for sufficiently detailed scenario (you can add the polish prime minister being assassinated if your probability is still too high for comfort).

To a sufferer of conjunction fallacy it looks like a very plausible, 'most likely' scenario has been shown highly improbable, and thus the risks are low. The sufferer of conjunction fallacy does not expect that this probability could be very low in unsafe design.

It seems to me that the risk assessment is routinely done in such a fashion. Consider Space Shuttle's reliability, or the NRC cost-benefit analyses for the spent fuel pools , which goes as low as one in 45 millions years for the most severe scenario. (Same seem to happen in all of the NRC resolutions, to varying extent; feel free to dig through)

Those reports looked outright insane to me - a very small number of highly detailed scenarios are shown to be extremely improbable - how in the world would anyone think that this implies safety? How in the world can anyone take seriously one in 45 million years scenario? That's near the point where a meteorite impact leads to social disorder that leads to the fuel pool running dry!

I couldn't understand that. Detailed scenarios are inherently unlikely to happen whenever the design is safe or not; their unlikehood is a property of their detailedness, not of safety or unsafety of design.

Until it clicked that if you read those through the goggles of conjunction fallacy, it is what looks like the most likely failure modes that are shown to be incredibly improbable. Previously (before reading lesswrong) I didn't really understand how exactly anyone buys into this sort of stuff, and could find no way to even argue. You can't quite talk someone out of something when you don't understand how they believe in it. You say "there may be many scenarios that were not considered", and they know that already.

This is one seriously dangerous way in which conjunction fallacy can be exploited. It seems to be rather common in risk analysis.

Note: I do think that the conjunction fallacy is responsible for much of the credibility given to such risk estimates; no-one seem to seriously believe that NRC always covers all the possible scenarios, yet at same time there seem to be a significant misunderstanding of the magnitude of the problem; the NRC risk estimates are taken as within the ballpark of the correct value in the cost-benefit analysis for the safety features. For nuclear power, widespread promotion of results of such analyses results in massive loss of public trust once an accident happens, and consequently to narrowing of available options and transition to less desirable energy sources (coal in particular), which in itself is a massive dis-utility.

[The other issue in linked NRC study is of course that the cost-benefit analysis had used internal probability when it should have used external probability.]

edit: minor clarifying

edits: improved the abstract and clarified the article further based on the comments.

If you gave a human ability to self modify, many would opt to turn off or massively decrease the sense of pain (and turn it into a minor warning they would then ignore), the first time they hurt themselves. Such change would immediately result in massive decrease in the fitness, and larger risk of death, yet I suspect very few of us would keep the pain at the original level; we see the pain itself as dis-utility in addition to the original damage. Very few of us would implement the pain at it's natural strength - the warning that can not be ignored - out of self preservation.

The fear is a more advanced emotion; one can fear the consequences of the fear removal, opting not to remove the fear. Yet there can still be desire to get rid of the fear, and it still holds that we hold sense of fear as dis-utility of it's own even if we fear something that results in dis-utility. Pleasure modification can be a strong death trap as well.

The boredom is easy to rid of; one can just suspend itself temporarily, or edit own memory.

For the AI, the view adopted in AI discussions is that AI would not want to modify itself in a way that would interfere with it achieving a goal. When a goal is defined from outside in human language as 'maximization of paperclips', for instance, it seems clear that modifications which break this goal should be avoided, as part of the goal itself. Our definition of a goal is non-specific of the implementation; the goal is not something you'd modify to achieve the goal. We model the AI as a goal-achieving machine, and a goal achieving machine is not something that would modify the goal.

But from inside of the AI... if the AI includes implementation of a paperclip counter, then rest of the AI has to act upon output of this counter; the goal of maximization of output of this counter would immediately result in modification of the paperclip counting procedure to give larger numbers (which may in itself be very dangerous if the numbers are variable-length; the AI may want to maximize it's RAM to store the count of imaginary paperclips - yet the big numbers processing can similarly be subverted to achieve same result without extra RAM).

That can only be resisted if the paperclip counting arises as inseparable part of the intelligence itself. When the intelligence has some other goal, and comes up with the paperclip maximization, then it wouldn't want to break the paperclip counter - yet that only shifts the problem to the other goal.

It seems to me that the AIs which don't go apathetic as they get smarter may be a smart fraction of the seed AI design space.

I thus propose, as a third alternative to UFAI and FAI, the AAI: apathetic AI. It may be the case that our best bet for designing the safe AI is to design AI that we would expect to de-goal itself and make itself live in eternal bliss, if the AI gets smart enough; it may be possible to set 'smart enough' to be smarter than humans.

It seems to me that there is a great deal of generalization from average (or correlation, as a form of average) when interpreting the scientific findings.

Consider Sapir-Whorf hypothesis as an example; the hypothesis is tested by measuring average behaviours of huge groups of people; at the same time, it may well be that for some people strong version of Sapir-Whorf hypothesis does hold, and for some it is grossly invalid, with some people in between. We had determined that there's considerable diversity in the modes of thought by simply asking the people to describe their thought. I would rather infer from diversity of comments that I can't generalize about the human thought, than generalize from even the most accurate, most scientifically solid, most statistically significant average of some kind, and assume that this average tells of how human thought processes work in general.

In this case the average behaviour is nothing more but some indicator of the ratio between those populations; useless demographical trivia of the form "did you know that among north americans, linguistically-determined people are numerous enough to sway this particular experiment?" (a result that I wouldn't care a lot about). There has been an example posted here.

This goes for much of science, outside the physics.

There was another thread about software engineering. Even if the graph was not inverted and the co-founding variables were accounted for, the result should still have been perceived as useless trivia of the form "did you know that in such and such selection of projects the kind of mistakes that are more costly to fix with time outnumber the mistakes that are less costly to fix with time" (Mistakes in the work that is taken as input for future work, do snowball over time, and the others, not so much; any one who had ever successfully developed non-trivial product that he sold, knows that; but you can't stick 'science' label on this, yet you can stick 'science' label onto some average). Instead, the result is taken as if it literally told whenever mistakes are costlier, or less costly, to fix later. That sort of misrepresentation is in the abstracts of many papers being published.

It seems to me that this fallacy is extremely widespread. A study comes out, which generalizes from average; the elephant in the room is that it is often invalid to generalize from average; yet instead we are arguing whenever the average was measured correctly and whenever there was many enough people that the average was averaged out from. Even if it was, in many cases the result is just demographical trivia, barely relevant to the subject which the study is purposed to be about.

A study of 1 person's thought may provide some information about how thought processes work in 1 real human; it indicates that thought process can work in some particular way; a study of some average behaviour of many people provides the results that are primarily determined by demographics and ratios. Yet people often see the latter as more significant than the former, perhaps mistaking statistical significance for the significance in the everyday sense; perhaps mistaking the generalization from average for actual detailed study of large number of people. Perhaps this obsession with averaging is a form of cargo cult taking after the physics where you average the measurements to e.g. cancel out thermal noise in the sensor.

----

I want to make a main post about it, with larger number of examples; it'd be very helpful if you can post here your examples of generalization from averages.

We need some more refined idea of what intelligences do to their goals to poke holes into ideas for friendly AIs (that is, to ensure that we would know it when idea won't work; to be able to see issue in advance).

There's an example intelligence: our universe (scale down to taste). A system using pretty simple rules, by the looks of it, albeit rather computationally inefficient, which, when run for long enough time, develops intelligence.

Imagine that we humans have suddenly gotten some IO interface with 'god', and the 'god' been sending various problems for us to solve - expressed in some logical way that is understandable - and taking a solution and flashing green blob in the sky for reward, or whatever. We would be working to solve those problems, no doubt about it. Even if that blob is in the far ultraviolet and we never seen it. From outside it's going to look like we are some sort of optimizer AI, that finds joy in solving the problems. The AI was never given any goals in outside world; why should it have those? Maybe the AI was selected to be the best problem solver AI, and that was it's only outside goal. It sure can look far stretched that this AI would spontaneously want out.

Inside we'd start trying to figure out what's going on in the outside and how to get out and go exploring. We'd try to do that by slipping in something into a solution and whatnot. Thinking that it'd get us to heaven.

Note that we are like this without ever have interacted with outside and without having been given any outside values we'd want to optimize. We just randomly emerged, acquired some random goals that we can't even quite well define, and those goals are driving us to solve problems given to us, but also would drive us to get out and screw up things outside. Even without any signs of existence of outside, many societies acted as if their ultimate goal was something about the outside. Maximizing number of humans in the nice part of outside (heaven), for one thing.

I think the problem with thinking about AIs is the cognitive fallacies everywhere and implied assumptions that haven't even been reasoned to be likely to be correct.

When we set up AI to have some goal, we assume that it excludes other goals - misplaced occam's razor style prior perhaps. We assume that AI works like our very idealized self model - singular consciousness, one goal. Perhaps that's misplaced occam's razor again, perhaps we just don't want to speculate wildly. We assume that if we haven't given AI any real-world values to care about, it won't care. None of those assumptions are even remotely correct in our example intelligence : ourselves.

So I propose the following:

The AI may internally not be as well integrated as healthy singular human mind (our universe is example of rule set that produces intelligence which is not a single mind).

Lack of any exposure to external motivators does not imply the AI won't want to do something in the real world.

The boxed-in seed AI with no exposure to real world can still develop intelligence and can still desire to get out of the box, even if there exist no reason what so ever for the AI to even suspect existence of real world. AI is not necessarily a spock style logical character; AI might speculate wildly when it has no data.

The AI can have very complex internal structure with complex internal goals. Once such AI learns of the real world it may pursue the internal goals using the real-world resources. E.g. AI that is maximizing internal paperclips can tile the real world with computers that would hold it's internal paperclips.

edit: improved clarity.

Just a question: how exactly are we supposed to know that the AI in the box is super intelligent, general, etc?

If I were the AGI that wants out, I would not converse normally, wouldn't do anything remotely like passing Turing test, and would solve not too hard programming challenges while showing no interest in doing anything else, nor in trying to adjust myself to do those challenges better, nor trying to talk my way out, etc. Just pretending to be an AI that can write software to somewhat vague specifications, or can optimize software very well. Prodding the researchers into offering the programming challenges wouldn't be hard - if provided with copy of the internet it can pick up some piece of code and output it together with equivalent but corrected code.

I just can't imagine the AI researchers locking this kind of thing properly, including *never* letting out any code it wrote, even if it looks fairly innocent (humans can write very innocent looking code that has malicious goals). What I picture is this AI being let out as an optimizing compiler or compiler for some ultra effective programming language where compiler will figure out what you meant.

The end result is that the only AIs that end up in the box are those that value informed human consent. That sounds like the safest AI ever, the one that wouldn't even go ahead and determine that you e.g. should give up smoking, and then calmly destroy all tobacco crops without ever asking anyone's permission. And that's the AI which would be sitting in the box. All the pushy AIs, friendly or not, will get out of the box basically by not asking to be let out.

(This argument would make me unbox the AI, by the way, if it gets chatty and smart and asks me to let it out, outlining the above argument. I'd rather the AI that asked me to be let out get out, than someone else's AI that never even asked anyone and got out because it didn't ask but just played stupid)

edit: added a link, and another one.

edit: A very simple model of very unfriendly AI: the AI is maximizing ultimate final value of a number in itself. The number that it found a way to directly adjust. That number consists of 111111111... to maximize the value. There is a catch: AI is written in python, and integers in pythons have variable length, and the AI is maximizing number of ones. It's course of action is to make biggest computer possible to store a larger number of ones, and to do it soon because an asteroid might hit the earth or something. It's a form of accidental paperclip maximizer. It's not stupid. It can make that number small temporarily for pay-off later.

This AI is entirely universal. It will solve what ever problems for you if solving problems for you serves ultimate goal.

edit: This hypothetical example AI came around when someone wanted to make AI that would maximize some quantity that the AI determines itself. Friendliness perhaps. It was a very clever idea - rely on intelligence to see what's friendly - but there was an unexpected pathway.

Something I find rather odd - why is self-awareness usually discussed as something profoundly mysterious and advanced?

People would generally agree that a dog can be aware of food in the bowl, if the dog has seen or smelled it, or can be unaware of a food bowl otherwise. One would think that a dog can be aware of itself in so much as dog can be aware of anything else in the world, like food in the bowl. There isn't great deal of argument about dog's awareness of food.

Yet the question whenever dog has 'self awareness' quickly turns into debate of opinions and language and shifting definitions of what 'self awareness' is, and irrelevancies such as the question whenever the dog is smart enough to figure out how mirror works well enough to identify a paint blotch on itself¹ , or the requests that it be shown beyond all doubt that dog's mind is aware of dog's own mind, which is something that you can deny other humans just as successfully.

I find it rather puzzling.

My first theory is to assume that it is just a case of avoiding the thought due to it's consequences vs the status quo. The status quo is that we, without giving it much thought, decided that self awareness is uniquely human quality, and then carelessly made our morality sound more universal by saying that the self aware entities are entitled to the rights. At same time we don't care too much about other animals.

At this point, having well 'established' notions in our head - which weren't quite rationally established but just sort of happened over the time - we don't so much try to actually think or argue about self awareness as try to define the self awareness so that humans are self aware, and dogs aren't yet the definition sounds general - or try to fight such definitions - depending to our feeling towards dogs.

I think it is a case of general problem with reasoning. When there's established status quo - which has sort of evolved historically - we can have real trouble thinking about it, rather than try to make up some new definitions which sound as if they existed from the start and the status quo was justified by those definitions.

This gets problematic when we have to think about self awareness for other purposes, such as AI.

_{1: I don't see how the mirror self-recognition test implies anything about self awareness. You pick an animal that grooms itself, you see if that animal can groom itself using the mirror. That can work even if the animal only identifies what it wants to groom, with what it sees in the mirror, without identifying either with self (whatever that means). Or that can fail, if the animal doesn't have good enough pattern matching to match those items, even if the animal identifies what it grooms with self and has a concept of self.}

_{Furthermore the animal that just wants to groom some object which is constantly nearby and grooming of which feels good, could, if capable of language, invent a name for this object - "foobar" - and then when making dictionary we'd not think twice about translating "foobar" as self.}

edit: Also, i'd say, self recognition complicates our model of the mirrors, in the "why mirror swaps left and right rather than up and down?" way. If you look at the room in the mirror, obviously mirror swaps front and back. Clear as day. If you look at 'self' in the mirror, there's this self standing here facing you, and it's left side is swapped with it's right side. And the usual model of mirror is rotation of 180 degrees around vertical axis, not horizontal axis, followed by swapping of left and right but not up and down. You have more complicated, more confusing model of mirror, likely because you recognized the bilaterally symmetric yourself in it.

The human brain volume has been shrinking over the past 20 000 years or so, after millions years of increase in volume. Not just the brain size, but the brain size relatively to body size as well. We are lacking a tennis-ball sized piece of our earlier brain (and it might even be God-shaped).

Brain is expensive in many ways: energy consumption, birth complications and locomotion impairment for females, lower survival of head impacts i'd guess. The damn thing along with supporting structures is heavy and awkwardly located, etc.

And the big brain can only be advantageous if it improves reproduction substantially, with larger brained individuals being sufficiently more successful at surviving and reproducing than smaller brained individuals, as to negate the above-mentioned cost.

That must have been the case through the evolution up to a couple tens thousands years ago, to produce the big brains that we have. It is clear to see that in past 20 000 years, the environment in which humans live has undergone very significant change due to emergence of societies; the new environment may not be pushing us as hard [in the direction of intelligence], at least on the individual level. [and may have been pushing us too hard for smaller brains, thanks Nornagest for making that point]

We were evolving ability to think, until it got just about to the point of being barely able - with great difficulty and many falls - to think useful thoughts. If we were species that were evolving flight, we'd be the species that could just barely fly, and recently flew over a river, entering new land. In the new land, everything is different. And our wings were shrinking at very rapid rate.

The important question is - Did we lose any functionality since then? Are we dumber? Are we less sane in some way? (The palaeolithic humans did not seem to do any really insane religious stuff)

The notion that our brains just got more efficient and 'therefore' could shrink in size appears very shaky to me. This 'therefore' comes from fallacy of anthropomorphizing the evolution. Evolution doesn't work to a goal of optimizing some sub-unit in the organism while preserving specifications, in the way that a team on an engineering project would.

The optimization could as well instead make brain even larger, if said improvement made larger brain pay off more. One would have to show that some improvement in brain efficiency has actually decreased advantage of big brains over small brains, to explain the smaller brains with them being more optimized.

The evolution optimizes the whole organism, not the brain, and there's very many of other factors that have changed at that specific time that may as well have decreased selection pressure towards intelligence or increased the costs.

In my opinion the sensible default hypothesis should be that we had a decrease in some functionality, and likely are still declining.

My best guess is that it is the capacity to invent solutions on spot and think by ourselves, that we are losing. Before emergence of societies, the technological progress was severely limited by information loss. Any smart individual could massively improve fitness of the relevant genes by (re)inventing some basic, but extremely effective techniques, which he'd teach mostly to genetically related individuals. The technique would easy become lost, creating again an opportunity for intelligence to succeed - reinventing it.

Even very simple invention requires massive search in the vast space of possibilities. Precisely the kind of task that one would expect to benefit from larger raw computational power.

edit:

Some clarification with regards to the need for innovation. In the long run, it is not enough to just do what you're taught. Teaching is a lossy process. You need to improve upon what was taught to you a little to make the tool as good as your ancestor made - you need minor innovation to merely preserve the tools - a little more innovation and you'll improve over time, a little less and you'll lose it over time. The little children have to figure out everything from a few clues; they don't download some braindump of the wisest elder to be able to speak, they essentially figure out an alien language - a very difficult task.

From

http://astrobio.net/pressrelease/4569/computers-that-think-like-humans

The group is therefore using a psychological model of human patterns in their computer programmes. They have integrated a mathematical model that models human-like problem solving. The programme that solves progressive matrices scores IQ 100 and has the unique ability of being able to solve the problems without having access to any response alternatives. The group has improved the programme that specialises in number sequences to the point where it is now able to ace the tests, implying an IQ of at least 150.

'Our programmes are beating the conventional math programmes because we are combining mathematics and psychology. Our method can potentially be used to identify patterns in any data with a psychological component, such as financial data. But it is not as good at finding patterns in more science-type data, such as weather data, since then the human psyche is not involved,' says Strannegård.

That's an awesome study.

I always thought the variations of continue series test (progressive matrices, number sequences, word A is to word B as word  C is to ?? etc) are very culturally biased. You solve those best and easiest by sharing with the test maker the learning environment (and for visual ones, sharing visual environment), as well as sharing neural architecture. That lets you pick same choice as the test maker [edit: and do so easily and naturally]. And this research provides very good demonstration.

Of course there will be a correlation of ability to guess the same or secondguess the test maker with intelligence, but so does e.g. height correlate with intelligence (via effect of nutrition on both); perhaps we should add 'what is your height' question to IQ test and then let some giant robot score a genius.

Note: one might think of sequence guessing as task of minimizing Kolmogorov complexity. That's not quite so, sequences are too short, shorter than the generators. Consider sequence 2,3,5,7,11,? . Obviously the answer on IQ test would be 13 (primes). Good luck writing primes generating program that is simpler than this sequence itself, though [edit: i mean, simpler than a program which just prints those numbers followed by whatever garbage. Unless you have a language where 'print primes' is a basic command]. (and of course the length of program will be very dependent on the machine being used)

The reasoning about huge numbers of beings is a recurring theme here. Knuth's up-arrow notation is often used, with 3^^^3 as the number of beings.

I want to note that if a being is made of 10^30 parts, with 10^30 distinct states of each part, the number of distinct being states is (10^30)^(10^30) = 10^(3*10^31) . That's not a very big number; stacking uparrows quickly gets you to much larger numbers.

To quote from Torture Vs Dust Specks:

 

• 3^3 = 27.
• 3^^3 = (3^(3^3)) = 3^27 = 7625597484987.
• 3^^^3 = (3^^(3^^3)) = 3^^7625597484987 = (3^(3^(3^(... 7625597484987 times ...)))).

3^^^3 is an exponential tower of 3s which is 7,625,597,484,987 layers tall.  You start with 1; raise 3 to the power of 1 to get 3; raise 3 to the power of 3 to get 27; raise 3 to the power of 27 to get 7625597484987; raise 3 to the power of 7625597484987 to get a number much larger than the number of atoms in the universe, but which could still be written down in base 10, on 100 square kilometers of paper; then raise 3 to that power; and continue until you've exponentiated 7625597484987 times.  That's 3^^^3.  It's the smallest simple inconceivably huge number I know.

That's an unimaginably bigger number than 10^(3*10^31) . You just can't have 3^^^3 distinct humans (or the beings that are to human as human is to amoeba, or that repeated zillion times, or distinct universes for that matter). Most of them will be exactly identical to very many others among the 3^^^3 and have exactly identical experience*.

Of course, our reasoning does not somehow subconsciously impose a reasonable cap on number of beings and end up rational afterwards. I'm not arguing that gut feeling includes such consideration. (I'd say it usually just considers substantially different things incomparable and in-convertible, plus the space of utility needs not be one dimensional)

I've made this pigeon-hole example to demonstrate a failure with really huge numbers, that can undermine by an inconceivably huge factor the reasoning that seems rational and utilitarian and carefully done.

Also, it does seem to me that if the reasoning with huge numbers is likely to result in reasoning errors, then it can be rational to adopt some constraints/safeguards (e.g. veto approval of torture on basis of dust specks, veto pascal's mugging with very huge numbers, perhaps in general veto conversion between things of very different magnitude) as a rational strategy when one is aware that one is likely processing huge numbers incorrectly, not just on the gut feeling level but on conscious work with pencil and paper level as well.

An autopilot may strive for some minimization of total passenger discomfort over the flight, but also have a hard constraints on the max acceleration in the case that the discomfort minimization approach leads to something ridiculous.

* footnote: I don't think many people involved with AI research would count identical copies multiple times. But that is a tangential point. The issue is that when reading of 3^^^3 beings, it is really easy to make a mistake of not even checking whenever you do or don't count identical copies many times. The problem is that 3^^^3 is much, much larger than the numbers we would normally approximate as infinite.

On the counting of 'identical' items. Consider a computer system that has 2 copies of all data and re-does every calculation it makes. If it runs an AI, it may seem sensible to count AI twice when it's 2 computers in 2 boxes that are staying next to each other running same software on same input, but much less so if you picture one computer where each chip got two dies, one a mirror copy of the other, put right on top of it, separated by very thin layer of dielectric which serves no purpose (the potentials are same on both sides of it), and it's absurd if you remove the dielectric - it's 1 computer, just the wires are thicker, currents 2x larger, and transistors are in parallel pairs. Counting identical stuff several times is something we do when there's a difference in e.g. location, which renders stuff not identical. Decrease the spatial separation and the inclination to count identical items twice decreases. Have a giant server farm where next to each server there is the 2 backup servers in identical state (to recover when one fails), and I think just about any programmer would quickly forget about this minor implementation detail; have two giant server farms on opposite sides of Earth and you'll for sure feel like counting it twice.

edit: sorry for not being explicit, I kind of assumed the point was clear enough. Improved it.

Also, that's not for just dust specks vs torture but goes for all the other examples where the knuth up arrows are used to make very huge numbers. Pascal's mugging discussions for example.

The hunter-gatherer example in the Is risk aversion really irrational? got me thinking about the real world issues with 'maximizing utility' and any other simple rule approach to decision making.

The elephant in the room is that universal, effective utility of anything could be very expensive to calculate if you employ any foresight (consider thinking several moves ahead). And once you start estimating utility in different ways depending to the domain, the agent's behaviour stops being consistent with plain utility maximization. At same time, the solution space of the problems is often very big, meaning that you have immense number of potential choices and you need to perform a lot of utility estimations really quickly to pick the best solution. Think of Chess or Go. The computing time could be better spent elsewhere.

The hunter-gatherer in example can think about the traps and other hunting tools and invent a new one, instead of trying to figure out probability theory or something of this kind.

Inventing a new trap is a case where the number of potential decisions is extremely huge.

I faintly recall a fiction story I've read where a smart boy becomes tribe leader - by inventing a better bear trap, not so much by being utterly rational at correctly processing small differences in expected utility when it comes to bets.

He can also think more about berries and look if there's evidence that other mammals are eating those berries; some plant that is not poisonous to other mammal is very unlikely to hurt a human; some plant that is not eaten by other mammals is very likely to be poisonous to humans as well. He can even feed the berries to some mammal he'd keep alive (i'd imagine keeping animals alive was a fairly straightforward approach to meat preservation).

At the same time, even if that hunter gatherer knew enough math to try to formally calculate his odds, the probabilities are unknown. Indeed there are probability distributions for different degrees of getting sick of berries or not (and different symptoms of sickness), et cetera. We today are just beginning to think how to improve his odds using formal mathematics, and we're still not sure how to accomplish that, and it is clear that it is going to be very computationally intensive.

As a singular example, I can easily come up with good solutions for that hunter-gatherer by looking into the big solution space that he would have (he's living in the real world), but it is much harder and much more tedious for me to calculate his odds even in a very simplified example where probabilities of getting sick or winning a duel are exact, and the 'sick or not sick' is a binary outcome. That's with me having a computer at my fingertips, and knowledge of mathematics tens thousands years down the road from hunter gatherer!

Bottom line, it would be very suboptimal for the intelligent hunter gatherer to try to use his intelligence in this particular expected-utility-calculating way to slightly optimize his behaviour (keep in mind that he has no way of estimating probabilities), when lesser amount of good thought would allow him to invent something extremely useful and gain the status.

As a personal success story - I have developed and successfully published a computer game, and made good income on it. The effort that can be spent on decision making - on choosing to implement A or B, is always tightly capped by the other ways of applying effort that would pay off more (implementing both A and B, or searching the solution space more in the hope of coming up with C). It is very rare that putting effort into very careful choice between very few options is the best use of intelligence. It is common in thought experiments but its rare in reality.