What I understood: We wouldn't be automatically right by saying the opposite of what stupid people say. This is because stupid people are wrong from many angles, and to be right one needs to be right in all angles. And some times, stupid people may be correct too. Or rather, stupidity of believers is not evidence against the belief. In fact, most beliefs are not completely wrong. We want to keep the correct parts and reject wrong parts, not reject everything once we realize it fails to predict our goals.
I restate my comment on Substack here.
I think this is the story of an AI, not a human. This is a future I find horrifying, where humanity dies out, never realizing it until the end. Many here seem to think it is enough as long as a super intelligence does not wipe out humanity, helping it instead. But for humanity, any being that makes humanity redundant is a death knell in the long run. This is a kind of Moloch situation.
To go into the specifics, when the author seem to use the 'Ship of Theseus' argument, he did not seem to realize that if the boat is dismantled piece by piece and a house is built with the pieces, it is definitely no longer a ship, let alone the same ship. In fact, the change starts with internalization of AI, or more precisely, when the protagonist stopped using his biological mind in favor of AI making the decisions.
As a materialist, I disagree so early with your chain of thought that we share only a little of our worldview. Our disagreement started when you start with
The Veil of Ignorance, but adequate.
This though experiment is interesting to read, but delves too far from reality. I find that it makes it very easy to mistake the map for territory.
But trivialities aside, I see that the thought experiment tries to construct the idea of a society that the thinker finds to be good enough, on average. But this is inherently flawed, since there are too many unknowns to even start.
Then comes the
you never had the chance of choosing who you are born as
point.... (read more)
I understand that you are saying that even a single resource lacking for one to "thrive" is poverty, that poor people are not thriving today because they lack some resource that they need to thrive. This resource was not provided to them even after the society became much more productive, and hence probably would not be provided to them if we implement UBI. You try to show this using a counter factual country where the critical resource is oxygen.
I think this is a false equivalence. The 'equilibrium' that enforces poverty is actually people themselves. I think the resource lacking is not something essential to simply survive. Simple survival, after all, can be... (read more)
I think many assign a much higher probability to the existence and usefulness of superintelligence than it warrants. My intuition is that they require the universe to have much more structure than we can currently detect. This is because our observations are highly accurate these days (at least in fundamental sciences like physics), and scientific theories give very powerful explanations for them.
This is because superintelligence depends as much on properties of the world as on the algorithms themselves. The same argument works for usefulness. Even if a superintelligence exists, it cannot do impossible tasks.
This is the main reason for my skepticism regarding what I term AI magicalism, in which it is expected... (read more)
I disagree with the author in that I believe the universe is 'compact' in the sense that what we know is a near approximation of what is knowable by the method.
For example, I believe our ancestors knew everything they can possibly know without knowing extra background knowledge or special kinds of thinking.
Similarly, I believe we now know nearly everything about the universe (the rules, not specific tools) given use of mathematics and sophisticated logic. And we do a much, much better job than our ancestors at predicting the world. So I am skeptical that superintelligence can exist.
You may say, how can you assume whatever we have today is the final stages of... (read more)
I do not think 'growth mindset' is necessary for growth if one understands what 'talent' really means. I define 'talent' in a task as competence at learning new things in that particular task. I think people generally see their current learning speed as limited by their 'talent', but it is actually limited by concentration/effort/dedication.
After certain stage, the later matter a lot more than what many think. We also see that people with growth mindset do not improve their talent, but the other things. It would be good if people with fixed mindset realize that talent is not everything. This is not a question of 'mindset', but unbiased review of competence function.
Curiously, there is a theory about how schizophrenia is due to differences in dimentionality of neurons in different parts of the brain. I wonder what you think about it?
Thankyou for pointing out holes in my argument.
I don't think Google search engine is an entity that I call a demon of statistics.
I classify thought processes as algorithmic and statistical. The former merely depends on IQ, while the later is more subjective, based on mental models. I am thinking along lines parallel to JonahS in his posts on mathematical ability.
To explain my reasoning, I think while it is difficult to distinguish simple statistical machines (as in smart keyboards, search engines) differ from demons of statistics, we must distinguish them based on their position in intelligence space.
Search engines do not give you sentences, but the result associated with the query, as I understand... (read more)
I disagree with your first point. You are saying people who use a tool are already 'post human' in some sense. But then, are people who can use abacus in 14th century post human? Are African tribes that use their technical knowledge to hunt animals, less human than a hypothetical tribe that never got to use anything like a spear, and fight with their bare hands? By that logic, chimps are more 'human' than humans!
I think we can draw a line. Algorithms are more or less things tools that give answers to what we want. It is a mistake to think they are above humans; computers just let us effectively use them.... (read more)