All of meijer1973's Comments + Replies

This emphasis on generality makes deployment of future models a lot easier. We first build a gpt4 ecosystem. When gpt5 comes out it will be easy to implement (e.g. autogpt can run just as easy on gpt4 as on gpt5). The adaptions that are necessary are very small and thus very fast deployment of future models is to be expected.

Fine-tuning, whether using RL or not, is the proverbial “cherry on the cake” and the pre-trained model captures more than 99.9% of the intelligence of the model. 

I am still amazed by the strength of general models. There is the no-free lunch theorem that people use to point out that we will probably have specialized AI's because they will be better. Current practice seems to contradict this. 

AI will probably displace a lot of cognitive workers in the near future. And physical labor might take a while to get below 25$/hr.

• Most most tasks human level intelligence is not required. 
• Most highly valued jobs have a lot of tasks that do not require high intelligence.
• Doing 95% of all tasks could be a lot sooner (10-15 years earlier) than 100%. See autonomous driving (getting to 95% safe or 99,9999 safe is a big difference).
• Physical labor by robots will probably remain expensive for a long time (e.g. a robot plumber). A robot ceo is probably cheaper

An interesting development is the development of synthetic data. This is also a sort of algorithmic improvement, because the data is generated by algorithms. For example in the verify step by step paper there is a combination of synthetic data and human labelling. 

At first this seemed counter intuitive to me. The current model is being used to create data for the next model. Feels like bootstrapping. But it starts to make sense now. Better prompting (like CoT or ToT) is a method to get better data or a second model that is trained to pick the best ans... (read more)

Specific Resources (Access to a DGX data center): Even if an AI had access to such resources, it would still need to understand how to use them effectively, which would require capabilities beyond what GPT-4 or a hypothetical GPT-5 have.

To my knowledge resource management in data centers is done by AI's. It is the humans who cannot do this. The AI already can.

Algorithmic improvement has more FOOM potential. Hardware always has a lag. 

Hanson's chance on extinction is close to a 100%. He just thinks it's slower. He is optimistic about something that most would call a dystopia (a very interesting technological race that will conquer the stars before the grabby aliens do). A discussion between Yudkowsky and Hanson is about are we dying fast or slow. It is not really a doomer vs non-doomer debate from my perspective (still a very interesting debate btw, both have good arguments).

I do appreciate the Hanson perspective. It is well thought out and coherent. I just would not call it optimistic ... (read more)

If I understand you correctly you mean this transfer between machine learning and human learning. Which is an interesting topic. 

When a few years ago I learned about word2vec I was quite impressed. It felt a lot like how humans store information according to cognitive psychology. In cognitive psychology, a latent space or a word vector would be named as a semantic representation. Semantic representations are mental representations of the meaning of words or concepts. They are thought to be stored in the brain as distributed representations, meaning th... (read more)

Your model has some uncertainty, but you know the statistical distributions. For example, with probability 80% the world is in state X, with probability 20% it is in state Y.

Nice way of putting it. 

• Mathematical definition: Optimization is the process of finding the best possible solution to a problem, given a set of constraints.
• Practical definition: Optimization is the process of improving the performance of a system, such as by minimizing costs, maximizing profits, or improving efficiency.

In my comment I focused on the second interpretation (by focussing on iteration). The first definition does not require a perfect model of the world. 

In the real world we always have limited information and compute and so the best possible solution is always a... (read more)

Strong world-optimization only happens if there is a robust and strong correlation between the world-model and reality.

Humans and corporations do not have perfect world models. Our knowledge of the world and therefore our world models are very limited. Still humans and corporations manage to optimize. Mostly this happens by trial and error (and copying succesful behaviors of others). 

So I wonder if strong world-optimization could occur as an interative process based on an imperfect model of the world. This however assumes interaction with the wo... (read more)

People are finding ways to push the boundaries of the capabilities GPT-4 and are quite succesful at that (in reasoning, agency etc). These algorithmic improvements will probably also work on gpt5. 

A lot of infrastructure built for gpt4 will also work on gpt5 (like plug-ins). We do not need to build new plug-ins for gpt5, we just swap the underlying foundational model (greatly increasing the adoption of gpt5 compared to gpt4).

This also works for agency shells like autogpt. Autogpt is independant of foundational model (works with gpt3.5, gpt4 and also g... (read more)

Here a summary of the Hanson position (by himself). He is very clear about humanity being replaced by AI. 

https://www.overcomingbias.com/p/to-imagine-ai-imagine-no-ai

I like your motivation, robotics can bring a lot of good. It is good to work on automating the boring and dangerous work.

I see this as a broken promise. For a long time this was the message (we will automate the boring and dangerous). But now we automate valuable jobs like STEM, journalism, art etc. These are the jobs that give meaning to life and they provide positive externalities I like to talk to different people soI meet the critical journalist, the creative artist, the passionate teacher etc. 

E.g. we need a fraction of the people to be journalis... (read more)

Thanks for the post. I would like to add that I see a difference in automation speed of cognitive work and physical work. In physical work the growth of productivity is rather constant. With cognitive work there is a sudden jump from not much use cases to a lot of use cases ( like a sgmoid). And physical labour has speed limits. And also costs, generality and deployment are different.

It is very difficult to create a usefull AI for legal or programming work. But once you are over the treshold (as we are now) there are a lot of use cases and productivity gro... (read more)

Became recently aware of the progress made in synthetic data and other algorithmic improvements. We have not pushed GPT-4 to the max yet. 

e.g. this paper https://arxiv.org/abs/2305.20050

It details how training on the steps in step by step reasoning as opposed to just rewarding the end result can give significant improvements. And there is so much more.  

Agreed, one of the objectives of a game is to not die during the game. This is also true for possible fatal experiments like inventing AGI. You have one or a few shots to get it right. But to win you got to stay in the game. 

Note that Hanson currently thinks the chances of AI doom are < 1%, while Yudkowsky thinks that they are > 99%.

It is good to note that the optimistic version of Hanson would be considered doom by many (including Yudkowsky). Doom/utopia definition Yudkowsky is not equal to doom/utopia definition of Hanson. 

This is important in many discussions. Many non-doomers have definitions of utopia that many consider to be dystopian. E.g. AI will replace humans to create a very interesting future where the AI's will conquer the stars, some think this is positive others think this is doom because there are no humans. 

Thanks for the addition. Vertical and indoor farming should improve on the current fragility (thus add to robustness) of the agricultural industry. Feeding 8 billion people will still cost a lot of resources. 

Mining however is different in that mining cost will ever increase due to decreasing quality of ore and ores being mined in places that are harder to reach. This effect could be offset by technological progress for a limited time (unless we go to the stars). Vast improvements in recycling could be a solution, but that requires a lot of energy. 

Solving the energy problem via fusion energy would really help a lot for the more utopian scenario's. 

Agency is advancing pretty fast. Hard to tell how hard this problem is. But there is a lot of overhang. We are not seeing gpt-4 at its maximum potential.

Agree, human in the loop systems are very valuable and probably temporary. HITL systems provide valuable data for training allowing the next step. AI alone is indeed much faster and cheaper. 

One difference I suspect could be generality over specialization in the cognitive domain. It is assumed that specialization is better. But this might only be true for the physical domain. In the cognitive domain general reasoning skills might be more important. E.g. for an ASI the specialized knowledge of a lawyer might be small from the perspective of the ASI.  

Good point. As I understood it, humans have an OOM more parameters than chimps. But chimps also have an OOM over a dog. So not all OOM's are created equal (so I agree with your point).

I am very curious about the qualatative differences between humans and and superintelligence. Are there qualatative emergent capabilities aboven human leven intelligence that we cannot imagine or predict at the moment.   

According to Deepmind we should aim at that little spot at the top ( I added the yellow arrow). This spot is still dangerous btw. Seems tricky to me.

Image from the deepmind paper on extreme risk.(https://arxiv.org/pdf/2305.15324.pdf) 

True, it depends on the ratio mundane and high stakes decisions. Athough there are high stakes decisions that are also time dependant. See the example about high frequency trading (no human in the loop and the algorithm makes trades in the millions).  

Furthermore your conclusion that time independant high stakes decisions will be the tasks where humans provide most value seems true to me. AI will easily be superior when there are time constraint. Absent such constraints, humans will have a better chance of competing with AI. And economic strategic dec... (read more)

Time should also be a factor when comparing strength between AI alone and an AI-human team. Humans might add to correspondence chess but it will cost them a significant amount of time. Human-AI teams are very slow compared to AI alone.

For example in low latency algorithmic stock trading reaction times are below 10ms. Human reaction time is 250ms. A human-AI cooperation of stock traders would have a minimum reaction time of 250ms (if the human immediatly agrees when the AI suggests a trade), This is way to slow and means a serious competitive disadvantage.&... (read more)