My understanding of LLMs:
I have some very basic understanding of LLM and the underlying transformer architecture. As I understand it, these LLMs are, as an analogy, basically a much-improved version of autocomplete. During training they are shown a word-sequence with some words masked out and the LLM is asked to guess what the correct word is. If it gets the word wrong the model gets adjusted to better predict the correct word the next time.
The attention mechanism is trained to give the LLM an idea which previous words are most important to pay attention to when predicting the masked word.
That’s all the LLM learns: what previous words to pay attention to and... (read 295 more words →)
I hope we will see multi-modal modells. To me it will be very important for the future of AI/AGI development to see whether we will get significant transfer/synergy-learning (from one modality to another).
IF these synergy effects would occur (and my prediction is that they will not occur to a significant degree), this would push my AI predictions towards shorter time horizons for useful AI and AGI.
If we wont see transfer learning it would significantly lengthen my AGI time horizon because it would indicate to me that we will have to come up with additional and new techniques to enhance our AI models towards generality.
I really like the idea of such an argument mapping system. In the early days of wikipedia I thought it might be a natural extension of the wikipedia project to have a debate-plattform attatched to it. So its interesting for me to see that a platform like Kialo exists - I didnt know about it before your post, thanks.
I hold the same opinion as you do, that many arguments are very complex and when you try to get a good impression of the state of the debate it is a complicated task, because its hard to find good, exhaustive and well structured debates. And a platform that systematizes this important task is very valueable for people and society in my opinion.
My intuition for the most promising path to AGI is with the Gary Marcus/Ben Goertzel Neuro-Symbolic camp.
As the years passed and so much money has gone to the pure learning, neural networks only, scaling-hypothesis branch of AI and imo. far too little has gone to improve on fundational methods I grew more sceptical of a rapid progress to AGI.
About four years ago there was a DARPA funding for the sort term (2 yrs?) discovery of "next generation" AI methods with a high budget (maybe 1.5 billion USD). I dont now of anything with substance that came from it.
Also, things like Hintons capsule networks seemed interesting to me but again nothing much came... (read more)
Thanks for the links. I think this will greatly enhance my understanding of the matter.