Should we kill recurrent memory over self attention ❓
Spending most of my time on time series problems, I often think about the consequence of memory and the sequential nature we are exposed to in the physical world.
Memory is the idea for a learning algorithm to store a representation of the system's state over time. Think of how much you remember from what you learned a week ago. Memory is partially observable today and naturally, in our conscious experience, large parts of the experience tend to fade.
Im going to discuss two main differentiable programming (deep learning) paradigms: Recurrent Neural Network and Transformers. Then discuss the premise for my question.
Recurrent differential programs (RNN's) enable... (read 272 more words →)
This seems like the only realistic aspiration we can pursue. It would require pressure from players that have centralized compute hardware where any large scale train runs require that level of code and data transparency. Hardware companies could also flag large acquisitions. Ultimately sovereign nations will have to push hardest, this alongside global cooperation seems insurmountable. The true problem is that there is no virus causing enough harm to take action, rather the emergence of intelligent phenomena none of us, even their creators, fail to understand. So beyond sparking hollow debate, what can we tangibly do? Where do the dangers actually lie?