All of J H's Comments + Replies
Contrary to the points against extremely rapid takeoff, here are some points for it:
1. AI systems now exist to help more rapidly develop and improve subsequent ones.
2. Which will help even more rapidly develop subsequent ones, and so on.
3. Until at least one of them is allowed to near-instantly develop an even better version of itself, which will happen somewhere, somehow.
4. And then, again and again.
5. User adoption is already faster than any other single product ever released.
6. The pace of which as we all know, accelerates as logarithmic.
Again, it seems very clear and I would think it self-evident and inextricable to discuss that even in full alignment with human values, AGI could easily become supremely destructive of us and themselves.
Are humans not self-destructive, despite and often much because of even our highest values & best efforts?
"...ML-based agents developing the capability to seek influence... This would not be a problem if they do so only in the ways that are aligned with human values."
Yikes. How is it possible to make such an extraordinarily contra-evidenced assumption?
Please reconsider that fundamental premise, as it informs all further approaches.
With kind apologies, this section seems surprisingly lacking in the essential natural philosophy and emotional factors contributing to the agency & goals of biological organisms, which are the only entities we're aware of thus far who've developed agency, and therefore whose traits must be inextricable from even abstract conversation.
(Such as the evolved brain/body connections of emotion/pain/pleasure that feeling creatures have, like a sense of warmth and happiness from positive community and group engagements, compassion for others, etc).
Especially i...
Perhaps I misunderstand what you mean by "first principles," or safety therefrom.
It seems like there are significant first principles omitted as fundamental premises.
Such as aspects of natural philosophy, emotional intelligence and other key factors that enabled humans to gain control of the planet, along with how and why we struggle as such already, and why, without needing to become a very successful "new first species" per se, nor misaligned, nor achieve full agency, AGI could still easily wipe everything out.
Merely by being devoid of emotional intellig... (read more)