Write down your basic realizations as a matter of habit. Share this file with the world.
Say you have just realized something that seems, in hindsight, pretty obvious - but that you were totally unaware of before now. You scratch your head, update your model of the world, and move on. Hold on. Update your model of your model. Why were you not aware of it...
I think what is actually happening is "yes, all the benchmarks are inadequate." In humans, those benchmarks correlate to a particular kind of ability we may call 'able to navigate society and to, in some field, improve it.' Top of the line AIs still routinely deletes people's home dirs and cannot run a profitable business even if extremely handheld. AIs have only really started this year to convincingly contribute to software projects outside of toys. There are still many software projects that could never be created by even a team of AIs all running in pro mode at 100x the cost of living of a human. Benchmarks are fundamentally an attempt to... (read more)