You heard the LLM, alignment is solved!

But seriously, it definitely has a lot of unwarranted confidence in its accomplishments.

I guess the connection to the real world is what will throw off such systems until they are trained on more real-world-like data.

I wouldn't phrase it that it needs to be trained on more data. More like it needs to be retrained within an actual R&D loop. Have it actually write and execute its own code, test its hypotheses, evaluate the results, and iterate. Use RLHF to evaluate its assessments and a debugger to evaluate its code. It doesn't matter whether this involves interacting with the "real world," only that it learns to make its beliefs pay rent.

Anyway, that would help with its capabilities in this area, but it might be just a teensy bit dangerous to teach an LLM to do R&D like this without putting it in an air-gapped virtual sandbox, unless you can figure out how to solve alignment first.

fwiw I'm kinda less interested in prompts like this (which... just obviously won't work, and are maybe funny but not particularly informative) and more interested in people experimenting with what sort of simpler tasks it can actually reliably do.

They ask the user to provide their own OpenAI API key. Is that wise? I am not a paid subscriber to OpenAI, so I haven't experienced how it works, but handing over one's key... won't that become risky at some point?