Above is a link to an interesting post about synthetic code generation with a transformer model trained on The Pile, which contains a large chuck of GitHub and StackOverflow. Due to CommonCrawl's deficiency in this area, the much smaller GPT-J-6B outperforms OpenAI’s largest publicly available GPT-3 models. The performance is impressive enough that one wonders how capable a 100+ billion parameter model trained on The Pile will be, let alone what an AlphaGo-level engineering effort towards the end of synthetic code generation would achieve.
As the The Pile was created to provide a dataset for 100 billion paramater+ models, we may not have to wait long. The examples in the post are clearly trivial, but I personally take this to be something of a fire alarm. I was not previously aware of how poorly-optimized GPT-3 was for code generation, and I have updated toward surprising gains in this area in the next few years.
I no longer consider agents with superhuman performance in competitive programming to be a ridiculous thing to pursue.
It is useful to remind myself of how shocked I would be to see such things in 2012. In 2012 I would have taken this as a sign that AGI was near.
Scenario-based planning postulates that one should predict symptoms emblematic of a given scenario and then robotically assume you are in said scenario once a sufficient number of these symptoms come to pass. I am unsure whether there is wisdom in this approach, but I find it a discomfiting line of thought.
Have you tried, y'know, testing your belief? ;)
You can Google its answers. I've been googling its answers and am not generally finding direct copy-pastes for each, though I'm also a bit confused about why Google is obtaining no results for short strings such as "(s[0:length] == s[length::-1])".
ETA: even if it's copying code but modifying it slightly so that the variable names match, it seems like (1) this is itself pretty impressive if it actually works reliably, and (2) I don't think the claim is that current tech is literally shovel ready to replace programmers. That would be a strawman. It's about noticing the potential of this tech before it reaches its destination.