How well can Claude write coding questions?
I'm curious as to how well Claude can write interesting coding and mathematics problems. This post is a partial product of that exploration. It is a much harder skill to come up with a good problem than to solve one of equal difficulty. While there is a large corpus of data on solving problems, there is very little on writing them. Problem authors typically just share their problem and not the thought process that went into it. I also think that it's a skill correlated to doing good research. FYI, I prompted Claude to write both interesting and novel questions but I did not seriously research whether the questions it wrote were actually novel. (I know for sure that some of the problems it makes are novel because they clearly don't have a workable solution and are therefore probably not published anywhere). Codeforces For those who don't know, Codeforces is a competitive programming site. Its contests are split up into three divisions: division 1, 2 and 3. Division 1 contests are the hardest and division 3 contests are the easiest. These contests typically run for 2 hours and typically contain 6-7 problems. The problems are labeled alphabetically with A being the easiest. I had a rating of 1600 and I competed in division 2 contests mostly where I would usually solve problems from A-D and no more. I first prompted Claude "Write a prompt that would help you generate an original an interesting codeforces problem designed for problem D in a division 2 contest." (Prompting Claude "generate an original an interesting codeforces problem designed for problem D in a division 2 contest" will make a boring and easy problem.) After some small tweaks the prompt ended up as this: Prompt Create an original competitive programming problem suitable for Problem D in a Codeforces Division 2 contest. The problem should have the following characteristics: Problem Difficulty and Prerequisites * Appropriate for upper-intermediate competitive programmers * Should re
I agree with this a lot.
Do you have any thoughts on how you make multi-agent interactions between virtuous AIs robust to defectors? It seems like a strong case for solving interpretability in order to verify weights have the virtue of honesty.