Definitely when:

• You are only going in circles. ** You need more data, to do so, you should preform an experiment.
• You can no longer remember/track your best created strategies.
• You can not judge value difference between new strategies and existing strategies.
• You spend x percentage of your time tracking/remember your created strategies. Where x is significant.
• There are better questions to consider.
• The value of answering the question will diminish greatly if you spend more time trying to optimize it. ** "It is great you finished the test and got all the right answers but the test was over a week ago" -- extreme example some times .../years/months/weeks/days/hours/minutes/seconds/... count.

It can be a hard question to get right in my experience.