Thank you very much—link fixed!

That's a really funny quote!

Multi-armed bandit problems were intractable during WWII probably mainly because computers weren't available yet. In many cases, the best approach is brute force simulation. That's the way I would approach the "blue box" problem (because I'm lazy).

But exact approaches have also been found: "Burnetas AN and Katehakis MN (1996) also provided an explicit solution for the important case in which the distributions of outcomes follow arbitrary (i.e., nonparametric) discrete, univariate distributions." The blue box problem is within that class.