I don't work on cosmic problems in my day-to-day work, but I encounter puzzles fairly frequently (what broke? why did performance at node x decrease at time y? why does z work well sometimes but not others?).

Advice I would give:

• Get some data. If you have too much data, arbitrarily pick a subset you can handle.

• Look for anomalies. Make histograms. Make other graphs. Last weekend I diagnosed a problem with a Windows service with a graph of the Fourier transform of a time series picked out of log files.