This post describes important true characteristics of a phenomenon present in the social reality we inhabit. But importantly the phenomenon is a blind spot which is harder to notice when acting or speaking with a worldview constructed from background facts which suffer from the blind spot. It hides itself from the view of those who don't see it and act as if it isn't there. Usually bits of reality you are ignorant of will poke out more when acting in ignorance, not less. But if you speak as if you don't know about the dark matter you will be broadcasting that you are a bad choice for those who are hiding to talk honestly with.

By having a handle for the phenomenon in the abstract, that problematic loop is much easier to break; even if you don't see it yet, you may much more easily notice that it might be present and act accordingly to search out information in a different way or simply avoid sticking your foot in your mouth.

Liam alone makes $10

Emma alone makes $20

Liam + Emma make $30

$30 - ($10 + $20) = $0, their synergy.

In general: the synergy is how much more or less the coalition gets than each member's individual contribution plus all subset synergies.

I liked this one! I was able to have significant amounts of fun with it despite perennial lack-of-time problems.

Pros:

• simple enough underlying mechanism to be realistically discoverable
• some debias-able selection bias
• I could get pretty far by relatively simple data exploration
• +4 Boots was fun

Cons:

• I really wanted the in-between-tournament matches to mean something, like the winners took the losers equipment or whatnot and you could see that show up later in the dataset, but of course that particular meaning would have added a lot of complexity for no gain.
• bonus objective was not confirmable (yep real life is like that but still :D)

It feels like this scenario should be fully knowably solvable, given time, except for the bonus guess at the end, which is very cool.

I think the bonus objective was a good idea in theory but not well tuned. It suffered from the classic puzzle problem of the extraction being the hard part, rather than the cool puzzle being the hard part.

I think it was perfectly reasonable to expect that at some point a player would group by [level, boots] and count and notice there was something to dig into.

But, having found the elf anomaly, I don't think it was reasonable to expect that a player would be able to distinguish between

• do not reveal the +4 boots at all
• do not use the +4 boots vs the elf ninja
• give the elf ninja the +4 boots to be used in their combat
• give the elf ninja the +4 boots afterwards but go ahead and use them first

It's perfectly reasonable to expect that a player could generate a number of hypotheses and guess that the most likely was that they shouldn't reveal the +4 boots at all, but they would have no real way of confirming that guess; the fact that they're rewarded for guessing correctly is probably better than the alternative but is not satisfying IMO.

I found myself having done some data exploration but without time to focus and go much deeper. But also with a conviction that bouts were determined in a fairly simple way without persistent hidden variables (see Appendix A). I've done work with genetic programming but it's been many years, so I tried getting ChatGPT-4o w/ canvas to set me up a good structure with crossover and such and fill out the various operation nodes, etc. This was fairly ineffective; perhaps I could have better described the sort of operation trees I wanted, but I've done plenty of LLM generation / tweak / iterate work, and it felt like I would need a good bit of time to get something actually useful.

That said, I believe any halfway decently regularized genetic programming setup would have found either the correct ruleset or close enough that manual inspection would yield the right guess. The setup I had begun contained exactly one source of randomness: an operation "roll a d6". :D

Appendix A: an excerpt from my LLM instructions

I believe the hidden generation is a simple fairly intuitive simulation. For example (this isn't right, just illustrative) maybe first we check for range (affected by class), see if speed (affected by race and boots) changes who goes first, see if strength matters at all for whether you hit (race and gauntlets), determine who gets the first hit (if everything else is tied then 50/50 chance), and first hit wins. Maybe some simple dice rolls are involved.

Not to toot my own horn* but we detected it when I was given the project of turning some of our visualizations into something that could accept QA's format so they could look at their results using those visualizations and then I was like "... so how does QA work here, exactly? Like what's the process?"

I do not know the real-world impact of fixing the overfitting.

*tooting one's own horn always follows this phrase

Once upon a time I worked on language models and we trained on data that was correctly split from tuning data that was correctly split from test data.

And then we sent our results to the QA team who had their own data, and if their results were not good enough, we tried again. Good enough meant "enough lift over previous benchmarks". So back and forth we went until QA reported success. On their dataset. Their unchanging test dataset.

But clearly since we correctly split all of our data, and since we could not see the contents of QA's test dataset, no leakage could be occurring.