I think this is plausibly the best scenario either of us have made to date.
The basic game was very good, layering simple rules on top of each other to create a complex system which was challenging to detangle but easy to understand once you know the rules. I was particularly impressed by the fact that you managed the (imo, near-impossible) feat of making an enjoyable D&D.Sci where handling data starvation was a key part of the problem: most players (including me) seem to have had the (sensible) thought "okay, let's filter for only potions with Onyx and Bone", and success past that point was predicated on realizing there weren't quite enough rows to justify being that picky.
The twist struck me as fair, funny and fun. It provided an object lesson in noticing when things don't quite add up, and letting the qualitative shade the quantitative; it also expanded the scope of the genre in ways I realize I've been neglecting to.
All that said, I have some (minor, petty) criticisms . . . not of the game itself, but how it was presented. Namely:
.This entry was billed as "relatively simple", but I think it was about median difficulty by the standards of D&D.Sci; pretty sure it was harder than (for example) The Sorceror's Personal Shopper.
."STORY (skippable)" was kind of misleading this time: the flavortext had a lot of little hints that the Archmage wasn't on the level, so someone who didn't read it (or failed to read between the lines, like me) would be at a (small) disadvantage.
."Archmage Anachronos is trying to brew Barkskin Potion" was A) the GM saying something false directly to the players, and B) a missed opportunity: if you'd written something like "Your goal is to help Archmage Anachronos brew Barkskin Potion", that would have been a subtle confirmation that giving him exactly what he asked for would lead to the best outcome (vs more aggressive / galaxy-brained forms of sabotage, or refusing to cast judgement on his pursuit of immortality, or any other reaction).
I'm glad you liked it, thank you!
This entry was billed as "relatively simple", but I think it was about median difficulty by the standards of D&D.Sci; pretty sure it was harder than (for example) The Sorceror's Personal Shopper.
I guess that's fair. There's a complication here in that...uh...almost all of my scenarios have been above-median complexity and almost all of yours have been below-median. (I should probably write down my thoughts on this at some point). I agree that this one wasn't simpler than most of yours, but I think that it was still a much more approachable entry point than e.g. Duels & D.Sci, or League of Defenders.
(It's possible we should try to standardize a 1-10 complexity scale or some such so that we can stick a difficulty rating on the top of each scenario.)
"STORY (skippable)" was kind of misleading this time
Fair enough, I can tweak that for anyone who finds the scenario in future.
I intended that the story should not provide much help...the intent was not for players to notice that Anachronos was suspicious in-story, the intent was for them to notice from the data, and for the hints in the story to be just some quiet confirmation for a player who realized the twist from the data and then went back to reread the story.
On the other hand, I was expecting more players to get the twist, and thought that I'd only really catch players who ignored the ingredient names entirely and just fed the data into an ML algorithm, so I'm clearly not very well calibrated on this. I was really quite surprised by how many players analyzed the data well enough to say "Barkskin potion requires Crushed Onyx and Ground Bone, Necromantic Power Potion requires Beech Bark and Oaken Twigs" and then went on to say "this sounds reasonable, I have no further questions." (Maybe the onyx-necromancy connection is more D&D lore than most players knew? But I thought that the bone-necromancy and bark-barkskin connections would be obvious even without that).
"Archmage Anachronos is trying to brew Barkskin Potion" was A) the GM saying something false directly to the players
I...think I'm in general allowed to say false things directly to the players as a D&D GM? If the Big Bad is disguised as your innkeeper while the real innkeeper is tied up in the cellar, I think I can say 'The innkeeper tells you it'll be six silver for a room', I don't think I need to say 'The man who introduced himself to you as the innkeeper.'
(Also, you are a Data Scientist. Sense Motive is not a class skill for you. Clearly you failed a Sense Motive check and so believed him!)
...I'll think about whether I want to tweak that line for potential future players.
I thought the flavour text was just right - I got it from the data, not the flavour text, and saw the flavour text as confirmation, as you intended.
I was really quite surprised by how many players analyzed the data well enough to say "Barkskin potion requires Crushed Onyx and Ground Bone, Necromantic Power Potion requires Beech Bark and Oaken Twigs" and then went on to say "this sounds reasonable, I have no further questions." (Maybe the onyx-necromancy connection is more D&D lore than most players knew? But I thought that the bone-necromancy and bark-barkskin connections would be obvious even without that).
Illusion of transparency I think, hints are harder than anyone making them thinks.
When I looked at the ingredients for a "barkskin potion", as far as I knew at this point the ingredients were arbitrary, so in fact I don't recall finding it suspicious at all. Then later I remember looking at the ingredients for a "necromantic power potion" and thinking something like... "uh... maybe wood stuff is used for wands or something to do necromancy?". It was only when I explicitly made a list of the ingredients for each potion type, rather than looking at each potion individually, and could see that everything else make sense, that I realized the twist.
If the Big Bad is disguised as your innkeeper while the real innkeeper is tied up in the cellar, I think I can say 'The innkeeper tells you it'll be six silver for a room', I don't think I need to say 'The man who introduced himself to you as the innkeeper.'
Perhaps, but you could also simply say "Yeah, the guy at the counter tells you the room will be 6 silver."
...bearing this out, it looks like Lorxus managed to get a perfect score with relatively little actual Data Science just by thinking about what it might mean that including lots of ingredients led to Magical Explosions and including few ingredients led to Inert Glop.
Not quite true! That's where I started to break through, but after that I noticed the Mutagenic Ooze issue as well. It also took me a lot of very careful graceful use of pivot tables. Gods beyond, that table chugged. (And if I can pull the same Truth from the void with less powerful tools, should that not mark me as more powerous in the Art? :P)
I guess I'm not clear on what "actual Data Science" would involve, if not making hypotheses and then conducting observational-experiments? I figured out the MO mechanic specifically by looking at brews that coded for pairs of potions, for the major example. The only thing that would have changed if I'd known SQL would be speed, I suspect.
...and documented his thought process very well, thank you Lorxus!
Always a pleasure! I had a lot of fun with this one. I was a little annoyed by the undeclared bonus objective - I would have wanted any indication at all in the problem statement that anything was not as it appeared. I did notice the correspondence in (i.a.) the Farsight Potion but in the absence of any reason to suspect that the names were anything but fluff, I abstracted away anything past the ingredients being a set of names. Maybe be minimally more obvious? At any rate I'd be happy to be just as detailed in future, if that's something you want.
1I liked the bonus objective myself, but maybe I'm biased about that...
As a someone who is also not a "data scientist" (but just plays one on lesswrong), I also don't know what exactly actual "data science" is, but I guess it's likely intended to mean using more advanced techniques?
(And if I can pull the same Truth from the void with less powerful tools, should that not mark me as more powerous in the Art? :P)
Perhaps, but don't make a virtue of not using the more powerful tools, the objective is to find the truth, not to find it with handicaps...
Speaking of which one thing that could help making things easier is aggregating data, eliminating information you think is irrelevant. For example, in this case, I assumed early on (without actually checking) that timing would likely be irrelevant, so aggregated data for ingredient combinations. As in, each tried ingredient combination gets only one row, with the numbers of different outcomes listed. You can do this by assigning a unique identifier to each ingredient combination (in this case you can just concatenate over the ingredient list), then counting the results for the different unique identifiers. Countifs has poor performance for large data sets, but you can sort using the identifiers then make a column that adds up the number of rows (or, the number of rows with a particular outcome) since the last change in the identifier, and then filter the rows for the last row before the change in the identifier (be wary of off-by-one errors). Then copy the result (values only) to a new sheet.
This also reduces the number of rows, though not enormously in this case.
Of course, in this case, it turns out that timing was relevant, not for outcomes but only for the ingredient selection (so I would have had to reconsider this assumption to figure out the ingredient selection).
Perhaps, but don't make a virtue of not using the more powerful tools, the objective is to find the truth, not to find it with handicaps...
I'm obviously seeking out more powerful tools, too - I just haven't got them yet. I don't think it's intrinsically good to stick to less powerful tools, but I do think that it's intrinsically good to be able to fall back to those tools if you can still win.
And when I need to go out and find truth for real, I don't deny myself tools, and I rarely go it alone. But this is not that.
You don't need to justify - hail fellow D&Dsci player, I appreciate your competition and detailed writeup of your results, and I hope to see you in the next d&dsci!
1
1...Lorxus managed to get a perfect score with relatively little in the way of complicated methods/tools...
I have struckthrough part of the previous comment, given the edit. I need no longer stand by it as a complaint.
First time poster/D&D Sci player - thanks so much for making this! I'm a complete novice at data science/stats, mostly just the usual sort of intuitions you get from the occasional Excel munging at $corporate_job. I just discovered D&D Sci a day or two ago, and picked this one for my first puzzle from the index because it was marked as high-quality and middle-of-the-road difficulty.
While I may not have much DSci experience, I do have a lot of experience solving programming puzzles in a programming language called APL - think of it like NumPy (so array/tensor oriented) with a longer/weirder heritage that you primarily interact with by typing (vaguely runic?) symbols. It's REPL-based, so it's quick to try different ideas, and it has a long-standing reputation as being good for "data exploration". I'll also say that if you want to match the aesthetics of casting data-analysis spells for an archmage, typing inscrutable wizard-ish symbols into a REPL feels on-point. :)
As an example, assume we've already defined a vector of strings r that contains the unique result names, and a vector of boolean matrices rpt ("result partitioned table", naming things is hard) where each matrix consists of the rows in the input filtered to the corresponding result. The following line
- AND-reduces each matrix in
rptto a boolean vector where1means an ingredient was used in every row (call these "required ingredient pairs") and catenates these into a single matrix. - Constructs a larger boolean matrix where each row is a possible combination of required ingredient pairs.
- For each matrix in
rpt, analyzes each row and determines whether it contains a required ingredient pair combination. Calculates the percentage of rows that contain a required ingredient pair combination, and renders this info as a 2-column matrix where the left column consists of the unique result names.
r,⍪100×rpt{(+/÷≢)∨/4=+/⍺∧⍤1⍤1 2⊢⍵}¨⊂x⌿⍨4=+/x←,[1 2]∨⍤1⍤1 2⍨↑∧⌿¨rptAs might be expected, this gives 100% for "Mutagenic Ooze" and 0% for "Acidic Slurry".
APL is also quite fast (especially on large boolean matrices!), and the above line returns in 47ms on my machine. Nice to be able to try a bunch of random analyses quickly.
I recorded my live solution of this and posted it on my APL-focused Youtube channel here in case you want to see a DSci novice flail around for ~2 hours in this weird language. If there's some rule against links or self-promotion or something (especially for a fresh account), let me know and I'll remove the link. Actually, I'll take feedback on any part of this post. :)
Anyway, on to the data analysis journey:
A simple first thing to do seemed to be partitioning the rows by result, allowing me to do a bunch of result-specific analyses. Looking at the average ingredient count for each result, I noticed that "Mutagenic Ooze" and "Magical Explosion" had the highest avg ingredient counts, and "Inert Glop" and "Acidic Slurry" had the lowest. Hrmm...
Also, partitioning the "Success?" column and looking at unique results for each partition reveals that, indeed, the wizard has 4 unique failure modes, and a bunch of potions that constitute a successful attempt.
Next, I looked at at the percentages for occurrences of each ingredient for each result, sorted from most-common to least-common. This showed that each potion had 2 ingredients that occurred 100% of the time, so from this point forward we have the concept of "required ingredients".
Also, no failure had an ingredient that occurs 100% of the time, meaning that adding an ingredient in and of itself doesn't automatically trigger any failure.
Scrolling through the data, I noticed that the most common ingredients for "Magical Explosion" were all "fantasy ingredients", while the most common ingredients for "Inert Glop" were all "real ingredients". I got a bit lucky here (or maybe the data was constructed this way?) that the sorted occurence percentages exactly partition the ingredient list into fantasy/real - if there had been any overlap, I might not have noticed this. Going in blind, I was kind of thinking whether there was some subset of ingredients that would always (or at least strongly correlate) with some failure, but... uh...
+/!∘24¨⍳24
With 167,77,215 possible subsets this would have taken a while to analyze. So, lucky that I noticed this!
With the hint of fantasy items, I was able to look at the rows for each result and see the percentage occurence of each result for each count of included fantasy items. This laid out the rules for avoiding "Magical Explosion" and "Inert Glop", so from this point I knew I wanted 3 fantasy ingredients in addition to the 2 required ingredients for Barkskin Potion.
Still don't know what causes "Mutagenic Ooze" or "Acidic Slurry", so next I looked at percentages of occurrences of pairs of ingredients for each result (being reluctantly ready to look at 3-sets, 4-sets etc.). Looking at the big matrix of data, the number "0" stood out to me, and focusing only on those we see there are a bunch of pairs that never occur for "Acidic Slurry". Noting that the required ingredients for Barkskin Potion were in that list, I was able to match it to the list of required ingredients overall. So, now we know that to avoid "Acidic Slurry", you just need to include some required ingredient pair.
Seeing as how "Magical Explosion" and "Inert Glop" are kind of opposites, I wondered if "Mutagenic Ooze" was the opposite of "Acidic Slurry" in some way. I messed up the analysis here - idea was to check occurrences of all possible pairs of required potion ingredients (basically my sample line above), but I did it wrong in the heat of the moment. Still, I was able to see that each pair of required potion ingredients occurred at least once for "Mutagenic Ooze", and this didn't happen for any other result except for "Magical Explosion", so I figured it was close enough to being an opposite of "Acidic Slurry" and I was just missing something.
So, time to formulate a guess! Take the 2 required ingredients for Barkskin Potion, add 3 fantasy items that the mage has available, and avoid including required ingredients for any other potion. Checked the solution page and... success!
Thanks again for putting this together, it was fun! I may do some other puzzles in the series, it's probably good to build this type of skillset in any case. :)
Success indeed, young Data Scientist! Archmage Anachronos thanks you for your aid, which will surely redound to the benefit of all humanity! (hehehe)
This is a follow-up to last week's D&D.Sci scenario: if you intend to play that, and haven't done so yet, you should do so now before spoiling yourself.
There is a web interactive here you can use to test your answer, and generation code available here if you're interested, or you can read on for the ruleset and scores.
RULESET
There are two steps to brewing a potion:
STEP 1: MAGICAL POTENCY
Any ingredient that doesn't exist in the mundane world is Magical, while any ingredient that exists in the mundane world is not:
The first step of potion-brewing is to dissolve the magical potency out of the Magical Ingredients to empower your potion. This requires the right amount of Magical Ingredients: too few, and nothing magical will happen and you will produce Inert Glop, while too many and there will be an uncontrolled Magical Explosion.
If you include:
If your potion got past this step OK, move on to:
STEP 2: DIRECTION
Some ingredients are used to direct the magical power into the desired resulting potion. Each potion has two required Key Ingredients, both of which must be included to make it:
*Well. Sort of. See the Bonus Objective section below.
Some ingredients (Angel Feather, Crushed Sapphire, Faerie Tears and Quicksilver) aren't Key Ingredients for any potion in the dataset. Angel Feather and Faerie Tears are nevertheless useful - as magical ingredients that don't risk creating any clashing potion, they're good ways to add magical potential to a recipe. Crushed Sapphire and Quicksilver have no effect, including them is entirely wasteful.
If you've gotten through Step 1, the outcome depends on how many potions you've included both the Key Ingredients of:
So, for example, if you brew a potion with:
STRATEGY
For best success, you needed to:
This was made trickier by the fact that you had only 4 Magical Ingredients available (Demon Claw, Giant's Toe, Troll Blood and Vampire Fang), and that Troll Blood and Vampire Fang were the Key Ingredients of a Regeneration Potion (and hence you could not include both of them).
Nevertheless, you could get a 100% success rate by:
BONUS OBJECTIVE
There was a secret bonus objective hidden in the dataset. It could not be found purely with math on the dataset, it required you to think about the ingredient names and the patterns you were seeing.
On examination of the Key Ingredients of each potion, most potions had Key Ingredients that seemed in some way vaguely metaphorically linked to the potion's effect:
There were two apparent exceptions to this, however.
This was not, in fact, true.
The Key Ingredients of a Barkskin Potion are in fact Beech Bark and Oaken Twigs, and the Key Ingredients of a Necromantic Power Potion are Crushed Onyx and Ground Bone.
Archmage Anachronos is not trying to brew a Barkskin Potion. He is trying to brew a Necromantic Power Potion in order to conduct a Dread Necromantic Ritual during the upcoming magical conjunction.
Not wanting to reveal this, he's swapped around 'Barkskin Potion' and 'Necromantic Power Potion' in his records of results, telling you that every recipe that actually produced a Barkskin Potion produced a Necromantic Power Potion and vice versa.
simon realized this, I don't think anyone else figured it out independently (as opposed to seeing his comment mentioning it).
DATASET GENERATION
Archmage Anachronos's algorithm for deciding which ingredients to use is as follows:
The main relevance of this algorithm to players was that some combinations of ingredients, especially ones that could produce potions and especially ones that could produce multiple different potions, had been brewed many more times than would happen under pure randomness.
LEADERBOARD
This scenario was intended to be relatively straightforward, and bearing this out we had several perfect scores:
Congratulations to all players! Particular congratulations to simon for figuring out the Bonus Objective! Maniacal cackling to all other players!
REFLECTION & FEEDBACK REQUEST
The goal I was shooting for with this scenario was to reward people for actually paying human attention to patterns, rather than just hoping for [YOUR FAVORITE ALGORITHM HERE] to solve all your problems for you.
The Bonus Objective was obviously written to require out-of-context thought. I also tried to set up the scenario to be amenable to thinking about what patterns might mean even with relatively little math: bearing this out, it looks like Lorxus managed to get a perfect score with relatively little in the way of complicated methods/tools just by e.g. thinking about what it might mean that including lots of ingredients led to Magical Explosions and including few ingredients led to Inert Glop (and documented his thought process very well, thank you Lorxus!).
How did this feel from the player end?
As usual, I'm also interested to hear more general feedback on what people thought of this scenario. If you played it, what did you like and what did you not like? If you might have played it but decided not to, what drove you away? What would you like to see more of/less of in future? Do you think the scenario was more complicated than you would have liked? Or too simple to have anything interesting/realistic to uncover? Or both at once? Do you have any other feedback?