Thoughts after poking at the data for a short while:
The most notable thing, to me, is that plausible extrapolation of the distribution of damage from the encounter types listed is far from being sufficient to explain how many ships are lost. Crabmonsters don't usually do a lot of damage and it looks as if maybe 1% of crabmonster encounters are terminal. Demon whales are the most unclear -- the PDF is still increasing at 100% damage -- but it looks as if maybe 100% is approximately the peak, in which case we should expect about as many sinkings as we've seen non-sinkings. Harpies are never known to do >24% damage. Krakens are never known to do >78% damage. Merpeople seem like they have a bit of a right tail, but there's no way to know how far it extends. It doesn't look as if it's likely to account for a huge numbre of sinkings. Nessie has what looks like a normal-ish distribution whose right tail extends a bit beyond 100% but I wouldn't expect more than a few percent of encounters to result in sinking. Pirates are never known to do more than 64% damage. Sharks are never known to do more than 56% damage. Water elementals are always between 74% and 85%. These seem like we might have maybe 10-20 sinkings from crabmonsters, 330 from demon whales, a couple of hundred from merpeople, 25ish from Nessie, and none from anything else. But actually there are 2367 sinkings.
This suggests that most of the sinkings come either from encounters that are way beyond the known distribution of damage, or from entirely different types of encounters we don't have a record of.
For the first option, maybe the demon whale distribution keeps growing rightward for a while longer. Not terribly implausible. Maybe the merpeople have a really long right tail. The water elemental distribution is pretty weird; maybe there are other types of water elemental with similarly narrow distributions living beyond 100%. For the second option, maybe sometimes the gods get angry and smite ships with lightning so that everyone dies, or maybe there's some horribly dangerous geographical feature that isn't on the maps because once you get close enough to see it you're already doomed. Or maybe sometimes the captain or crew decides that they can sail far off course, sell the cargo, keep the proceeds, and live happily ever after.
What can we do about all these? We can completely stop merpeople attacking. Maybe merpeople are a lot of the mystery sinkings. Might be worth it. We can bring krakens and demon whales down by 2% per oar, and weirdly this stacks additively, so buying 50 oars might be worth it since demon whales are our leading "in-sample" candidate to explain the sinkings. Except that we top out at 20 extra oars. Better than nothing, probably worth it. Shark repellent is a terrible idea because sharks are among the nicer encounter options. Rifles for harpy protection seem dumb since harpies are never known to do much, but if we're battling unknown unknowns something that offers to prevent all damage done by harpies is tempting. (Though it's hard to know how we could actually know that they will prevent all damage even in hypothetical cases where the harpies do >4x more damage than they have ever done to any ship that's returned to tell the tale.) Water elementals are pretty common and the foam swords will always save a lot of damage when they attack -- and if indeed there are currently-unknown super-water-elementals doing >100% damage, they might save us.
As for the completely unknown unknowns, maybe we can bring a priest or institute big rewards for foiling mutinies, or something.
(Perhaps I should take the name of the ship as a hint that we're facing the first kind of scenario rather than the second.)
Since my top priority is to survive, I buy off the merpeople (45gp) and fill up on extra oars (20gp). Now I could get rifles (35gp) just in case there are occasional super-harpies, but it's not clear that that's more likely than occasional super-elementals so I'll get the foam swords instead (15gp) since that does much more to reduce damage in the non-fatal cases. I have 20gp left, with which I can halve crabmonster damage (saving, by eyeball, maybe 20% damage on average in the 4% of cases when they attack) or knock 20% off Nessie-and-pirate damage (saving, by eyeball, maybe 16% of damage in 4% of cases when Nessie attacks, plus maybe 12% of damage in 20% of cases when pirates attack); the latter is clearly better damage-wise and might be better sinking-wise too. So: tribute to the merpeople, max extra oars, foam swords, two cannons.
But I am still painfully aware that I don't know where those sinkings are really coming from, and if there were some way to get more information about that (e.g., fit some ships with lifeboats) it would be a hell of a good idea to do it first.
I didn't see any obvious sign of large seasonal or directional effects. There are some long-term drifts; e.g., pirates seem to be becoming less common, harpies, merpeople, sharks, and maybe krakens seem to be becoming more common. I don't think these effects are large enough to change the decisions above. I haven't looked for correlations (e.g., maybe when a ship has been attacked by pirates it becomes more or less likely that the next ship going the same way will be). I do have more detailed notes on what the damage-distributions look like for the various kinds of encounter, but the level of detail above seems sufficient. (They don't appear to vary much with month, year, or direction.)
[EDITED to add:] Hmm, but buying off the merpeople might be risky? It would mean more attacks by other things that might turn out to be potentially ship-sinking. In particular, more demon whales, and our best demon whale countermeasure isn't very good. I'm not terribly sure that paying the merpeople is actually wise.
Looks like you've put in more work than I did below and come up with very similar answers, the one diff I see is that you tried
foam swords as a speculative guess that water elementals are non-linear and explain the extra sinkings/a way to reduce non-sinking-related damage
whereas I just went for
a third cannon for a bit of marginal equity against Nessie
I haven't looked much into dependence on time and direction yet, apart from
noticing that the pirates decline in relative frequency
but, I would like some clarification about whether the 100gp budget is for a single set of interventions we'll use on all trips, or is spent each time on a (potentially varying) arrangement. (edit: I see abstractapplic already responded to such a question from Measure: it's a single set obtained up front and not changed.)
My current thoughts:
From looking at the probability distributions, I mostly agree with gjm; my current recommendations are the same as GuySrinivasan's and Measure's.
Demon Whale distribution scares the crap out of me, and I probably panic buy all 20 oars allowed. I mostly agree that the distribution looks like it could be peaking near 100% damage, but I am not at all confident of this and think it could be consistent with something growing a lot bigger beyond the cutoff. I expect 250-1000 of the destroyed vessels to have been destroyed by demon whales. While it looks like it will be in control before needing 20 oars, I am uncertain enough to value oars pretty high. Budget spent = 20gp.
Merpeople have a very wierd looking distribution. It doesn't seem to tailing off at the end, and so (like gjm) I am very uncertain about what happens after the 100% cutoff. I think (like GuySrinivasan) there's a possibility that it has a multimodal distribution with another peak beyond 100% (not saying bimodal since there even looks like there might be a small peak around 25% distorting the main peak of about 50%, though this could very easily be random). I figure merpeople are responsible for around 200 (assuming no extra peak) or potentially vastly more (assuming an extra peak) of the losses. Merpeople are potentially another solid choice for mitigation imo, unless removing them from the encounter table puts in demon whales in as the substitute. Budget spent = 45+20=65gp
I do not agree with gjm that only about 1% of crabmonster encounters are terminal. The distribution seems to be tailing off very slowly, visually more or less consistent with a triangle-shaped distribution. A simple linear extrapolation would suggest a few percent which I would take as a lower bound. But it might not be linear, but slowing in how it tapers off, so it might be much much more than this. For all I know (apart from the finite number of sinkings) it might not even sum to a finite value. On the other hand, we only really care about crabmonster attacks that do less than 200% damage, since the only relevant intervention reduces damage by 50%. I estimate that between about 50 and about 250 of the destroyed vessels to have been destroyed by the relevant part of the crabmonster damage distribution, with potentially unlimited numbers destroyed by crabmonsters outside that range. Despite the expected max of about 250 mitigatable losses I consider arming carpenters a pretty solid choice for 20gp. Budget spent = 20+65=85gp.
Nessie looks like a pretty straightforward distribution where I assume about 10-20 losses are from the bit of the distribution we can't see. A single cannon (which is all that we can afford after the other purchases) should suffice. Budget spent = 10+85=95gp.
Conclusion: 20 oars (20gp) + pay off merpeople (45gp) + arm carpenters (20gp) + one cannon (10gp), total 95 gp spent. This is the same plan previously recommended by GuySrinivasan and Measure.
Nothing else looks like it can kill us, unless e.g. some bimodal distribution has one of its humps located entirely within the >=100% zone.
However, stepping out of the pure data analysis and into reasoning about the fantasy world, it seems strange that pirates would bother to attack us if they only ever do 64% damage. They are intelligent, after all. Maybe they run away if in a hard fight, not sticking around to do more damage than that, and take over the ship entirely if they win? I might consider a second cannon (also provides extra insurance in case nessie's distribution extends further than expected). Dropping 3 oars would provide the funds, and will probably still be enough for the demon whale.
(and...also anticipated by Measure on the pirate theory).
I just realized I may have misread things?
Suppose a ship was about to take damage such that our records would log it as "80%".
If we "halve" that damage, I think our records would read "40%".
If we "reduce by 10%" that damage, would our records read "70%" or would they read "72%"?
General:
I didn't notice any variation over years or months except that in the last five years Pirate attacks were less common and did slightly less damage. Considering we're making voyages in both directions over ten consecutive months, such variation shouldn't change the analysis much anyway. The most important factors are the distribution of possible damage amounts for each type of encounter and the relative frequency of each type.
Encounters:
We can ignore everything except Demon Whales, Crabmonsters, Merfolk, and Nessie since those are the only lethal threats.
Interventions:
Allocations:
There was a choice at the end between [Merfolk tribute + 1 Cannon] or [Foam swords + 3 cannons]. I'm eyeballing that without intervention Merfolk and Nessie have about equal share of lethal encounters and 1 Cannon should significantly reduce lethal Nessie encounters. Eliminating Merfolk encounters does slightly raise the chance of encountering a Demon Whale, but overall I think it's still an improvement.
After reading gjm's point about unexplained excess sinkings, I'm less confident in my choices. I had briefly looked at them at the beginning of my analysis and concluded that most of the unknowns were Demon Whales, but Super-Water-Elementals seem like a real possibility as well considering their unusual damage distribution (and thematically that makes more sense than Super-Harpies). If I could have a do-over, I'd give up arming the carpenters to get the foam swords and a second cannon.
As long as we're considering things thematically, Pirates could probably board and commandeer a vessel without doing enough damage to risk sinking it, and they're the most common encounter, so maybe some of the excess unknowns are successful Pirate attacks?
Evidence against that theory is that
pirate attacks have changed somewhat in frequency, but ship-sinkings don't seem to have.
Possible counter-evidence:
Pirates have a bimodal distribution (around 20% and 40% damage) and only the 40% part of the distribution seems to have declined. So, this looks like two different populations and theoretically, the 20% pirates could be the strong, smart pirates who win a lot and back off early if they won't get an easy win, while the 40% pirates could be weak, stupid pirates who go all out every time.
Still all totally speculative of course.
There is a chance that the extra missing ships come from unknown unknowns. I will spend gold as if they do not.
I give the merpeople 45gp tribute, reducing my absolute risk by 5.6% per trip.
I outfit the lower decks with 20 more oars for 20gp, reducing my absolute risk by 4.9% per trip. All from demon whales.
I purchase one cannon for 10gp, reducing my absolute risk by 0.1% per trip; Nessie can't sink me.
I arm the carpenters for 20gp, reducing my absolute risk by 0.2% per trip; crabs can't sink me.
Altogether I spend 95gp to reduce my risk from 10.6% per trip to 0% per trip, and report that costs may be saved if desired by not really telling anyone about cannons or carpenters, if the value of a successful voyage plus the cost of replacing a boat and crew is greater than about 10000 gp.
My method was to inspect graphs, noting that most encounters do not sink ships, noting that only the kraken seems to have any time-based concerns but that krakens do not sink ships and that I'll likely take the full insurance against them anyway due to demon whales, and noting that crabs appear to have some sort of simple linear decline, Nessie is likely normalish (I'm guessing something like 4d12+60), lots of things are normal and some look bimodal, demon whales are probably normal - fit - 109+-13ish, and merpeople are VERY SCARY because if they're bimodal then the second hump kills a lot of sailors. So then I estimated how many ships sink due to crabs (42), Nessie (23), and whales (1070), leaving 1135 from unknown causes. PROBABLY THE VERY SCARY MERPEOPLE.
I've put my attempt in my Github but I'll try to summarise it here.
Initial thoughts:
Edit: Changed to Rot13 as I failed on spoiler tags
V fgnegrq ol ybbxvat ng gur qngn naq V svtherq gung gur qvfgevohgvbaf cebonoyl jrer dhvgr avpr. Sebz ybbxvat ng gur qngn gurer frrz gb or abguvat rfcrpvnyyl jrveq tbvat ba jvgu qverpgvba, zbagu be gvzr. Gurer ner n srj abgnoyr guvatf gung ner n ovg yrff avpr guna gur erfg.
Cvengrf ner ovzbqny naq gurer bar bs zbqrf ner erqhprq va vagrafvgl nsgre nccebkvzngryl 62 jrrxf. Jngre ryrzragnyf ner havsbez, juvyr zbfg bs gur erfg ybbxf yvxr gurl pbhyq or pncgherq ol Tnzzn qvfgevohgvba. Nf fhpu V nffhzrq gung V pbhyq zbqry rirelguvat nf n Tnzzn qvfgevohgvba, rkprcg cvengrf gung jrer n Ovabzvny bire gjb Tnzzn naq jngre ryrzragnyf gung jrer havsbez naq gura n Zhygvabzvny sbe jung rapbhagre.
Ng svefg V gevrq n shyy Onlrfvna nccebnpu gb zbqry sbe gur cnenzrgref ohg ZP fnzcyvat guvf tbg n ovg gebhoyrfbzr naq V qvqa'g jnag gb chg va gur gvzr gb qb vg cebcreyl. Vafgrnq V qvq n ebhtu ZYR rfgvzngr jrer V nffhzrq vaqrcraqrapr juvpu grpuavpnyyl qvqa'g ubyq nf gurl funer gur haxabja qngn naq V unir nebhaq 700 qngn cbvagf sebz gur haxabja ynoryf gung jrer hanyybpngrq. Arireguryrff, V abj unq fbzrguvat V pbhyq hfr gb frnepu sbe gur orfg fbyhgvba.
Gelvat vg bhg vg qbrfa'g ybbx tbbq, hfvat nyy gur vagreiragvbaf yrnqf gb fbzrguvat gung unf nebhaq 3 % punapr bs fheivivat gur phzhyngvir qnzntr sebz gra pbafgvghgvir gevcf. Vs guvf vf gur pnfr V jbhyq gel irel uneq abg gb tb.
Gur orfg nggrzcg V sbhaq (gubhtu gur engr jnf fb fznyy fb V qvqa'g trg n tbbq pregnvagl sbe guvf) jnf gb hfr funex ercryynag, cnl gur zrecrbcyr n gevohgr naq ohl 15 rkgen bnef sbe gur erfg bs gur tbyq. Gur erfhyg jnf gura
* Punapr bs fheiviny 0.25 %
* Nirentr qnzntr ba fheivivat fuvcf 77 %
* Tbyq yrsg 0
V unira'g purpxrq guvf gubhtu, fb gurer zvtug irel jryy or fbzr oht gung tvirf zr jebat erfhygf. Gurfr vagreiragvbaf jrer pbagenel gb jung V rkcrpgrq ng gur fgneg. Gura V jnf guvaxvat nobhg ubj gur erqhpr gur pregnva qrngu rapbhagref naq gung ybj evfx rapbhagref jrer tbbq. Guvf vagreiragvba frrzf gb or gb vapernfr gur punapr bs gur fznyyrfg evfx rapbhagref (rira vg guvf vapernfrf uvtu evfx rapbhagref nf jryy) naq ubcr sbe gur orfg.
Ships are fully repaired every time they make port.
Ah, I missed this. Now things look much simpler.
Fixing this mistake, I see that now the best result for my model is from buying
Woodworker weapons
Merpeople tribute
20 extra oars
1 extra cannon
with results
Survival rate for all 10 trips was estimated to 95 %
Average damage 41 % per trip were you survive
95 gold spent
Is the quip about Vaarsuvius' Law intended to be taken seriously here? I.e., are we to assume that without exception every voyage involves exactly one encounter? (And that that's why there are no entries in our records saying "no encounters"?)
Similarly, are we to assume that every voyage does have an entry in our records?
All your assumptions are correct, with the debatable exception of the last one: you have a record for every voyage
My guesses after some pretty dirty analysis:
By looking at the data, squinting at some sumcurves, fitting some very speculative lines and sacrificing a pidgeon to the diviners I guess that of the 2300 ships that were lost were caused by:
1000 were due to Crab people (whose damage distribution doesn't look like any of the others', so I am guessing 50% of their attacks do more than 100%)
700 were due to Demon Whales (Looks like at least 2/3 of their attacks do more than 100% damage)
300 were due to merpeople (their damage might have 10% above 100)
200 were due to Nessie (seems around 15 - 20 % of damage above 100)
The last two are the most imprecise. None of the remaining encounters look like they could have any serious chance of sinking a ship, so let us leave the last 100 or thereabouts to them.
Judging by this I will suggest getting Armed Carpenters, 20 Oars and either 3 cannons or bribe the merpeople. I wil swing with the cannons, since Varsuvius law tells me that fewer merpeople means more other monsters, potentially Demon Whales. At last, if less damage is more important than saving money we could get foam swords, they won't save our lives but might save on the ship repair costs in the long run.
Spoiler protection
Known harpy, kraken, merpeople, and shark attacks seem to be on a gentle rising trend.
Known pirate attacks fall substantially from 1401 onwards, but are still the most common. Despite this the year and month doesn't appear to have much impact on losses, and there is no obvious correlation between the month and the frequency of a given attack.
The direction doesn't appear to have much impact either.
Looking at how the damage is distributed across known encounters
- Harpy peaks at around 10-15 then falls away, none have done more than 23
- Kraken peaks at around 55 then falls away, none have done more than 78
- Pirates peaks around 15-20, and falls away, none have done more than 64
- Sharks peaks at around 20 -25 then falls away, none have done more than 56
- water elemental is strongly peaked around 80, but falls away rapidly, none have done more than 85
Suggests the only serious threats are:
- crabmonsters
- demon whale
- merpeople
- nessie
All of these peaked below 100 percent damage with 1 exception the demon whale, which suggests this is the biggest threat. With this in mind I should definitely invest in all 20 oars for 20 gp.
Nessie is responsible the causing the second highest number of incidents with very heavy damage, so investing another 30gp in cannons looks sensible.
Merpeople are the next biggest threat, but bribing them could make matters worse if it leads to more dangerous encounters. There were 2678 known merpeople encounters. If we assume they are spread evenly across the other 8 encounter types that means at least 334 demon whale attacks. Looking at the distribution of damage caused by the demon whale attacks it looks like the peak is likely to be well over 100 percent damage, and they represent a very real threat even with all the extra oars. It is not at all clear if this would represent a net reduction in risk, and when I consider that the high cost would prevent me investing in crabmonster defenses the benefits of this look dubious.
The crabmonsters are the only other encounter that do a large amount of damage, so arming the carpenters for 20gp looks sensible.
For the remaining 35 gp investing money to reduce a minute risk to my life and impressing the admiralty with less damage done to the ship seems a lot more attractive than minutely increasing the risk to my life and impressing them with saving money. Of the lesser threats the water elementals do the most damage, so if any of them are going to get me it is most likely to be them, so I will invest the last of the gold in the foam swords.
Final decision:
- 20 oars @ 20gp
- 3 cannons @ 30gp
- Arm the carpenters @ 20gp
- Foam swords for the deck crew @ 15gp
- 15 gp under budget
Is it assumed that we spend some amount on interventions up front and then undergo all ten voyages with that setup? Are repair costs unaffected by interventions?
Yes to the first question.
To the second: you can aim to reduce repair costs with your interventions, but since they won't come out of your budget that's more of an optional extra for further impressing the brass.
Spoilered:
Looking at the 99% and 98% damaged ships:
These look fairly similar, and suggest that those are the threats that are likely to be most dangerous (until of course some more serious threat is reliably unsurvivable so we don't find out about it, or some threat deals non-linear damage...but let's ignore that for now. I was assured in Linear Algebra class that the real world never works like that.)
Given this:
Adding up Armed Carpenters (20) + Tribute to Merpeople (45) + 20 Oars (20) + 3 Cannons (30) makes 115gp, which we can't quite afford. My best guesses:
Only four creatures have been known to do significant damage, implying that they've probably sunk some ships:
All attacks by these creature appear to be roughly the same relative frequency on a yearly basis.
Demon whales look the scariest, let's max out at 20x oars: -20% for 20 gp.
Nessie has probably taken out a few ships but she rarely does more than 90%. Definitely worth a cannon: -10% for 10 gp.
Merpeople and crabs both have weird long tails. Let's skip the merpeople entirely for 45 gp.
What's sinking all these ships? Is it...murdercrabs? Let's arm our carpenters, -50% for 20 gp.
We've never seen any of the other creatures come close to sinking a ship, so we won't worry about sharks with lasers on their heads or mega-harpies. I have 5 gp remaining, which I keep as a fee for my services. Alternately, I'd consider trimming off 5 oars for a second cannon, but I'm not sure it's worth it.
If I'm using all the gold, I would spend 20 on arming carpenters, 45 on mermaid tribute, 15 oars and 2 cannons. If I want to save gold, I would avoid the cannons. I think the big risks are evenly spread between demon whales, merpeople and crabmonsters. With some risk from nessie.
... Ah, no, those are the "100%+" lines, right? So for those we know that the ship sustained enough damage to sink it, we don't know what amount of damage it "would" have suffered if it had been tough enough to take more than 100%, and (after looking briefly, haven't looked more carefully yet) in all these cases we don't know what the cause was, only that something caused the ship to sink.
Vaarsuvius’ Law (“every trip between plot-relevant locations will have exactly one random encounter”)
I appreciate the Order of the Stick reference!
So, Oars, Bribes, one cannon, and then either Axes or another 2 cannons. I'm guessing the cannons will do more, based on speculating the extrapolations of Crab vs Nessie.
This post should be spoilered. Use >! to hide information that could tip somebody off, which in this case is everything.
You were prepared for gratitude, a commendation from the Admiral, your own department, parades in your name. You were also prepared to hear that your ‘list of helpful suggestions for ensuring supply ships survive random encounters’ was an impudent insult to the collective intellect of High Command, and receive a public execution for your trouble. What you weren’t prepared for was what happened: being allocated a modest stipend, assigned to a vessel, and told that if you’re so clever you should implement your plans personally.
You have 100gp to spend, and your options are as follows:
You’re completely confident in the effectiveness of your ideas, but much less confident that you know which combination would make the best use of your limited budget. To investigate this angle, you’ve procured a record of random encounters encountered by the ships travelling your assigned route; unfortunately, it’s missing some important information for the ships that sank, due to everyone who could fill in those details being dead.
As you board the Gray Swan (why do they give these ships such charmingly unique names when they’re all built and operated identically?), it occurs to you that this might have been intended as an execution after all. The dataset suggests that without any of your clever plans, the survival rate for a journey along your route is a little below 90%, and the Gray Swan is scheduled to make ten trips – five northbound voyages, five southbound – in quick succession. Hopefully this indicates nothing more than your superiors wanting to test your interventions very very thoroughly.
Your top priority is to save your skin. Secondary priorities are minimizing total damage taken and spending as little gold as possible, to impress High Command and return to their good graces.
What will you do?
(Notes:
I’ll be posting an interactive letting you test your decision, along with an explanation of how I generated the dataset, sometime next Monday. I’m giving you a week, but the task shouldn’t take more than a few hours; use Excel, R, Python, a priori knowledge, or whatever other tools you think are appropriate. Let me know in the comments if you have any questions about the scenario.
If you want to investigate collaboratively and/or call your decisions in advance, feel free to do so in the comments; however, please use spoiler tags or rot13 when sharing inferences/strategies/decisions, so people intending to fly solo can look for clarifications without being spoiled.