Fisher Information is hugely important in that it lets you go from just treating a family of distributions as a collection of things to treating them as a space with its own meaningful geometry. The wikipedia page doesn't really convey it but this write-up by Roger Grosse does. This has been known for decades but the inferential distance to what folks like Amari and Barndorff-Nielsen write is vast.