You probably already know this, but the author of Worm has written another long (though not so long) serial called Pact and is in the middle of another called Twig. If you enjoyed Worm and haven't read those yet, you might give them a try.

Harry Potter and the Natural 20 has a lot of characters optimizing in interesting ways in response to the constraints of their universe, but their universe's laws are a lot less realistic than Worm or HPMOR. The Ethshar books by Lawrence Watt-Evans are not one coherent story but there are a lot of books set in the same world. Miles Vorkosigan has the whole "cunning main character thinks outside the box to beat impossible odds" dynamic and there are a bunch of books that follow him and his life, although the fact that he ages throughout them gives them a pretty different feel than the chronologically much shorter timespan of HPMOR.

There's a subreddit for rational/rationalist fiction that may interest you or have a longer list of suggestions than here. I think the subreddit was www.reddit.com/r/rational