I have lots of thoughts about software engineering, some popular, some unpopular, and sometimes about things no-one ever talks about.

Rather than write a blog post about each one, I thought I'd dump some of my thoughts in brief here, and if there's any interest in a particular item I might expand in full in the future.

Context

I loved Jamie Brandon's series Reflections on a decade of coding. It's been nearly a decade since I first learnt to code, so I think it's about the right time to write my own.

He starts off by pointing out that advice has to be taken in the context of where it's coming from. So here's my background.

I have 8 years experience as a backend developer at companies ranging in size from a 30 person startup to Google. I have never worked on a frontend^[1], mission critical software, or performance critical software. All of our products have had users, and some level of scale, but never hyperscale.

I have also worked on a number of open source libraries, including the C# compiler.

If your current circumstances don't match that, it's likely my learnings won't be applicable to you.

I've also focused on areas where I think I have some insight, and avoided areas where I don't trust my own opinion any more than the next developer's.

All rules are made to be broken

Each of my thoughts here are pointers in a particular direction. None of them are universal, and all of them would require a full length blog post to describe exactly where they apply or do not. Instead treat them more as directional: compared to where I think the average developer stands on the issue, I want to move them more to the left or the right.

Topic 1: Programming Languages

a. Languages have advantages. There are better crafted languages and worse crafted languages. Plenty of outages have happened because you used the wrong language. But it is extremely rare for a product to succeed or fail based on the language, so ultimately make a sensible decision and move on.

b. The biggest factors affecting which language to choose are:

• The deployment model - if you want a light self contained binary you'll want c/c++/go/rust etc. Not C#/Java/python/node etc. If you want to run on the browser, java/typescript is usually your best bet.
• Which libraries are available. Often you're forced to use a particular library written in a particular language and that decides the issue.
• Which languages your team is familiar with.

c. Once that's whittled down your options, there's issues like memory safety, typing, fp, threading model etc. Everybody likes to go on about these endlessly, and there are plenty of situations where these are important, but they are way less important than the previous issues.

d. Going against the grain imposes high costs. Sure, you can write machine learning pipelines in Java, or compile c++ to wasm for the web. But you will find a lot of libraries missing, and much less documentation or prior art. In very particular situations it's worth it, but in most cases, swallow your pride and use that garbage language you hate so much.

e. Programming languages have particular styles. Linq is all the rage in C#, but trying to do the same in golang is far less popular, and is looked down upon. It also works less well. For the sake of other developers working with you, try to stick to the standard styles for that programming language.

Topic 2: Microservices

a. In all but the largest services, microservices are likely to cause a significant performance degradation. The advantage of using less RAM is easily outweighed by the cost of going over the network all the time.

b. Microservices make things far harder to debug, since you need to trace calls across multiple services.

c. Microservices makes code harder to develop, since testing requires you to spin up multiple services.

d. Microservices makes deployments harder, since each microservice might be at any of a number of versions, and you need to work out whether any member of the cartesian product of all these versions is incompatible.

e. Unless you invest a lot of thought and effort into your architecture and setup, microservices are likely to degrade rather than improve reliability because there are more moving parts which can go wrong.

f. Tooling can mitigate some of the issues above, but only somewhat, and requires significant investment in integration and education.

g. It is often (but not always) a lot easier to scale a monolith horizontally than to rewrite it to use microservices.

h. The biggest advantage of microservices is that it encourages you to split your code into vertical slices, with clear boundaries between parts. It's much easier to monitor where code crosses the service boundary and make sure those are actually necessary and well defined. Actually running the code as microservices has far fewer advantages.

h. Microservices are great if you have a clearly defined microservice, which you own, you have users of from outside your team, and you offer your users a well defined API with good versioning guarantees. Basically you treat your microservice like a product.

i. Sometimes microservices are necessary in other situations. But they will cause you pain.

Topic 3: Methods/Functions

a. Far better to duplicate code and then merge it later, than to share code, discover that different callers need slightly different things, and then try to modify it to be everything to everyone. DRY is overrated.

b. If a method can be made into a pure function, without any particular loss of performance, make it pure.

c. When choosing where to extract smaller methods from a very long method, do so where the smaller methods will be pure, take as few parameters as possible, and do something which can be clearly defined.

d. It is fine for methods to be long. Especially if you are updating state, often better to update the state in multiple places in one method, than spread out across multiple methods. However, extracting out the bits that don't update the state can make it easier to see those bits that do.

e. It's perfectly fine to use mutation inside a "pure" method, just so long as from the perspective of the caller it's pure.

Topic 4: Architecture

a. Solving a specific problem in a specific context allows you to take shortcuts and skip solving the difficult parts of a general problem. Don't jump to solving a general problem when you don't need to for your use case.

b. At the same time, thinking about how to solve the general problem can give you ideas for how to architecture the solution for your specific problem. When implementing it though, feel free to take shortcuts the general solution wouldn't allow.

c. You might think you've written a generic library suitable for executing arbitrary dependency graphs, but no plan survives first contact with the enemy. If you want your solution to be shared by other people, be prepared to maintain it and continuously upgrade it, in ways that may be detrimental to your own use case. If you don't want to do that, don't expect it to be suitable for other use cases.

d. This even includes things like - if you only use it for your use case, you can log wherever you need to, using whichever log library you usually use. If you want to share it, logging becomes a pain in the ass.

e. Structuring your application into vertical slices, each of which has as few dependencies as it needs and often talks directly to underlying services it needs, tends to work better than an onion style architecture with e.g. a thick shared data access layer which everything uses. A thick data access layer has to be everything to everyone, but most components don't need all that. The vertical slices all talk to each other via well defined APIs instead of arbitrarily calling into each other. This allows each vertical slice to choose the architecture that makes the most sense for itself.

Topic 5: Testing

a. Even when I think something is so simple it clearly doesn't have any bugs, it usually has bugs. Writing tests is the best way of catching those bugs.

b. Manual testing is important for sanity checking your work, but only automated tests can provide thorough coverage and future proofing.

c. Tests should ideally be written at boundary points where they will only break if the specification of the tested component breaks. I.e. refactoring should usually either remove tests completely because you've deleted the subcomponent, or leave them unchanged. Only adding new features should regularly break tests.

d. For large complex components, it is worth investing a significant amount of effort into making it simple to add new tests. For example, in a compiler, have a testing library where you can write some code, and the tests will automatically generate and store the emitted assembly. Ensure that it is trivial to update all these goldens whenever emit changes. That way it is easy for reviewers to see impacts of your changes.

e. Golden tests are a great testing method where they're applicable. Google has an amazing internal golden library which I'm a huge fan of.

f. Even for smaller components, write your tests in a way that lowers the trivial inconvenience of adding a new test. For example, in golang, consider writing your test as a table test, even if you only want to test one case. That way the next time you make a change, the test doesn't need to be refactored for you to add a new test case.

g. Unit tests vs end to end tests lie on a spectrum. Every component is made up of sub components, and sometimes you test closer to the root, other times closer to the leaves.

h. All else being equal, tests closer to the root (E2E tests) are better because they ensure that all the subparts are wired up correctly.

i. On the other hand, the larger the component being tested the slower the test is likely to be, the more flaky its likely to be, the more likely you are to need to use fakes or mocks, and the harder it is to know what contributed to the failure.

j. Also, some particular edge cases are hard to cover E2E, and easier if you test just the relevant component.

k. IMO if you are able to write E2E tests that are deterministic, reliable, use genuine implementations of all interfaces, and are reasonably fast it is best to put most of your focus on E2E tests, but with ample use of unit tests as sanity checks, to test particular edge cases, to test well defined and well specified components, to test areas where a subcomponent is complicated and you feel it needs the extra coverage, etc.

l. As said above, E2E tests vs unit tests is a spectrum. So applying the above advice implies focusing your tests at the largest component where they can still be deterministic, reliable and fast.

m. Prefer real implementations over fakes, and fakes over mocks.

n. If a test needs to be significantly updated every time you make a change, and is preventing you from refactoring/adding new features, delete the test. Badly written tests have negative value.

o. If a test uses a lot of mocks and fakes, you're likely testing at the wrong level. The test is likely to break whenever you refactor, and likely doesn't actually check what you want it to check. Try to use real implementations instead of mocks/fakes, or to refactor your implementation to split out the business logic from the external calls, and write integration tests to cover the external calls.

p. Integration tests are often flaky. Increasing the number of tests increases the flakiness, so either invest a lot of effort into reducing this flakiness, or use integration tests judiciously to sanity check your wiring instead of testing your overall logic.

q. Flakiness is problematic because it means you ignore real failures. Work hard to reduce it.

r. If you have an integration test which isn't flaky or slow, e.g. because you're integrating with an extremely reliable and fast service, then the integration test isn't problematic, and beats using a fake/mock.

Topic 6: Code Review

a. It takes just as long to review someones pull request now or in an hour. But if you review it in an hour there's a decent chance they're twiddling their thumbs the entire time. And the longer you leave the code review the greater the chance of merge conflicts. Unless you're in the middle of a flow, code review should almost always be your top priority to unblock other people.

b. The purposes of code review include: education (two way), having a second pair of eyes on the code, knowledge sharing (two way), ensuring quality, and stopping people taking shortcuts^[2]. It is not to impose your personal coding preferences on the other person. Everyone has their own style. Either officially document that something is the team style and get team buy in, or let it pass.

c. It is perfectly fine to approve a large pull request without a single comment, or to leave dozens of comments on a small one. Keep your standards constant instead of looking for something to pick up on and then calling it quits.

d. If you trust the other developer, leave feedback and then approve. You trust them to handle the feedback, so don't block them unnecessarily. You should still consider reviewing their code after they merge it to calibrate your trust.

e. Code review is important. Put effort into it. If you're just an LGTMing machine, you can be replaced with a rock.

Topic 7: What makes a good developer?

a. A talented developer can fluently translate a high level description of an algorithm into code in a language or ecosystem they are familiar with. Mediocre ones look more like they're trying to tie their shoelaces with their eyes shut, constantly trying to work out what they've done and where to go from here. This is true even for talented Junior developers. If you can't write code fluently, practice until you can. That doesn't mean the code will be working on the first try, but the outline will pretty much be there.

b. Being able to keep a large and complicated codebase/architecture in your head is a superpower. If you know exactly which parts of the code do what, and how they all connect together, you will have a tremendous advantage when designing new features, implementing code changes, or diagnosing bugs. I have no idea if this is something you can practice.

c. As you gain experience you should be able to pattern match what you're doing to more and more problems in the past. You should recognise that X is a perfect problem for a relational database, but in Y you're essentially turning a database into a message queue and it's better to use an actual message queue. But this will only happen if you gain experience in a wide variety of areas, and work with more experienced developers. Don't let yourself have 1 year experience 10 times over instead of 10 years of experience.

Topic 8: Career

a. Tech companies pay orders of magnitudes higher salaries for software developers than non tech companies. Get out of a non-tech company as soon as you can.

b. Getting your first job is really hard, but once you have some experience getting the next job is a lot easier. Accept almost any software development job to start off with^[3], then after a year start looking around for something with better pay, a closer location, better working conditions, etc.

c. You generally get larger pay rises by switching between jobs than getting promoted, but the benefits to doing so diminish as you climb higher up the ladder. Still, consider testing the waters every so often to see what you can get.

d. The higher in the career ladder you go the less you'll be judged on your code, and the more you'll be judged on your architecture, your designs, your knowledge, and your product ownership. It's really difficult to advance by just being able to translate design documents into working, tested, well written code. Seek opportunities to write design documents, contribute to product decisions, and to take ownership of large projects, including proactively doing research.

Topic 9: Team structure

a. Whenever you need something from outside your team there tend to be much longer delays than when you need something from inside your team. For that reason try to structure teams such that every team has everything it needs for its day to day work. Some companies have a separate databases team and a separate deployment team who need to approve all changes to the database or run all deployments, and this tends to be a disaster.

b. Within a team, you can either have everyone contribute to a specialised area, with a few members having deeper expertise, or have a dedicated specialist/subteam. IME, the first approach tends to be more effective, but if skill differences are too substantial, the second option might be the only viable choice (e.g. noone on the team has sufficient background in frontend technologies).

1. ^{^}

   To be fair, I did add a feature to a WPF application, my CTO saw my colour scheme, and banned me from touching the frontend ever again.

2. ^{^}

   Sometimes I know that something isn't going to pass code review, so I add tests to it or refactor it even though I can't really be bothered.

3. ^{^}

   But only if its software development. Too many people accept a job as tech support or whatever "temporarily" then find it almost impossible to switch to development.