The story of storage — part 1
From time to time I solve system design problems. What is it about? Well… Here, in IT and programming we create systems. Systems that solve problems. We create it by writing code that is being executed by machine and… You know the rest. From time to time we encounter complex systems that solve tons of various problems and live (I mean, being developed, changed and maintained) for long time. Obviously, you cannot just write one single file of code that represents system and solves all required problems. You have to carefully break problems into smaller ones and solve them one by one, then make these small solutions for small problems somehow work together. That is how we create m̶o̶n̶s̶t̶r̶o̶u̶s̶ things in IT, yup.
As you’ve understood, in this article I will talk about system design. The reason I’m doing that is that soon I will publish some of my developments in system design of business applications. The framework that I’ve called Reinforced.Storage. Prior to that, I’d like to explain some problems that it solves and how it does so. In this part I will outline common theoretical knowledge that I’ve got when developed it. Also some common information about system designs for business applications will be provided. So here we go.
Complexity problems
Additional complexity and limitation about large systems is that you (almost) never develop them alone. Nowdays team stays behind any more or less complex system. In this case, the purpose of system design is to declare some “common approach”, some, if you wish, mind pattern that all team members can use to break incoming problems into pieces and imagine in which form they must provide results. Literally, what code, in which form and where they should write to make existing large system to solve one more problem. It is better when all team has unified understanding about it. But reallity is little more merciless: without clear “rules of game” every team member will try to use approach that seems right for he/she. As result, we are getting systems looking like this:
So system design has to be strict and define clear way of how do you add new functionality, modify old one and remove unnecessary. It is good when system design also considers system’s future and provide ways to extend system or migrate to other approaches.
Accompanying requirement here is simplicity. System design has to be as simple as possible in order to be understood even by junior developer. More simplicity — more understanding, more understanding — less mistakes.
By the way, about mistakes. Another problem is coming from bugs, that appear in every software. In complex systems, bugs become literally the force that can stop your business and make you to lose significant amount of money.
How is it related to system design? Well, directly. Good system design must foresee typical mistakes that developers do and, by its nature, reduce conditions that will allow to produce such stupid mistakes. That we call “software stability”. Personally I like semi-haskell approach here: if it builds — it works. We all understand that practically this is unreacheable, but I try to make my designs in way to catch as much mistakes as possible during compile-time. In C# that I use most of time it is like state of art, but I will show some practical approaches to achieve that. Later :)
A note about testing
Another way of ensuring software stability is to write tests. Yup, you have probably read tons of articles regarding various testing approaches — TDD, mocking, property-based testing etc. But this information is practically useless unless your system’s design is not test-friendly. Seriously, nobody will write nor support unit (integration, smoke, sanity etc) tests in your system if every line of test becomes hours of pain and tears for developers, hundreds and thousands of bucks for management.
Consider .NET’s EntityFramework (ver. 6 and below, not sure about EF Core). This is thing to communicate database that provides its own access point to DB: DbContext. Actually it is decorated database connection that, literally, cannot be mocked somehow without creating test database. Naive unit testing here fastly leads to seting up of separate database for testing. Moreover, you can validate your application logic only after calling .SaveChanges() — literally, after writing changes made by your application to DB and querying them you can assert that your logic worked as expected. That is example of non test-friendly design out of the box. That is why people create UnitOfWork, DAL/DAO reference implementations, DDD approaches and other stuff like this around EF — just to bypass this limitation of non-mockable DbContext.
So, besides simplicity and strictness, I’d include requirement for test-friendliness to this ad-hoc system design wishlist. Especially it is important when you develop application for business or commercial product. From my experience — the problem of building test-friendly design is one of biggest headache for business applications developers. We will also touch and discuss this topic.
However we will talk about business applications. According to my experience, regardless of years passed, IT industry still has not developed some common and working approach of how to write business applications. Designing of business applications now is like medieval medicine: “take this herbal pill, hope it helps”, “obtain volf’s tooth and stir-fry it exactly for 34 minutes under full moon then put it onto your display”, “have you heard that all diseases happen because of nerves?”, “people say that some magician discovered holy grails that will gift us eternal life”. No, seriously: “all of our problems come from mutable state! let’s get rid of it”, “you know, asynchronous I/O will solve our performance problems”, “all problems are of monolitic architecture! microservices will save everyone!” etc. And there are always 2 or 3 companies that used suggested approach and it helped. All others live as they can and develop systems as they understand them. So that’s why I decided to summarize knowledge and approaches that work in one single place. But before we will get to code (in next parts) — let’s discuss some theoretical things. First of all we will understand what is actually business logic.
Here comes the business
No single conversation about applications for business happens without mention of business logic. But.. meh… what business logic actually is? Everybody talks about it, but almost no one has real clue what actually meant. And, moreover, what should be meant by this term. Thousands of times I’ve heard sentences like “we have moved our business logic into separate assembly”, “business logic should not be located within MVC controllers”, “DDD groups business logic by business domans”, “business logic in static methods” etc. If you would ask people what they mean by that — most likely they will point to some vague volume of code, full of ifs, fors and foreaches, that somehow uses database and/or other external services. If you’re lucky — then this code will be spreaded among several classes. If you are extremely lucky — it will utilize IoC and interface segregation.
But finally! Does business logic mean “some code that we have written and have to maintain now” then? Obviously no. Come on guys, that does not seem like valid way to define terms. Too many cases fall under this definition.
That is why I propose another way to define business logic:
Business logic: functionality that makes decisions of what side-effects must occur and when
Side-effect: interaction with outer system that leads to changing its state
Outer system: stateful entity that you can not directly control (create, destroy, change state). You can only ask it to do something regarding its state (in common case: without any warranty of your request to be satisfied)
Pretty abstract words, so let me explain.
Technically everything that is not located in piece of RAM that your application directly controls must be considered as outer system. And side-effect, in other words, is when you ask some outer system that is not in scope of “your” memory to do something that changes its (outer system’s) state.
Quite often communication with outer system is being performed according to its protocol. Protocol is system of common terms (often, such terms are merely abstract) and instructions (often, instructions are order-sensible) that you use to communicate with outer system. Simply, it is set of more or less exact rules of how do you “ask” outer system to do things (request) and how do you receive results of your request (response).
Outer systems and side-effects
But instead of talking about protocols, let’s bring some examples of outer systems and side-effects that you can initiate on them:
- Terminal (console). Is the most simple outer system that everyone encounters when trying just to learn computer programming. Nowdays you can ask console to output some text and this action will be actually side-effect initiation. But code that decides what terminal should show and when considered as business logic in this case. In .NET Framework terminal is represented by static Console class that exposes some methods to write text, change font, encoding, terminal size etc;
- File system. I think it would be #2 in the list of most popular outer systems worldwide. Look: usually you do not directly access blocks of HDD or cells of SSD. You ask file system to do something for you in its space of abstractions through communication channel that is represented (in .NET) by FileStream class. But finally you also initiate side-effect: FileStream will ask operating system that will ask file system, that will ask disk driver to do something… etc. All these entities, obviously, are not located in your application’s RAM. You cannot create them from your code nor destroy. You own only communication channel — FileStream: you can create variable of its type, construct it, use its methods, dispose it. But what code do you write to decide what should be written to file? Yup. You need business logic for that;
- Other OS process. Almost no comments here — there are tons of instruments to establish IPC: COM services, pipes, shared memory etc. The fact is that you, obviously, do not own other process nor can destroy it. You can ask it to destroy itself. Or ask operating system to destroy/create it. Or, maybe, ask someone else. But you do not own the process. So, other OS processes are always outer system for you. But again, imagine that you have built all the channels to communicate with other OS process — what would you “say” to it? You have to make decision. So you need business logic;
- Everything that is located behind network. Database, web-service, client’s browser — all these things are something that you do not own, but can send instructions. And these instructions will lead to changing state. Database’s state is data that is stored inside it, browser’s state is UI that user observes. Regarding web services — there could be large zoo of various stateful things that you change hidden behind particular method of service;
Et cetera. I’m sure that you understood and can find more examples of outer system with side-effects by yourself. Pretty clear, eh? Do that yourself in your spare time.
Well, after we understood what are outer systems and side effects — let’s get back to our definition of business logic. So, if you consider this definition then everything will take its place. For instance, you easily could judge what is business logic and what is not (what, in my opinion, is much more practically useful).
What is NOT business logic
Let’s find out some examples of what is not business logic:
- Converting data: you own data in source format in your memory (string containing JSON), you want to turn it into another format (class instance). No interactions, not side-effects. No business logic. Actually business logic happens not when you e.g. deserialize JSON from string, but when you decide that deserialized data will go to DB or to the client side (but do not actually write it there — only decide!);
- Glue code: IoC registrations, fluent configuration of EntityFramework, Automapper, serialization attributes (or other attributes), middleware operations, declaration of configs etc. Here we interact with other systems for sure, but we write code that does not make any decisions. Roughly speaking, we tweak channels of communication with outer systems. But it is not business logic unless you intentionally write software that is responsible of communication channels. Even in this case I would not say that such software contains some business logic. There is logic of course, but it is not related to business somehow;
- Reading data: according to definition of side-effect, reading data is not changing state of outer system that you read data from. Please keep in mind that practically it is not always so: imagine that you have MySQL deployed on your laptop. If you would open 2 mln connections to it that will SELECT 1000 rows each — you naturally change MySQL’s instance state from “running” to “hanging” :) In this article I would not float into this swarm because it is separate topic requiring me to touch cloud of connected technologies and if I will do that — article will end up with pricing comparison between redis server on amazon and corresponding solutions in Azure. Let’s leave it behing the scenes for now. Important thing now is that reading data is not business logic;
- Code that immediately interacts with outer systems. Well man, I see that you call file.Write(...). Then it must be that you exactly know everything that should be written, eh? What? No? Well, it is because you didnt read this article till the end and do not have business logic :)
Conclusion before part 2
Business logic is your application’s core. Now we understand that it must stay aside of side-effects as far as possible. I bet that in most of applications business logic is tightly connected to side effects (e.g. direct SQL in code). In next parts I will try to propose way how we can set business logic free from side effects and where to put it in applications. We already have too much text for first article :)
