Image from the tomaszjaniak blog
Show us your Architecture
It so happens that you have been working on a project for a couple of years and right now you are sitting up to your elbows in the code, the tests are burning, the laptop cooler is making noise, trying to cool the heat of the processor overheated by the compiler, just a few more hours - and the refactoring will be finished. And then we urgently need to drop everything and draw the Architecture of the project. A large client is planned, and in order to close the deal, it is necessary to undergo an audit. And in this case, there is nowhere without Architectural Diagrams. So what if the project is still small, and a dozen of its developers in real work did not need any Architecture there. Here is the code - everything is clear. But there is no way out. It is necessary - then it is necessary.
If you know how everything works, the architecture of any product can be drawn in an hour. And this is usually enough for an audit. After all, the point of auditing is not for outsiders to really understand your product architecture. The whole point of the question is to determine whether the product developers themselves have an understanding of what they are doing. And if you speak confidently enough and answer questions without hesitation, everything will go smoothly. But nevertheless, when this story ends, somewhere in the depths a worm will settle, which will sharpen the question: but still, what kind of Architecture does the product actually have? After all, if they ask, probably others have Architecture. And if we do not have it, this is clearly a mess. There is a great desire to do everything well in order to be responsible for the Architecture next time as it should be.
How do you do it right? You need to draw diagrams, describe features in documents, and then keep all this up to date. Then it will be Architecture. But, having presented the scope of work, it quickly comes to the understanding that all this is not worth the time spent. Here is the code - everything is clear. And if another client comes, then in another hour it will be possible to draw a new diagram.
And after a few years, you may find yourself in the opposite situation, and the documents that will describe the architecture of another product will not be any better. Of course, it is useful to know that others have a very bad architecture too. But when the size of a project starts to measure in millions of lines of code, and the number of developers in hundreds, the question of the need for an Architecture still becomes significant.
Here, within the framework of the workflow, various Architectural diagrams begin to appear, Architectural Proposals begin to write on features, all this is discussed, reviewed, validated, approved and, of course, becomes outdated even before these processes are over. There is no need to say that such documents reflect the real structure of the application code. And when it comes to implementation, it happens that the developers don't read these documents at all. Or something completely different from what was written is being implemented. Someone misunderstood someone, or it was urgent to do it, and there was no time to discuss.
Maybe everything is as it should be. Perhaps, Architecture is the Bureaucracy around speculative documents, divorced from reality? But intuition tells us that after all, no. So what, then, is "Architecture"?
Architecture is ...
What happens if you ask the question "What is Architecture?" ten professional developers? If you don't pry into Wikipedia, will they give the same answer?
Architecture is both a set of important decisions, and the structure of elements, and their interfaces, and a system of provisions, rules and agreements, and formed approaches that maintain order, and solutions to global problems, and decomposition with interaction, and a high-level description of the system, and its life cycle. parts, and patterns of structure and behavior, and what is difficult or expensive to change or remove, and fundamental decisions and trade-offs, and much more.
Architecture is also how an application fulfills its purpose.
API is an Architecture, a data schema is also an Architecture, and a structure of modules and services. There is an Application Scaling Architecture, an Application Reliability Architecture, a Component Interaction Architecture, and a Security Architecture. And of course, each feature has its own Architecture. And all this together is also Architecture.
Can all this be considered an answer to the question "What is Architecture?" - perhaps yes. Everything seems to be correct, everything seems to be clear. But is this answer practically useful, something that can be used in a real workflow? Not really.
How do you know if a decision is important or fundamental? What are global challenges? What does it mean expensive or cheap? How do you estimate complexity? Where is the border between high-level and low-level? What is order? What is the purpose of the system?
These are all questions that you can also try to answer. Formulate definitions, explain with examples, at worst, organize the decision-making process. But this will still remain something controversial, something intuitive, depending on personal perception and understood by everyone a little in their own way. This means that there will always be conflicts based on the fact that different people interpret the same situation in radically different ways. Developers will complain that architects are tampering with implementation details, architects that developers are breaking the Architecture. If there is no clear boundary, conflicts are inevitable.
Everyone probably has some general understanding of what Architecture is. But it is difficult to name a single, formally verified and mathematically precise definition.
On the other hand, if you ask "What is Realization?" - the answer is likely to be unambiguous: "Here is a link to the code repository!" And you can't argue with that. And there are no doubts or ambiguities in the interpretation of this answer. 1 + 1 = 2, Implementation is code.
And it's very easy to work with the code. If you want to understand the implementation, you can open the project in the development environment. If changes in the code need to be reviewed, then there are branches in the repository. If it's time to release a new version of the application, tests are launched.
Of course, the structure of the project can be confusing, the reviews are of poor quality, and the test coverage is incomplete. But, at least when working with code, it is always crystal clear what and when to do.
But what if the Architecture was expressed in code, just like the Implementation?
Many mainstream programming languages are used in modern development. Most of them appeared long ago or very long ago. And while they are still good, they are getting out of date a little. These are weakly typed languages: JavaScript, Python, PHP, Ruby, and strongly typed languages: Java, C / Objective-C / C ++, C #. They are being replaced by new languages that, within the framework of the same platform, try to solve long-standing problems, sprinkling them with syntactic sugar, but without introducing fundamentally new concepts if you look globally: Swift, Kotlin, TypeScript, Go. These are all languages built on the Object Oriented paradigm. Also, the Functional approach to programming has gained mature popularity, which is both implemented in the listed languages and presented in specialized languages, but less popular: Erlang, F #.Languages that implement substantially new concepts are also beginning to gain popularity: Rust, Haskell.
All this diversity has one thing in common: they are all Implementation languages. None of these programming languages are expressions of Architecture. But there is one exception - SQL.
SQL is the only mainstream programming language that is Declarative. It is built on a relational algebra model and has a narrow specialization - working with stored data. But still, why can SQL be considered the language of Architecture?
First of all, SQL gives access to meta information about the data schema. You can easily get a list of tables, columns, relationships, indexes, views, roles, and more from the database. Documenting the storage schema and being able to visualize it is a great help in understanding the structure of your application. A data schema description is a type of high-level architecture description.
Second, SQL provides a data query language that allows the client to query any combination of data described in the schema. If there was no query language, then for each data model required by the client, either a separate API method would have to be implemented, or the client would be forced to make several API calls to different parts of the data schema and combine the required model on its side. The presence of the query language makes it possible to exclude from the description of the Architecture implementation details associated with the enumeration of API methods for obtaining specific data models according to the general scheme. The query language excludes implementation details from the description of the Architecture.
SQL provides three key properties that are important for describing Architecture:
- Declarativeness - SQL allows you to separate the description of the Architecture from its Implementation.
- Analyzeability - SQL provides access to meta-information of a data schema, allowing you to visualize, document, and analyze its representations.
- Correctness - SQL allows you to define the constraints of the data model, which makes it possible to prevent a variety of data integrity violations by the Implementation.
Perhaps, SQL can be called a tool for describing Architecture in terms of data storage, but it is clearly not enough to describe the full Architecture of a product.
Other means of describing Architecture
The direct analogue of SQL is GraphQL, which also allows you to define the data schema and provides access to it through the query language. At the same time, GraphQL is built on graph theory, not on a relational algebra model. This makes it possible to work with hierarchical data models in a more natural way, in contrast to SQL, where working with hierarchies can be difficult.
GraphQL is often promoted as a client-server communication solution, but this is not the only area of its application. Similarly, through GraphQL, you can provide server-to-server communication, and even server-to-base communication. The fact that modern databases provide only a SQL interface, but do not provide a convenient API for working with hierarchical queries, is an offensive omission.
If SQL is a means of describing the Architecture of the data storage schema, then GraphQL allows you to describe the Architecture of the data schema already at the business level, in terms of a specific application area. In this sense, GraphQL allows you to define the Architecture at a higher level, closer to the real application area of the product.
Code modules and microservices, together with their APIs and dependencies, are also vehicles for expressing Architecture in terms of structure and relationships.
Architecture and Implementation
For a small project with a dozen developers working on, a separate description of the Architecture is often not required. If the project code is written neatly and well-structured, this may be enough to understand the overall picture and directions of development.
As the project grows, various documents appear, describing both its general Architecture and details of new parts and major improvements. This is quite a working approach when several dozen people are working on a project.
But for a fairly large project, when there are hundreds of developers, this approach starts to fail. There are too many documents, and their content begins to diverge more and more from the Implementation. It becomes more and more difficult to express thoughts in the same way for all. Rallies for dozens of people are organized to discuss many private topics, while really important changes can be implemented without any discussion at all. All this leads to the need to organize more and more processes, when some have to offer something, others follow, others review, and still others do. Bureaucracy. As a result, developers start to write more documents and less code, and this is not the most significant problem.
Of course, there is nothing wrong with the Bureaucracy as such. In any work, its organization is important. The problem is the amount of Bureaucracy per unit of useful work. If, in order to make One, you need to hold two more, Three, Four, Five rallies, this is no longer good anywhere.
Even a simple question, what change a developer can make himself, because it is an Implementation, and for what a request for a review, because it is an Architecture, starts to cause difficulties. You have to come up with a variety of Architectural triggers, that if you change the data schema or touch on cryptography and security issues, this needs to be reviewed Architecturally, and other topics seem not to be, but this is not certain. All this needs not only to be formulated and explained to hundreds of developers, but also to ensure that the described rules are followed. Such a process will constantly fail.
All these problems are related to the lack of a clear definition of the Architecture.
Just as changing a ".java" file in a repository unambiguously indicates a change in implementation, changing a ".architecture" file can indicate a change in architecture. Of course, ".architecture" files do not exist in nature. But for each project, you can define which aspects of it specifically relate to its architecture, and make it explicit.
If a change in the data schema is considered a change in the Architecture, then the files containing the SQL migrations must belong to the Architecture. If the API is also part of the Architecture, you can define the API in the form of the same swagger files, and rely on them. If the structure of the modules is Architecture, then the change in the description of the modules is an Architectural change. Etc.
Not all aspects of Architecture can be expressed by standard means. Each project has its own specifics. For example, if the product uses a specific model of rights, roles, types of subscriptions, add-ons, and so on, then you should think about defining the Architecture of this aspect of the product in some suitable Declarative form, thereby highlighting the essence of the Architecture from the implementation details. Of course, such changes may require significant code rework. But this is a substantially better investment in defining a Product Architecture than trying to describe its rights model in a document.
Formal representation of Architecture in code is possible, and only this gives a clear definition of what an Architecture is. This view makes it possible to analyze the Architecture and automatically generate documentation on it, which will always be up-to-date. All the rest: documents, diagrams and diagrams made by hand - Bureaucracy.
An Architecture can have a formal expression in code, and only this defines a clear boundary between Architecture and Implementation. And such a border is needed.
Engineering and Evolutionary Approaches in Architecture
Over the last half century, the development industry has come a long way from waterfall models to iterative development. And now it seems that the industry has gone too far in this matter. In some places, the work of an architect no longer resembles that of an engineer or even a gardener, and more and more begins to resemble the work of a seasonal worker hired for the sole purpose of weeding beds.
What distinguishes engineering from craft is the understanding of the laws of nature, regularities, on the basis of which one can make calculations and conclusions, design solutions, and not experiment. Any carpenter can make a perfectly acceptable stool, but engineering knowledge is needed to make the stool optimal: light, reliable, stable, simple and cheap to manufacture. An engineering approach allows you to get the optimal result quickly and with the least effort. Unfortunately, in software development, an engineering approach is not always possible.
Often, the result that one wants to achieve is very vaguely defined, many factors that significantly affect the architecture of the solution are unclear or not known at all, and the work on the implementation of the solution itself can be faster and cheaper than detailed elaboration of its Architecture. This is one of the reasons why the iterative approach to development has earned such popularity. Sometimes even beyond measure.
In the worst case, the development of the Architecture can boil down to the fact that each development team simply gives a review to the architect a description of what they are going to implement. In such a situation, the work of an architect is reduced to weeding out. And here you have not forgotten? Have you taken this into account? And here it is wrong. Do you want to add an index here? And so iteration after iteration.
It is necessary to weed the weeds. But what will a garden be like if you only do this? Flower beds will wither in the shade of fruit trees, strawberries and carrots will alternate, and all this will be traversed by countless paths, wasting the available space. It is difficult to talk about any Architecture here.
For Architecture to appear, an architect must do at least the work of a gardener. Berries and vegetables should be grown separately - the structure, and where people go most often, you need to lay paths - connections. Observing the particulars, the architect can act proactively, organizing and systematizing the particular features that arise in the process of Realization into a single structure. Like landscape design, an architect can shape the natural course of events, reinforcing positive trends and preventing negative ones.
And here we need a clear definition of the Architecture in the code. Data schemas, structures of modules and services, their API and interaction rules. Without defining these tools, the architect will either get bogged down in an implementation review or naively hope that the developers will really follow the rules described in some documents there, which, of course, will not happen.
If there is a formal presentation of the Architecture and the rules for its changes, this can already be called an Evolutionary approach to the development of Architecture. And that's not bad. But should we be limited to this?
Yes, initially the requirements may look vague, the factors may look vague, and the system itself is so complex that it is easier to do than to design. But over time, these things start to become clear. With practical experience in the development of Architecture, the details of the organization of its application domain appear more and more clear. It becomes clear that this general solution is implemented in five different ways, and that users suffer from this diversity. That some things are out of place and need to be moved, while others are already outdated and replaced by new solutions, so they need to be removed. And new developments stop looking so new - and we have already implemented this and that. Do you remember what all this led to? Let's do it right away so that we don't have to redo it later. After all, doing it right at once is usually faster than necessary.But this is only if you really know how to do it right. And with experience, such an understanding often comes.
Good Architecture is not something that has grown by itself and then has been a little organized, but something holistic, symmetrical and organic. An engineering approach to the development of Architecture is possible, just understanding the rules and patterns can take some time.
The formal definition of the Architecture of a particular project does not have to be static. Deprecated API methods can be declared as Deprecated, and missing ones can be declared as Not Implemented. The meaningful parts of the code can be marked for placement in separate modules. You can plan to split the table in two, or move them to another database. Etc. The architecture can be changed without changing the implementation. The implementation will catch up. And to catch up faster, the process can be slightly automated by making automatic reminders in team chats. In the same way, changes made by developers in Architectural files can automatically go to the architectural team chat.
An engineering approach to the development of Architecture is possible.
At the same time, one should not assume that the work of the architect is something fundamentally better or worse than the work of the developer. She's just different. The developer, first of all, works in the Imperative style, while the work of the architect is more Declarative. It happens that it is difficult to work out the Architecture, but the Implementation is a routine. It also happens vice versa.
Architecture and design
The terms Architecture and Design are often used interchangeably. You can say: System Architecture or System Design. By and large, it is clear what this is about. However, there are significant differences in these terms.
Design, first of all, means something visual, something that can be imagined or seen. Schemes, diagrams, models. Design allows you to visually show the relationship between entities, trace chains of interaction, see structural differences and analogies.
But Architecture is not only visual, it can include rules and constraints, inferences and mathematical calculations. Concepts that are poorly visualized.
Of course, both are important.
It is good if important aspects of the architecture are explicitly defined in the code. But this is not enough. It is important to make the Architecture visual, and the principles inherent in it automatic.
If the Architecture includes a data schema, there must be a visual representation of it. Modules have dependencies - they also need to be rendered. If there is an API, there should be documentation on it that is directly related to the structure of services. Etc. Design is a visual representation of Architecture.
If the Architecture has certain principles, for example, "client applications can only access certain types of microservices, but not all", then if there is a communication model, they can be checked automatically. Likewise, you can control the links between microservices and databases. If there are certain rules for the design of data models, the same rules for naming tables, then they can also be checked automatically. In development, automatic code checks are used, the same can be done for the Architecture. You can write autotests on Architecture.
Conclusion
The modern developer has a huge range of amazing tools available to make his job easier. Development environments, build systems, various libraries and frameworks, infrastructure services and containers. Developers don't write code in Notepad.
The Architecture is also code, and for working with the Architecture, the same powerful and rich toolkit is possible as for working with the Implementation. Google Docs isn't the only architectural tool you can use, and it's far from the best. Of course, the range of ready-made tools available to the architect right now is not nearly as wide as the range of developer tools. And not everything that an architect can potentially use works out of the box. Still, things are not so bad.
Many tools have been around for decades, such as SQL, you just need to learn how to use them correctly. New technologies such as GraphQL are starting to gain traction, and they offer hope that the task of describing a business domain will become as mundane over time as describing a storage domain. Tools for documentation and visualization of the application structure and API are available, including Swagger. You can use the same frameworks and integration tools to test the Architecture as you would to test the Implementation.
Architecture - Declarative. Probably, Bureaucracy of work with documents will never completely disappear from the work of an architect, but already now its volume can be significantly reduced. Architecture is also code, and over time, the available tools for working on the Architecture can become as advanced as the tools for working on the Implementation. We will see.
-
Dmitry Mamonov,
Wrike
PS I started the article with a list of philosophical questions to which, I hope, I was able to give quite specific answers. At least that's my point of view. Did you find something useful or new in the article? How do you yourself answer these questions? Write in the comments.