A year later, while designing the new Windows 95 operating system, the guys took out an entire canteen. But after a couple of months of work, it became clear that there would not be enough space in it either. And if you allow applications to continue to communicate with the operating system and with each other in the old way, you get a real hell of endless interconnections and dependencies. A fundamentally different approach was needed ...
But what if we make a single mechanism through which everyone will communicate with each other? Eureka!
The result was an engineering masterpiece, the Win32 API. It is a set of functions for linking Windows components and applications to each other, one of the factors that has made it so popular for Windows 95 and later versions of the Windows operating system.
Integration
For project managers and those responsible for integration, the transfer of data between different systems evokes feelings, sometimes close to panic. From the point of view of process management, integration pleases with a huge number of little things and bottlenecks in which something can go wrong.
The situation is especially acute in large companies, where there may be 10-20 different information systems in the integration circuit, and it is necessary to ensure the continuity of business processes when transferring data in all directions.
Planning and managing the architecture of such a complex of applications is not a trivial task, and, unfortunately, it cannot be solved in the spirit of “let everyone use rebbit”, “install WSO2” or “let's replace everything with microservices”. No, examples have been and will be, but the case, at best, ends with two or more "cores" of integration, at worst - disruption of business continuity with unpleasant consequences for the participants.
In this article, I want to describe the main stages of the process, as it usually spontaneously develops in large companies. Introduce a scale of the company's integration development from 1 to 10. This scale, of course, will reflect the growth of the company itself, the growth of the level of development of administrative technologies in the company. And also I want to outline the main vectors of development: where to run, what to do and who is to blame for all this. Joke.
For liveliness, I took the example of an abstract manufacturing company. It will make it easier to show that increasing IT complexity is not an end in itself, but a response to all the new challenges that a growing company faces. If your company is not a manufacturing company, but, for example, a service sector, the difference is small. Some steps can be shifted one level or two up or down while maintaining the overall scenario.
Also, there are almost no names of specific products or technologies in the article. in the field of integration “there are no roads, there are directions”. There is no specific product that you just put in and use. It will always be about developing, at least, connectors. 1C, Windows, Office are mentioned as widespread software, but instead of them, without losing meaning, you can substitute SAP, Linux, etc.
Of course, the examples have been simplified for the sake of clarity. Several questions will remain open. And the approach itself, as is always the case in IT, is not the only correct one, so I invite everyone to the discussion.
This article will be of particular interest to IT executives, architects, integrators, as well as anyone who works in fairly large companies.
Level 0
Congratulations, you recently graduated from university and decided to go into business. You have just registered an individual entrepreneur.
In the near future, you will get acquainted with the amazing world of business, in which there will be both pleasant moments and not so much. You have already heard something about one of them from social advertising - taxes.
To pay taxes (if you have no imputed income), you need to somehow calculate how much money you earned in total for the period. It is unlikely that an individual entrepreneur will immediately start working by bank transfer, and when working with cash, a cash register is required in Russia recently.
Of course, you will want to compile a catalog of your goods and services so that you can immediately select it from the list - and punch a check. And at the end of the month, calculate and see that position A brings you x% more than position B.
So a simple 1C appears on your laptop - or any other similar product.
Level 1: denial
Business is going well, you have an assistant, then another one.
You have opened a second outlet in another area of the city, and there you need to put a second ticket office. It can no longer be connected to your laptop.
You read the instructions for the checkout, set up a connection via the mobile Internet. Or contact the company where you bought it, they will be happy to help you there. There is no talk of any kind of integration yet.
Level 2
Business is getting better, you have managed to enter the corporate sector. I may have embellished a little, but you were able to interest two local companies. Not very big, yet they are real legal entities with whom you entered into your first real contracts!
And one more contract, the third, with a supplier, but somehow he doesn’t have very good oral agreements on deliveries: he violates deadlines, there are questions about quality.
With the first client, you were able to agree on a prepayment, and the second convinced you to postpay, and now you need to keep records in the context of clients in order to understand exactly who was "weighed in grams". Fortunately, your 1C has a directory of contractors, and you solemnly entered the first two entries into it. Another supplier, just in case. And one more - "other".
You got a headache after yesterday's party. You went to the bank for an account statement - and "found" your first visit there! Of course, such an occasion could not be overlooked and noted. And then you thought, what if one day your business grows so much that payments will come every day?
Level 3: anger
Several times everything almost disappeared. Commitment has made you make a big difference in your business. Problems began to arise, in general, in the simplest things. Where you definitely did not expect them.
First, for the first time in several years of hard work, you decided to go on vacation. Recently, a good order was completed, some free money appeared, why not take a break?
We agreed on everything, appointed the smartest employee as his deputy, told him to call if something went wrong, and left to rest for two weeks.
There were no calls for the first week and it was perhaps the happiest and quietest week of your life. The last one is.
Then the bell rang. No not like this. CALL. Not only everything went wrong, you had to urgently return home - and restore everything that was almost broken without you: relations with clients, processes, accounting.
You had to part with the smartest, but you made two conclusions for yourself:
1. It's time to hire the first manager, a leader. And the point is not even that you do not have time, but that you simply cannot rest for a day. Since the IP work scheme does not imply a hierarchy, it means that you form a legal entity and become an LLC. With the renewal of all contracts.
2. You understand that you need a tool that will allow you to understand at any time how the business is going now. Regular reports. Well, okay, not at any time, but at least once a week? Not? Okay, once a month.
Secondly, in connection with the formation of a legal entity, you had to contact the 1C franchisee to buy a version of the program for legal entities.
The newly hired manager brought in a smart accountant - part-time, of course. She also explained that in order to work, you will need to order the development of a couple of reports from the 1C franchisee, without which there is absolutely nothing. She also took over the calculation of salaries and personnel records management, just started keeping them not in 1C, but in another program that they were taught in college.
Level 4
Several more years passed. There were ups, there were downs, there were all kinds of things. The crisis has died down in the country, but somehow it did not particularly affect you. In particular, due to the fact that you always clearly understood the state of your business, and could quickly "cut bones" where necessary.
You have developed a reputation as a reliable partner, the number of clients has exceeded 20, and the company already employs about 50 people. Several departments have appeared!
More and more reports are needed for work, so you hired a 1C-nickname programmer, who eventually mastered and began to refine the personnel system as well. And one more guy, an IT guy of all trades to support employees, set up computers, printers and other equipment.
The results of your work, goods, no longer fit in your bedroom, hallway, garage - and you decide to rent a warehouse.
We started thinking about a program for managing a warehouse and deliveries (there were a couple of cases - they took the goods to the wrong client), but the 1C nickname suggested that you can buy a warehouse module for 1C. Or not to buy anything at all, he will write better himself - and precisely taking into account your needs.
You also confused the discounts that you promised to different customers several times, and recently you even billed a few million to the wrong customer, which made him very stressed. All employees as one insist that it's time to implement CRM.
An IT guy recently graduated from university - and offered to write his own CRM, which would include your entire production cycle. 1C-nick promised to help him. And in just a couple of weeks, they sketched a working prototype, it was quite possible to use it!
Both spoke and behaved confidently enough, and at that moment you seriously thought: should you develop your own IT service, develop your own solutions - or turn to some integrator, use outsourcing services?
After thinking, listening and reading a lot of opinions, you decided not to choose outsourcing. Firstly, it is almost always more expensive than on-staff employees, because it feeds not only the performers, but also their leadership, and many taxes are also paid.
Secondly, the quality of such work is far from always guaranteed, even if the performers are the most titled (in an informal setting, you once asked them about the percentage of successfully completed projects on time, oh, in vain ...)
And, thirdly, and this the most critical: not a single outsourcer will do the job for you "as for himself." It goes against his business interests to tie you to him in every possible way.
As a result, you believed the convincing voices of your IT specialists - and decided that it would be better that way, to have your own developments! It is clear that later you will need to hire another developer - and more than one. It is clear that they will mess with the timing, that they will not do what they will be distracted. The main thing here is that the leader is sensible. And the nuances - so they all have nuances, they are in the accounting department, and at the security, and in the AXO.
From about this moment, too many things start happening in business, and I will not describe the development of business in the events, we will concentrate on increasing the complexity of the IT infrastructure, increasing the number of systems in the loop and developing integration.
It is also around this level that there is a danger of stepping on one of the biggest and most controversial rakes in the history of corporate integration: allowing systems to communicate directly with each other. See the example of the diagram below: CRM "goes" for data directly to the warehouse and personnel systems.
Level 5: bargaining
With the growth of business, an office was opened in the neighboring regional center, then in the northern capital. The next critical moment in business and company development has come - regional scaling.
A lot of
After several incidents of information leakage, you decide to hire an information security specialist. He, of course, turns almost the entire IT infrastructure upside down - and quickly wins the sincere "love" of all IT specialists. But only for a while, tk. knows how to clearly justify his requirements, and, in general, the IT specialist himself, of the same blood with them. His first requirement was to close all unnecessary ports - and send information over secure channels. He gave a lot of good advice on monitoring, backups, security, and other usefulness, but most importantly in the context of this article, he brought a couple more information systems to the company that needed to be integrated with the rest. In total, you have already got 7-8.
A corporate portal, an intranet, was recently launched, and it has become a good helper for employees. However, the biggest revelation for you was that the portal failed to display the structure of the company and the list of employees. No, the guys showed something, but what is this structure, where did they get it? It's just some kind of hybrid between a snake and a hedgehog! What about the list of employees? You personally fired several of them, why are they still showing up? Are they in the personnel system? And in 1C? Where does this data come from? ..
1C developer *) came out with a proposal to buy and implement a module for accounting master data - this is when all information on all general directories (for example, by structure, employees, contractors) is collected and controlled in one system **)... And also so that all exchanges of information go through this system, then this process will be visual and manageable. If there are any mistakes, they will be immediately noticed - and this is especially important. A couple of months ago there was already a precedent when salary information was not downloaded from the personnel system to 1C, and the salary was not transferred to employees on time.
What is needed for this? Choose such a system. Developers advise a solution on the 1C platform - this is a constructor that you need to refine for yourself. Also, the developers insist that all exchanges take place in real time, no night exchanges and scheduled exchanges. The future of this decision is not yet very clear, but the rationale for its necessity sounds clear and logical. Take it!
*) , 1 . MDM ( -) ESB ( ) . SAP, IBM, Oracle, — Fuse, Mule .
**) , ( — ) API. , 20-30, 40-50 , , . .
6
The territorial expansion of a company can be compared to an explosion! For a couple of years, several dozen offices have been opened, the staff of the company has increased several times. The first foreign offices were opened in several countries of the former USSR. The people there are, in general, ours, the processes are all clear, legislation and taxation practically do not differ from Russia.
The main question in planning the IT landscape and integration at this moment is whether it is possible for each information system to work for all countries at the same time? In practice, each system has its own nuances. Accounting and personnel systems - they must be kept strictly in different bases. MDM and ESB had to be modified in such a way that country binding appeared in most of the reference books. Moreover, some elements (for example, full name, positions, names of contractors) had to be stored in several languages at once.
We also managed to keep the corporate portal unified by making several sections in different languages. CRM, which includes, in addition to client functions, also the automation of your main business processes, has also successfully worked in all countries of presence.
In fact, you are happy with the situation in the IT and IT team. The foundations once laid made it possible to carry out expansion without any special difficulties and stop factors. It’s hard to imagine how much effort and resources it would take you now if the company did not have master data and a data bus. We are generally silent about CRM!
Level 7: depression
It would seem that everything is fine. Exchanges are running, data is running, business processes are running. But.
Recently, it is more and more often heard from different sides that the data is incorrect. All systems, all integration have been checked and tested many times. No, there are no technical errors or failures, everything works like clockwork.
Once the corporate portal congratulated you on your anniversary with the company. Everything would be fine, but the experience seemed several years less than you actually work - and the date is not the same. Are you thinking, remembering your graduation from the university, the first years of work ...
Something went wrong. The developers went to figure it out, brought up the story. It turned out that with some kind of renegotiation of the employment contract, the personnel officer sealed himself up, entered the wrong date. It is clear, the usual human factor, that's okay.
The next day, you signed an important contract. Prepayment? No, for several days the money will not be credited to the account. Have you listed it exactly? Listed exactly! And what is wrong? A call to the bank, they say: a mistake! It turned out that the data on the details in the MDM system is outdated: they have changed. And the employee who was supposed to fix them got sick.
And there is a lot of this, almost every day, with almost every reference book and every process. The number of dissatisfied with the situation in the company is growing, employees blame the master data system for everything, operators make excuses over and over again and talk about employee mistakes.
You even wondered whether to replace the products you are using. Here, at one presentation, they said that SAP products (Oracle, IBM) are not buggy! The developers object: no, it's not about the products, it's about the processes. An external audit was conducted, he confirmed that the systems are technically working correctly.
With large amounts of information, a problem was discovered that no one had paid attention to before due to its insignificance - the quality of the master data. Data becomes outdated, data loses its relevance, people who enter this data are constantly making mistakes. And this is no one's fault: errors will inevitably appear on volumes of hundreds of thousands of records.
An additional factor that increased the likelihood of errors was that at one time you chose a harmonized data entry scenario. This means that data can be entered into any system, and then collected in MDM. For example, a new counterparty can be added to CRM, through the corporate portal, in 1C - in any way that is convenient for a particular employee at a particular time.
Data quality problem? Okay, developers probably have a lot of data quality control tools to automate the process. How much? And what, just write checks with your hands?
Unfortunately, there was no easy solution to this problem. It could have been ignored for a while, but in the dynamics it became clear that after a couple of levels of company development, this problem would simply bury you. And no one concrete will be to blame for this. But how difficult it is to implement complex solutions!
Level 8
There is a solution to the data quality problem!
Yes, this required a qualitative change in the approach to working with data in all its diversity. The change project was almost jokingly called “quality for quality” by someone.
The concept of a person responsible for data has appeared (it is obvious that it cannot be an IT department), for each specific entry in each directory. All processes were analyzed - and a mechanism for classifying errors, a mechanism for creating data checks was developed. Indeed, as data accumulates, more and more types of errors appear, which means that the process of establishing rules and checks should be built, in fact, into the data entry process itself.
There was a long debate about whether the data quality control function should be centralized or decentralized. The Finance Department believed that the quality of contract and invoice data was their area of responsibility. The administrative and economic manager considered the data on the property as “his own”, and so on. It quickly became clear that in terms of different locations in the company, there are different criteria for data quality.
For one, one cut is needed, for the other, another - it got to the point that some of the requirements turned out to be even contradictory. Therefore, a register of such data requirements was compiled, which itself is master data, since all other systems now need this information. And it was decided to centralize the data quality management function itself, giving it serious powers up to changing any processes in the company.
Changing processes in the company revealed an interesting fact: people who enter data, as a rule, are not interested in what their mistakes will later affect others. And it is difficult to motivate such employees to monitor quality and correct mistakes “just because”. But it also turned out that if they describe the consequences (here is an irrelevant props, because of it employee X will not receive a salary tomorrow; there is a mistake due to which the money will go to the wrong place), this increases their interest in the process.
A data quality control system was developed - DQS, as an MDM module. She launched checks (on a schedule and when adding each entry) - and reported the error personally to the person who made the error. With a description of the possible consequences.
The rules in the system were first 100, then 500, after a year there were already about a thousand! The HR department helped a lot with trainings, suggested options for motivating employees, made infographics and a cartoon indicator of the number of errors, noting their decrease. A competition has even begun to find more bugs and add more rules! With a quarterly bonus, of course.
The next level of development of the DQS system is intelligent rules, fuzzy rules that aggregate rules (for example, the total value in two reports describing the same entity from different sides must coincide).
The highest level of DQS is quality control of master data and even their systemic content based on government open data. An example is services for counterparties on the tax website, they allow you to control the correctness of the details in your MDM.
Level 9: acceptance
For the quality of the data, for the correctness of the reports generated on their basis, you can be calm. Bug fixing is embedded in your company's genetic code - and happens by itself.
A clear understanding of the current state of processes, work with reports in dynamics, quality control of production, work with clients allowed you to come out on top in the market. Your company name has become virtually synonymous with the industry.
However, your business has slowed down. It looks like you are close to occupying the entire free market, and further growth is possible mainly through acquisitions and mergers. One of the main problems in acquisitions is the integration of the new team with all their established processes into your company.
Let's leave the administrative part and business processes outside the brackets. If we talk about integration, technically the task itself is not very difficult. Training of new employees is carried out, then a certain point in time is fixed, the “X” hour. After the "X" hour, all newly launched processes, new contractors, new contracts must be kept strictly in your main systems. These systems will stand up to 10-20%, even 50% growth without any problems. All modern systems of the class MDM, ESB, CRM, etc. allow you to scale yourself, simply by purchasing new server equipment.
In parallel with this, at about an hour "X" you need to freeze the information systems of the acquired company - and generate their dump, prepare information, bring it to your format. It may turn out that some of the data will be missing - it will be necessary to either replenish it, or temporarily create fictitious entities, from which their purpose will be clear. Gradually all of them must be replaced, enriched with real data. The timing and responsibility for this work should also be determined.
Level 10
Having unloaded the reporting from the BI system once again, you looked at the numbers - and thought. The number of employees in your company will soon be six figures, this is the population of an entire city. The annual turnover is close to 0.1% of Russia's GDP - and exceeded the GDP of the Altai and Tuva republics. On the territory of the former. The USSR market is all yours! And, to be honest, he is fully occupied. There is nowhere else to grow on it.
Recently, an event that you have dreamed of for many years has happened! You have taken over a small company operating in the EU market - and have put several new flags on the map.
If, in terms of business and administrative issues, this takeover does not differ much from companies within the former. USSR, then technically integrating them into your infrastructure is far more worrisome.
The main reason is the strict legislation of the European Union with respect to personal data (and, as experts explain, in the European Union almost all data can be attributed to personal data), the so-called GDPR is a provision on data protection and confidentiality in the European Union.
In addition to data quality, several more categories appear, primarily in relation to personal data. You must capture these categories through all processes: consent, contract, public assignment, vital interest, legitimate interest, or legitimate demand. You must be ready to delete any personal data upon the occurrence of certain conditions, at the request of the person himself (data subject). Also, you must not only ensure data protection (you have had this for a long time, thanks to the information security service), but also regularly conduct audits of this protection.
And most importantly: the data must be stored on servers in the European Union. In the future, these or similar requirements will appear in more and more countries, and for their violation, simply huge fines are provided. In the case of the European Union, this is up to 20 million euros, or up to 4% of the company's global turnover.
Almost always at this moment it turns out that the architecture of your information systems does not allow you to directly work with geographically distributed databases. However, if you have been able to successfully reach this level, your IT team will most likely come up with solutions. Since these changes will take several months, if not a year, it is better to prepare them in advance.
Local company, regional, federal, what's next? Global?
The whole world is waiting for you!
Outcomes
I want to share a few conclusions from the story above. Not all conclusions are unambiguous - and I assume that they may cause objections. Basic:
- A thoughtful approach to master data organization and integration is a matter of strategic importance to the company. Mistakes in them can stop the growth of the company, or even bury it altogether.
- It is not only the master data itself that matters, but also its quality. As the company grows, the issue of data quality becomes critical.
- At the same time, each next level of development of MDM, ESB, DQS is expensive and entails additional costs. There is a clear scale, based on which, you can form the current infrastructure. It is not necessary to create it with a margin of more than 1-2 levels.
- The responsibility for data quality should not be the responsibility of the IT department, but the entire business. IT's task is to provide convenient tools for implementing this responsibility.
- The data quality management function should be centralized. The quality control process is distributed among all participants in the work with data, starting from their initial input.
I invite you to the discussion!