R-chain blockchain platform: general architecture and evolution

Hello to those who are following the development of the use of decentralized platforms in corporate and inter-corporate processes. Our publications on this topic were interrupted at the beginning of 2018. No, Raiffeisenbank has not stopped working in this direction: the time has come to move from methodological calculations and familiarization with the capabilities of individual technologies to the implementation of specific business cases in an industrial setting or as close to it as possible. This article is quite voluminous and at the same time informative. Therefore, we hope for your involvement and warn about the tutorial format.











In 2016-2017, we completed a number of research projects to build a decentralized trade finance platform. Used test Ethereum (Rinkeby) as the underlying distributed ledger platform and Ethereum Swarm as a decentralized file sharing tool. In addition to the general issues of building a decentralized platform, we tested the possibilities of smart contracts, the use of oracles and arbitration smart contracts. Some of these results are





Based on these research projects,





The result of this rather long-term work was, as the military say, “acceptance for supply” of IT of Raiffeisenbank of the regular decentralized platform R-Chain. It is now offered as an element of customer service for corporate groups of various sizes.



We share the main features of building this platform, and also talk about the evolution of the use of various "technical" components in it - blockchain platforms and distributed file systems.

Content

• Features of corporate and inter-corporate systems
• General platform architecture
  
  
  • Architecture of software components
  • One-stop business process
  • Benefits of using a one-stop business process
  • Disadvantages of using a generic business process
  • Host network monitoring
• Evolution of the use of various technical components
• Experience in research and operation of corporate platforms and their further development

Features of corporate and inter-corporate systems

In order for the implementation of the platform into industrial operation to proceed smoothly and without excesses unexpected for developers, it is necessary to understand that the platform proposed to the future participant must meet the following requirements:

• The platform must have an operator - a legal entity that is responsible to the participants for its functioning. The option - every man for himself, everything will be decided by the consensus of the participants, and so on, typical for public blockchain platforms, will not work here.
• , . , - - 15 — , .
• « » , . , , , « » , .
• , , . , — .
• «» . , - — , , . , , — , .

Many in this list will be surprised to find that the words "transactions per second" and other performance ratings are missing. Our experience shows that productivity is not the most important condition for a successful platform implementation. The performance just has to be "sufficient", and there are many data flow architectures that can dramatically increase the throughput of the underlying platform - from data registers to lateral turbo streams. The likelihood that your project will die due to inattention to the above points is hyperbolically higher.

General platform architecture

Architecture of software components

The applications that we used in research projects in 2016-2017 were written on the basis of direct integration of dialog interfaces with the "technical" components of the distributed platform - geth and swarm. But, having analyzed the possibilities and consequences of this approach, we came to the conclusion that it is necessary to introduce another layer between the business backend and the "technical" components - an adapter of unified business objects. This technique belongs to fairly standard patterns for building software architecture, which, nevertheless, does not reduce its effectiveness. As a result, the software architecture of the node of our platform began to look something like this:







In this architecture, the business applicationdoes not work directly with abstractions of DLT components, but with a certain conditional "universal business process" (hereinafter referred to as the process) - it creates a process, changes the state of the process. The reduction of the data structure of the universal process and operations on it to the data and operations used in the DLT client is carried out by a set of components designated as "Mirror of business processes". The "mirror" also performs the reverse transformation of the information received from the DLT client, and also filters the information by processes that are not related to the participant - the owner of the node. Thus, on each node, a database of business processes in which this node takes part is kept in a synchronous state, and in the format that is most convenient for a business application . Business Applicationinteracts with this database - a database of business processes, receiving information about the state of processes and transferring operations to change these states.

One-stop business process

Naturally, the question arose, what properties should be endowed with a universal business process so that it ensures, on the one hand, the maximum use of the advantages of blockchain platforms, and on the other hand, maximum flexibility and functionality for use in a Business Application. An additional condition was the ability to implement the selected properties on most of the common DLT platforms, the functionality of which is significantly different in some aspects (Ethereum / Quorum / Masterchain, Hyperledger Fabric, Corda, EOS, Waves). Based on the experience of our own and others' projects, we came to the following conclusions.



A universal business process should have the following attributes:

• Process parameters (process type, status, context attributes of the Process)
• Role list of process participants
• List of process-related electronic documents
• Process transition map

At the same time, the platform should, at the blockchain component level, provide the following conditions for the distribution of the process in the network:

• Completeness and integrity of information
• Confidentiality of electronic documents outside the circle of participants in the process
• Control of following the process transition map
• Storing the history of changes in process states

To implement these capabilities, “framework” smart contracts with the corresponding sets of properties and methods were developed.



Let's dwell on some of the listed attributes and conditions - what, how and why.



Process parameters- are conditionally open, since they are transmitted directly through a smart contract. For some blockchain platforms, they are fundamentally public (Ethereum / Masterchain), for others they can be closed by standard means of ensuring data privacy (Quorum - private smart contracts, Hyperledger Fabric - channels and private data). Probably the most important of the parameters of the “core” of the process in our implementation is the “type of process”, since it carries not only semantic, but also functional load - depending on the “type of process”, the DLT adapter chooses a smart contract template that this process will be introduce yourself. Why is this needed? Obviously, there are countless types of transactions and the same diverse business processes that ensure their implementation.In a fairly large number of cases, business processes essentially differ only in the transition map (from the platform point of view) and can be implemented with one single smart contract that supports an arbitrary transition map (more on that below). But quite unique moments can be associated with a specific business process:

• (, )
• (, )
• (, )

Trying to formalize, "format" and cram all this potential diversity into one single template of smart contracts is absolutely utopian. The development of a unique template for a specific type of business process is a much more efficient way, providing both high flexibility and the ability to "hard" flash the necessary process elements and critical checks directly into the "core" of the blockchain component. This will exclude any possibility of manipulation by individual participants. In addition, the platform as a whole will combine the unification of interfaces for the interaction of business applications with processes and a high modularity of the implementation of their specific functionality.



The “kernel attributes” of our process also include “status” and “note”: the first is a brief description of the process state (“New”, “Canceled”, “Closed”, “OnApproval”), the second is a “long” string with a more detailed description to the "status". We limit the length of a note to somewhere around 1000 characters (for example, "Insufficient funds in the account"), since electronic documents attached to the process are intended to transfer significant amounts of information (especially confidential).



Process transition map- describes whether a participant with a specific role can change the state of the process and to which one. The control over the admissibility of transitions is carried out by the "core" of the blockchain component and cannot be faked by a "powerless" participant. In addition, a navigation map can, for example, be used by a business application to determine the possible actions of the node owner in the current status of the process for the appropriate management of dialog components.



Transfer of confidential data.To transfer confidential information, electronic documents are used that are attached to the process. The business application uploads the necessary documents to the local file storage "Mirrors" and indicates them in the operation for changing the state of the process. Before transferring from local storage to DFS, the document is encrypted with the public keys of the recipient nodes - participants in the process. After the generated crypto container is transferred to DFS, the link to it and the hash of the original document are transferred to the process's smart contract. Upon receipt of information about a change in the state of the process, the file details are retrieved into the business process database, and then the crypto-container is extracted from the DFS using the link and decrypted with the key of the receiving node. A check is made to ensure that the hash sum of the document specified in the smart contract of the process matches,the document is placed in the local file storage and becomes available for the business application. Thus, the business application works only with the "open" version of the document - all the worries about its secure transfer are taken over by the "Mirror".



The history of the process state change is a sequence of "frames", each of which corresponds to one operation of the process state change. In our implementation, we store the following data for each "frame" of history:

• status
• note
• the identifier of the initiator of the state change transaction

The history of changes is recorded at the level of a smart contract and allows not only to track the sequence of transitions for audit purposes, but also enables the business application to correctly process the chain of events, even in the event of their simultaneous arrival (for example, freezing, interruption in functioning, and so on).



Ensuring legal significance- a very important issue, noted by us in the section "Features of corporate and inter-corporate systems". We initially proceeded from the concept that legal significance should not be provided by means of a blockchain platform, but through the use of an external PKI that has regulatory support or an appropriate level of trust among platform participants. Roughly speaking, an electronic document that provides a legal basis for your actions (payment document, contract, demand, etc.) and attached to the process must be signed on the basis of a "kosher" PKI (in Russia - GOST, somewhere abroad, for example, SSL or PGP / GPG). The line-of-business application will verify the "external" signature and take the appropriate action. Or not, depending on the result. Someone will saythat this is not "evangelical" and "we need to convince lawyers of the legal significance of blockchain transactions." We went through many steps of this journey and the result was always the same. However, in the case of RussiaMasterchain certification opens up certain opportunities in this regard - as the saying goes "Happy hunting!"

Benefits of using a one-stop business process

What did this approach give us in the end?

• Expanding the circle of potential developers of decentralized business applications. «» - - , - «» , . . -, Corda, «» Ethereum, Quorum. , - «» - .
• «» . , , , , , - . , «» «» ( ), «- » . , « , ...», . «» -, «» . , «» -, . , - , . , , , , , , «» insufficient funds for gas * price + value gas required exceeds allowance or always failing transaction.
• «» -. - - , -. -, , 75-95% . , , « , » . , :
  
  
  
  >>> - DFS, , , «» -. Ethereum, «» . - (TON!), . , . … , , .
  
  
  
  >>> - . , Ethereum — , , , , , — .
• . . . , , , , , , . , , , , — .
• . , , « , ». , . — -. , - ( ) — - .

-

Here, probably, as it is sung in a famous song - "Behind them you can hear a murmur ..." - for example, that the Hyperledger Fabric or Corda chaincode, unlike the slightly too elegant Solidity, allows you to implement almost infinitely complex logic of business processes, and this approach profane their functionality. Well, yes, they will be absolutely right. To the murmuring ones, I will remind you of several well-known messages from the field of software engineering:

• Where to put business logic - in the stored procedures of the database or in the code of business applications?
• Which is better - a mainframe or a special computer?
• Are you sure the baseline you choose will maintain compatibility over future life updates? And in general, will it survive the next 2 years?

The answer is simple enough if:

• You have a lot of money, time and free blockchain specialists
• You are sure that the blockchain basis you have chosen will not have to be changed from the word "never"
• You really need to "squeeze" the platform's capabilities by 101%

Well, then - a special calculator ... in the sense of Hyperledger Fabric or Corda with stitching on the cheyncode and other "chiseling in stone". If not, think for yourself ...

Host network monitoring

Perhaps for some it will be a revelation, but a well-organized monitoring is the basis for the successful operation of software systems in an enterprise. And this concept includes not only (and even not so much) standard infrastructure monitoring of servers, but functional monitoring of software components. Your distributed platform should not only work correctly, but also make mistakes correctly, providing the support service with a sufficient amount of sane information that will allow you to quickly identify, identify and fix the failure. It is even better if the monitoring system has proactive capabilities - it allows you to identify potential "bad things" and blocks their possible consequences before "happened".



If at every moment of time you do not understand what state the nodes of your network are in and what is happening on them, but are going to work "according to users' signals" - it is better to immediately free up space in the queue and do not take the time of your dear customers.



Based on the above, from the very beginning of the development of our platform, a proactive monitoring system was built into it. Let's describe the principle of its operation:

• In the blockchain basis of the platform, a special smart contract is established that is responsible for the collection and distribution of monitoring data (for brevity, we will refer to this smart contract as SCM)
• , (), « », , . , - .
• - « », — .
• DFS- « », .
• , DFS .
• , , :
  
  >
  
  > «» -
  
  > «» DFS
  
  > - ( Ethereum- )
  
  > «»
  
  > «» — , «»
  
  > «»
  
  > «»
  
  > audit trail from business applications

At a certain value or combination of values ​​for monitoring indicators, Mirror automatically pauses the processing of its queues of operations, blocks the appearance of potential undesirable consequences, for example:

• In case of a critical lag in the control label of the blockchain channel, which is interpreted as a node dropping out of the blockchain network or a complete disruption of its functioning
• In case of a critical lag in the control label of the DFS channel, which is interpreted as the loss of a node from the DFS network or a complete violation of its functioning
• In case of errors in the operation queue, all subsequent operations associated with this business object (universal business process) are blocked

Special attention was paid to the handling of database errors used by the "Mirror". This database is used not only as an interface with business applications, but also as a status stack for the state machine of the "Mirror" operation queues. Experience has shown that in the presence of specific errors when changing data in the database tables, looping operations can occur with massive re-sending of transactions and other pleasures. Once, due to a similar mistake, in two days we “made” a semi-annual volume of the chain on Quorum. Therefore, if an error of this kind is detected, "Mirror" is completely turned off and waits for a manual response from the support service.



The Central Monitoring Nodes retrieve information from all network nodes (including themselves, by the way) from the SCM and analyze it, allowing timely detection of such dangerous or potentially dangerous conditions as, for example:

• - DFS-
• - DFS-
• -
• «»
• «»

The picture below shows an example of a simple monitor for one of our test networks:







More "upper-level" proactive monitoring schemes were developed and implemented, including business applications, but here we come to the shaky border of intellectual property of specific customers.



Let's not hide the fact that in some of our blockchain networks, monitoring traffic makes up the overwhelming majority of total traffic. In this regard, there were even thoughts of using a floating schedule for the frequency of "probing parcels" - more often during working hours, less often during non-working hours. But it's worth it. Really.



In general, monitor whatever you can think of as long as your decentralized network bandwidth allows. You will praise the Gods of Programming more than once or twice, well, and the authors of this article, of course :)



For as one of the interpretations of Murphy's law says: "The error usually lies in a position that no one doubts."

Evolution of the use of various technical components

Having considered the general conditions for the deployment and operation of corporate decentralized networks, as well as the architectural principles that we used to build the R-Chain platform, we now turn to the story of how and why its individual technical components evolved in the process of implementing specific projects.



The first was the project of issuing an international guarantee , in which our partners were colleagues from Belarus - March-December 2018.



We started with the Ethereum - Ethereum Swarm - Crypto-Pro (DLT-DFS-cryptography) configuration, which has proven itself well in research projects. Instead of the Ethereum Rinkeby public test PoA network used, the Ethereum PoA private network and the Ethereum Swarm private network were raised. Initially, no technical problems emerged, but we ran into a "cryptographic" trouble - one of the Belarusian participants flatly refused to use the cryptography tools we offered, referring to the local law on electronic document management. At that time, it was not possible to find a high-quality solution “in the moment”, but there was a steady understanding of the difficult and important role of cryptography in the success of international projects.



Already in the process of running control transactions on the real network infrastructure (each participant deployed a node on their resources), failures in the work of Ethereum Swarm were identified - file losses were at the level of 20%. It has been suggested that the loss is due to problems in the Swarm client when sending multiple files in parallel. In general, this assumption was confirmed: experimentally, we managed to find a pause between sending individual files to Swarm in 5 seconds. During the transition to a completely combat network configuration, which, due to the peculiarities of the network segmentation used in the Raiffeisenbank infrastructure, required the creation of a transit Swarm node, a critical problem emerged - Ethereum Swarm allowed up to 30% of files to be lost when working through a transit node.The “layered” architecture and good monitoring system made it possible to successfully carry out the real issue of the guarantee in the mode of “manual pumping of gasoline”, but the fate of Ethereum Swarm was sealed. It must be said that the declared ability of Ethereum Swarm to work in topologies with no direct sender-receiver connection was one of the main reasons for choosing it as the technological basis for DFS, and its inability to work reliably in this mode created a lot of problems.and its inability to work reliably in this mode created a lot of problems.and its inability to work reliably in this mode created a lot of problems.



It should be noted that the private network based on Ethereum in this project pleased with its recoverability. The project schedule assumed that the closing of the issued guarantee would be made 3 months after its issue, and during this pause, some of the participants stopped their nodes. Nevertheless, upon restarting the network without any dances with tambourines in 1 day, it restored its integrity, and the operation of closing the guarantee went without any complaints.



The next project was the creation of an intra-group network for the Ascona Group of Companies - September 2018 - current time.



Based on the experience of the international guarantee project, IPFS (InterPlanetary File System) was chosen as the technological basis for DFS. It worked fine for sending files in parallel, and it did not require special mode adjustments. Perhaps the only weak point of IPFS is the impossibility (specified!) To work in "transit" topologies. When building networks with a large number of participants, the implementation by each of them of a "full star" of access from each to each is, to put it mildly, an organizational problem. On the other hand, all participants implement access between themselves and the operator's "support" nodes. Therefore, to organize the smooth distribution of files, the following mechanism was implemented:

• When a file is attached to a smart contract of a certain business process, the DeliveryFile event is generated containing a link to the file
• DeliveryFile IPFS. IPFS , , . .
• , , , «» ,

Thus, the Ascona project started in the Ethereum - IPFS - Crypto-Pro configuration.



The use of Crypto-Pro to encrypt files and "external" signatures of legally significant documents ensured the simplicity of the legal binding, as well as the absence of claims from the Information Security departments, which in turn had an extremely positive effect on the timing of obtaining the necessary approvals from both the bank and parties to the companies of the Ascona Group of Companies. In general, the project developed in a working order and, having passed the pilot stage, it was at the finish line to go to production and here ...



... and here in two projects at once - conditionally "alien", but in which we participated as experts, on similar configurations we caught mega-forks of thousands of blocks, with the loss of transactions of a part of the network. An analysis of the logs and interpretations of the blockchain community led to a disappointing conclusion - the use of Ethereum PoA (and in some cases even PoW) in compact networks with a small number of nodes (and corporate networks belong exactly here) has a high risk of such monsters. In addition, a mysterious bug began to appear periodically in our test network, when a node dropped out of the network and no longer wanted to synchronize with it. Even after reinstalling Ethereum and completely stripping. In short, it became clear that the prod network needed an alternative to the blockchain basis. And fast. Very fast.



The solution turned out to be Quorum - practically the brother of Ethereum. The number of improvements in the "Mirror" was minimal, the business application, of course, did not require any improvements at all.



At the moment, the transition to Quorum has brought only advantages:

• Raft consensus used eliminates forks
• The absence of empty blocks reduces the size of the chain

The absence of forks allows us to do without a pause while waiting for the conditional finalization of transactions, which was previously required in order not to handle a potential rollback of transactions, and we had up to 6 block generation cycles. This, firstly, naturally increases the platform's performance, and secondly, it removes very difficult problems that arise if the fork has exceeded the calculated pause and the state of the "mirrored" business objects that it touches ceases to be consistent with their blockchain state.



The only, perhaps, unpleasant feature of Quorum is the ability to generate a mega-block several megabytes in size upon restart after a long pause, which simply jams the DLT adapter when trying to unload its contents. But, strictly speaking, the service desk shouldn't sleep that long.



As a result of all this dramatic evolution, we came to the Quorum - IPFS - Crypto-PRO configuration , which we now use on the Russian domestic market.



Perhaps someone will ask a logical question: "Well, you haven't heard about Quorum before, or what?"... We've heard about Quorum, Hyperledger Fabric, and EOS. The author of this article even attended the first Corda workshop in Russian in the fall of 2017. Probably, specifically for an intelligent answer to such questions, Hegel invented his Dialectics. The small team that started the research in 2016 had good experience in developing interactive applications for Windows, and public Ethereum (test one is understandable) had the lowest entry threshold from blockchain platforms. And since we were interested in conducting research specifically on the blockchain topic, and not in tinkering with different dockers, without which it is simply unrealistic to launch "adult" Quorum or Hyperledger Fabric (and not on all virtual Windows platforms is possible), then the choice was obvious. As the research results began to attract the attention of the bank's business units and its partners,it became possible to expand the team, entrust boots to shoemakers, and pies to cakes, get hold of Linux servers, and so on. And, naturally, no one threw away the developed solutions as long as they retained their development potential. Dialectics and Evolution.

Experience in research and operation of corporate platforms and their further development

The author of this article took part in a fairly large number of blockchain projects carried out in Raiffeisenbank, in the FinTech Association and in some other places - both as a developer and as an expert on decentralized platforms. Some of them were purely research projects, some ended up as pilots, some grew to fairly large industrial networks of several dozen nodes.



What are the main conclusions that can be drawn from all this experience?



1. There is a certain variety of blockchain platforms, which are very different in their "consumer" qualities:

• "Threshold of entry" and ease of network deployment
• bandwidth
• functionality of smart contracts
• information closure options
• development time and cost

Therefore, I think it is impossible to say that any of the platforms will turn out to be absolutely dominant. Each has its own circle of potential users and tasks, where its use is most rational and cost-effective. This applies to Ethereum, Quorum, Hyperledger Fabric, and Corda. Here, as with programming languages ​​- only Vasya and Petya, who know one language each, will argue to the point of stupidity about which is better - "pluses" or "toad". And Semyon Petrovich and Albert Ivanovich, who know a dozen of them, will talk peacefully - when "pluses" are better, and when - "toad".



2. Despite the fact that some of the DLT platforms (for example, Hyperledger Fabric and Corda) provide the ability to transfer large data items - in fact, files, most likely, the blockchain basis with smart contract mechanisms and the file transfer functionality will remain separate. This is due to the following points:

• , DLT-. . Hyperledger Fabric Corda 2M « », , IPFS 100M. - - pdf (, ), 50M — , .
• - , ( + ), , .
• , , S3. , , « », , DFS. .
• , , -, «» .

3. Currently, there is a chronic shortage of specialists for technical support services of decentralized platforms. More precisely, they simply do not exist. Absolutely. In most of the projects I know of, the lion's share of the technical support work is done by the developers or research engineers who created these platforms, which, of course, is not good. I think this is due to the youth of the direction and gradually the development of technical instructions, response templates, monitoring schemes and other methodological materials necessary for organizing the efficient work of the support service is underway. One of the problems here is the lack of good overview courses in Russian on specific blockchain platforms. Everything has to be passed from hand to hand. But a support specialist in an enterprise is not a developer, and he is focused on other issues: monitoring,diagnostics of errors, ensuring the reliability and recovery of systems after accidents (do you think you will have no accidents? Yes, of course). And, frankly, the probability of a corporate project dying due to poor support is significantly higher than due to poor development quality. Therefore, attracting solid specialists with experience in supporting and operating enterprise systems is an important, if not the most important factor in the fact that the project will live and develop for a long time, and will not wither as soon as a couple of founding fathers leave it.Therefore, attracting good-quality specialists with experience in supporting and operating enterprise systems is an important, if not the most important factor in the fact that the project will live and develop for a long time, and will not wither as soon as a couple of founding fathers leave it.Therefore, attracting high-quality specialists with experience in supporting and operating enterprise systems is an important, if not the most important factor in the fact that the project will live and develop for a long time, and will not wither as soon as a couple of founding fathers leave it.



4. One of the most murky legal areas is the formalization of relations between the operator and network participants, which is aggravated by the fact that the operator, on the one hand, is not the owner of the network and its resources, and on the other hand, is obliged to ensure its functioning, even if the this measure is in conflict with the interests of individual participants. The balance between the rights and obligations of the operator, its "means of influence" on the network participants, the financial responsibility of the operator - all of this is now the subject of very difficult disputes. The simplest question:How to ensure synchronous replacement of critical software by all network participants, despite its seeming simplicity, results in very heated discussions? The emergence of examples of legal formalization of the position of the operator and network participants based on the experience of platforms that have already been released in prod will significantly accelerate the introduction of decentralized networks as a significant element of corporate and inter-corporate relations.



If you have reached the end, there is also a bonus: some of the questions about the current state and development paths of decentralized corporate platforms are reflected in the material prepared by the author of this article for one of the blockchain resources (the material is designed for a wide range of readers ).