Reindexer's porting history - how to conquer Elbrus in 11 days

Hello everyone! This is Anton Bashirov, a developer from Rostelecom's IT cluster. Import substitution is gaining momentum, and Russian software is penetrating deeper and deeper into our everyday IT-shny essence of life. Elbrus and Baikal processors are becoming more in demand, the community is expanding, but thoughts about the need to port our entire beloved technological stack to the unexplored E2K architecture sound worse than stories about a production cluster burning in flames.

Working in the Elbrus implementation team, I had the opportunity to literally and figuratively touch our domestic processors, so I want to share my experience, talk about what pain threshold is needed to withstand porting the NoSql native database and not go crazy, and also to acquaint developers with the world of Elbrus and its inhabitants.

So, the guest in the studio is the Reindexer database, the development of our IT cluster.

It is worth saying why the choice fell on Reindexer, and not another database. Firstly, everyone's favorite and well-known Postgres is already included in the Elbrus OS packages. There is no need to transfer it. Secondly, our guest has already passed tests on ARM, therefore, it's time for him to conquer Elbrus.

It's worth mentioning that Reindexer was born as a  choice between Elastic and Tarantool . I definitely want to try to start it on a domestic processor.

Day 0. Meeting the guest

A few words about our guest:

: NoSQL in-memory database Reindexer

:  3.1.2

: 

:   @olegator99

: https://github.com/Restream/reindexer

: 

:  – C++, CMake

:- ;

- ++11 C++14;

- .

:- ;

- SQL 500K queries/sec PK ;

- full-text search ( Elastic, );

- server embedded;

- : Go, Java, Rust, .NET, Python, C/C++ ( ) PHP.

? . «»!

1. , ?

, . 8C 6.0.1, CMake.

, ! , – LCC.

, LCC GCC . LCC : gcc -> /opt/mcst/bin/lcc*



.

?

, , . , :

«» . .. « », , . : , , . .

– . . . , , , . LCC – , CPU.  .

, CMake.

, CMakeLists.txt , Reindexer. , , «Write once, run anywhere».

, , . .

– , :

№2:

? —  , CMake, .  make -j8



:

, «» , , /++ .

, Reindexer  __E2K__



  __LCC__



:

:

, messages CMake.

- , CMake , : jump_fcontext



  make_fcontext



. , :

, Reindexer .

, !

, :

# file reindexer_server
reindexer_server: ELF 64-bit LSB executable, MCST Elbrus, version 1 (GNU/Linux
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

# file reindexer_tool
reindexer_tool: ELF 64-bit LSB executable, MCST Elbrus, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux.so.2, for GNU/Linux 2.6.33, with debug_info, not stripped
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

. -. : reindexer_server reindexer_tool.

. Reindexer, :

• CMake-;

  
  
  
  
  

• ;

  
  
  
  
  

• ASM .

  
  
  
  
  

3. , ,

- , web-ui.

Reindexer.

- :

. i5 .

. (gdb E2K ) CLion, .

Reindexer :

. - free ( free jemalloc). , . :

1. work around - , QueryEntry ExpressionTree, , . , , MakeDeepCopy(),  mpark-variant.

   
   
   
   
   

   expression tree  .

   
   
   
   
   

   2 , variant , , , .

   
   
   
   
   

   

   - .

   
   
   
   
   

2. TODO: , . ( mpark-variant e2k):

   
   
   
   
   

inline constexpr DECLTYPE_AUTO visit(Visitor &&visitor, Vs &&... vs)

#ifdef E2K //Fix for Elbrus
    -> decltype(detail::visitation::variant::visit_value(lib::forward<Visitor>(visitor),
                                                 lib::forward<Vs>(vs)...))
    {
     return detail::visitation::variant::visit_value(lib::forward<Visitor>(visitor),
                                                 lib::forward<Vs>(vs)...);
    }
#else
    DECLTYPE_AUTO_RETURN(
        (detail::all(
             lib::array<bool, sizeof...(Vs)>{{!vs.valueless_by_exception()...}})
             ? (void)0
             : throw_bad_variant_access()),
        detail::visitation::variant::visit_value(lib::forward<Visitor>(visitor),
                                                 lib::forward<Vs>(vs)...))
#endif
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

5. ! …

, , , .

.

, ASM  make_fcontext jump_fcontext



 ?

, ASM Reindexer C++ , - boost/context.

?

– , . :

-

- (suspend)

- , (resume).

, :

• Libcoro ( C/++)

  
  
  
  
  

• Koishi ( , /++)

  
  
  
  
  

• Boost ( , /++, !)

  
  
  
  
  

• Node-fibers ( NodeJS libcoro)

  
  
  
  
  

• Tarantool (fibers libcoro)

  
  
  
  
  

• Kotlin ( , C++)

  
  
  
  
  

• C++20

  
  
  
  
  

• Goroutine

  
  
  
  
  

Koishi:

:

1. , , ;

   
   
   
   
   

2. - ;

   
   
   
   
   

3.  koishi_yield
   
   
   
    – test1  koishi_resume;
   
   
   
   

   
   
   
   
   

4.  koishi_resume
   
   
   
     – cofunc1  koishi_yield.
   
   
   
   

   
   
   
   
   

reindexer_tool, Reindexer . .

. , ASM ( ), .  , .

:

• 1:  ASM . , . , ASM - .

  
  
  
  
  

• 2: . , .

  
  
  
  
  

• 3: .

  
  
  
  
  

Koishi , E2K : makecontext_e2k()



  freecontext_e2k()



.

Koishi, Koishi, ?

– , API C/C++ :

• ucontext

  
  
  
  
  

• ucontext_e2k ( )

  
  
  
  
  

• fcontext

  
  
  
  
  

• win32fiber

  
  
  
  
  

• ucontext_sjlj

  
  
  
  
  

• emscripten

  
  
  
  
  

, , .

Koishi Reindexer:

:

, :

backend- Koishi fcontext ( boost , Reindexer). « – !» , E2K .  ucontext_e2k.c

, ( amd64, E2K):

11.

, . .

Reindexer 300 , .

, . Segmentation Fault.

:

struct ConnectOpts {
  /*   ,      */
  uint16_t options = 0;
  int expectedClusterID = -1;
};
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

« , ?» – . . , , .

.

ASM,

:

, . , – , Segmentation Fault.

,  addd



. segfault. ,  bool anyField = false



   bool anyField



, .

, – !

. Reindexer LCC v.1.25.16, 1.25.14, . , 15 . 16- , .

LCC v.1.25.16:

C++ , ASM , – ! ( asm- ):

1. gestp - %dr3

   
   
   
   
   

2. setwd -

   
   
   
   
   

3. setbn -

   
   
   
   
   

4. addd - "" ( 0x20) %dr2

   
   
   
   
   

5. addd -

   
   
   
   
   

6. ldb - false %r5

   
   
   
   
   

7. stb - false %r5 anyField1

   
   
   
   
   

8. addd -

   
   
   
   
   

9. addd - anyMethod ( anyMethod %dr0)

   
   
   
   
   

10. disp - anyMethod

    
    
    
    
    

11. call - anyMethod

    
    
    
    
    

– , , .

, Reindexer .

?

. :

• Reindexer Server

  
  
  
  
  

• Reindexer Server

  
  
  
  
  

• Reindexer Tool

  
  
  
  
  

• Reindexer Tool

  
  
  
  
  

• Reindexer Tool

  
  
  
  
  

• Reindexer

  
  
  
  
  

• 100% Reindexer

  
  
  
  
  

:

• C++

  
  
  
  
  

• C++

  
  
  
  
  

• ASM E2K

  
  
  
  
  

• 
  
  
  
  

• C++

  
  
  
  
  

:

• ASM E2K ( fcontext ASM i5 ucontext/ASM E2K)

  
  
  
  
  

• Reindexer

  
  
  
  
  

• 
  
  
  
  

• 
  
  
  
  

• E2K

  
  
  
  
  

?

– . 80% , . - , . , .

, , , !