Building a Netflix Distributed Tracing Infrastructure

For future students on the "Highload Architect" course and everyone interested, we prepared a translation of an interesting article.



We also invite you to watch the open lesson
"Patterns of horizontal scaling of storages".










Our group -  Kevin LewNarayanan ArunachalamElizabeth CarrettoDustin Haffner , Andrey Ushakov,  Seth KatzGreg BurrellRam Whiteheilling (Ram Vaithilingam)Mike Smith  and  Maulik Pandey





«@Netflixhelps  " " ?» — Netflix Twitter





, , — , . . Edgar, . , , Edgar.





:

Edgar , Netflix,    , - . , () Netflix, (CDN) . , . , . .





, , , . , . Edgar — , .





Figure 1. Troubleshooting Edgar Session Crash
1. Edgar

Edgar, , . , , Java- . 2017 , Open-Tracing Open-Zipkin, , , Netflix.





Open-Zipkin, Java Spring Boot.  Mantis  , Cassandra. : , . , , . .





: ?

, . , , . . , , , .





. - .   HTTP B3  Open-Zipkin. , , , Java Node, ( paved road). legacy-, , Spring Boot. « », (Python, NodeJS, Ruby on Rails .), paved road.  ,   .





, , . Edgar . , Edgar . , , .





: ?

, , , , . , , , . , , , .





, , , . , .





( ).   , 100 % , . , . , , . - . .





 





Mantis — Netflix. Mantis , , backpressure. Mantis   Mantis Publish. , . , . Mantis . Mantis  Raven    Mantis (MQL). , .





Elasticsearch: , . , . - Elasticsearch, . , Elasticsearch . - , .  Cassandra  : . Cassandra , .





Cassandra. , . ,    . , :





  1.  Elastic Block Store (EBS) EC2 SSD.





  2. , .





  3. , .





Cassandra, EC2 SSD. EBS Elastic SSD , AWS EBS EC2. , Cassandra . 2019 Cloud Database Engineering (CDE) EBS EBS Elastic.





Time Window Compaction Strategy (TWCS, « ») Cassandra SSTable, - EBS. , , SSTable , . , Zstd Cassandra . Cassandra 71 % 35 , .





, Edgar 1 % . , , , , . Mantis . , Edgar. , , , , , . 20 % . , .





, . Edgar 10 . . , , . , Cassandra Cassandra S3.  Cassandra, , , S3.





Table 1. Timeline of storage optimization
1.

Edgar, .









,  Telltale  Netflix. Telltale -  Atlas. .









- , -    (FIT).









, ,     . , , AWS.





A-/B-





data science  A-/B-  , A/B.





?

Netflix . Edgar :





  • . , , .





  • , Netflix .





  • , , , .





, Edgar , « " " ?». , Netflix , « »!





, . ,   .






"Highload Architect".





" ".












All Articles