Clever Geek Handbook

S3 architecture: 3 years of evolution for Mail.ru Cloud Storage



Storage Corridor by St-Pete



Hello! I'm Mons Anderson, the architect of the Mail.ru Cloud Solutions platform , and I'll tell you how we built our S3 storage, how it works, which solutions turned out to be successful, and which ones would be worth changing if we started the same project from scratch now.



@Databases Meetup by Mail.ru Cloud Solutions & Tarantool. :



  • Mail.ru, S3-;
  • , Mail.ru Cloud Storage;
  • ;
  • : ;
  • ;
  • SSL-.


, .



Mail.ru, S3-



S3 Mail.ru, , .



Mail.ru . storage- — 36 12–14 . , .



«» (pair). — unit . , , , , .



— , , . , , 3/2.





(pair) —



PairDB — Tarantool. , , — Tarantool, .



PairDB , , , , . , , , . PairDB — .





Pair DB:



, , , , — FileDB. , : - -, .



File DB: ,

— Nylon, . , PairDB, FileDB. stateless-, , , FileDB , , , .





Nylon:



- . — Streamer. HTTP-: PUT, , GET, . HTTP — .



Streamer'y, Nylon PairDB, , , WebDAV .



, storage — nginx , . Streamer , , . .





Streamer:



, S3-



, , S3-. PUT . . S3. S3, , :



  • — ;
  • HTTP;
  • — ;
  • HTTP-S3 Endpoint. S3 — , .


. .





, S3 API. S3 API Amazon, XML . - , .



Nginx. SSL, , Lua (, ).



S3 Tarantool. S3 , Streamer.



Nginx + S3 API + Metadata

Nginx + S3 API +





, S3. — . . . URL. — blob, , . : — URL, ACL ( ), — .



: , , . , : uploads chunks. .



Data schema



b2b- , .

Tarantool.



Billing



S3-:



, : , .



-. , - . - : S3- , , - . , -.



, - , , S3.



Tarantool. - — 21 , , . : -, . . , , . , Token Bucket. -, , , , . S3-, .



Rate Limit System



. S3 , . : Streamer, FileDB, PairDB, Storage. .



nginx, , SSD RAM-. Tarantool, , . , 32 , Tarantool .



Data cache



, , capacity (), , , , .



:



, - — .



S3- Tarantool. Tarantool, - . , API, — Tarantool, . , . .



Active failover



,



, — . , .



: , , . , , . , , , . , .



Brought objects that make no sense to shard



, — , . .



Growing objects carried away to a sharded cluster



, : , ACL. . , , , .



:



  • , , ;
  • , , ;
  • lifecycle — , , .


Final sharded scheme



, . , , , , .



Sharing proxy



, , .



, . 256 . . — , , :



f(bucket, shards) = subset



, , . map-reduce , , , . , . , , .



— , , . , :



f(object, subset) = shard



, , — .



Bucket sharding scheme



, , . . , — , , .



Final sharding scheme





— . , . , .



, . nginx, S3 API, , primary- ,



hotbox



, : « , , — , ». , , URL .



Hot and cold storage



Icebox , Hotbox, - . Hotbox Icebox , .



, , , , , . . .



primary-. Tarantool :



  • , Hotbox;
  • Tarantool ( — Icebox), ;
  • , , , .

    Synchronizing buckets



, , , , . , Hotbox, Icebox — , .



Migrating, , , , .



. API Icebox, Hotbox, , Nginx.



, , Nginx API Icebox .

Icebox nginx S3 API — :



Switching incoming requests



, — , Migrating, Local.



Data migration



, , .



Cleaning Legacy



:



  • Non-sharded. , .
  • Sharded.
  • , Migrating .


:



  • , .
  • .
  • : , , .


SSL-



Nginx. Nginx, OpenResty, Nginx LuaJIT.

— SSL-. S3- , CNAME. HTTPS : SSL-.



, SSL Nginx. Nginx, OpenResty, Nginx LuaJIT.



Nginx . ( ). ssl_certificate_by_lua, TLS-. Tarantool: .



, , Let's Encrypt.



SSL certificate storage



, -,





. . , , , , , , . .



Tarantool . . , .



. , , . S3 .



S3 «». , . .



. , , , , .





. , Amazon, , .



. , .



Tarantool. Tarantool , , , .



@Databases Meetup by Mail.ru Cloud Solutions&Tarantool. Telegram Kubernetes Mail.ru Group.

, S3 .





More articles:


All Articles