👨‍👨‍👧‍👧 👨🏽‍🌾 📡 Optimization of stored data by 93% (Redis) 🏷️ 👨🏽‍🎓 🍡

I would like to share my experience in optimizing data in order to reduce resource costs.

Sooner or later, the system raises the question of optimizing the stored data, especially if the data is stored in RAM. An example of such a database is Redis.

As a temporary solution, you can increase the RAM thereby gaining time.

Redis is a no-sql database, you can profile it using the built-in command redis-cli --bigkeys , which will show the number of keys and how much each key takes on average.

The volumetric data turned out to be historical data of the sorted sets type . They had a 10 day rotation from the app.

The project is in production, so the optimization shouldn't have affected the users in any way.

The data represented events of price / delivery date changes for the offer. There were a lot of offers - about 15,000 in each feed (price list).

Consider the following example of event data for an offer:

{"EventName":"DELIVERY_CHANGED","DateTime":"2021-02-22T00:04:00.112593982+03:00","OfferId":"109703","OfferFrom":{"Id":"109703","Name":" LG SN11R","Url":"https://www.example.ru/saundbar-lg-sn11r/?utm_source=yandex_market&utm_medium=cpc&utm_content=948&utm_campaign=3&utm_term=109703","Price":99990,"DeliveryAvailable":true,"DeliveryCost":0,"DeliveryDate":"2021-02-24T23:49:00+03:00"},"OfferTo":{"Id":"109703","Name":" LG SN11R","Url":"https://www.example.ru/saundbar-lg-sn11r/?utm_source=yandex_market&utm_medium=cpc&utm_content=948&utm_campaign=3&utm_term=109703","Price":99990,"DeliveryAvailable":true,"DeliveryCost":0,"DeliveryDate":"2021-02-23T00:04:00.112593982+03:00"}}

This event takes 706 bytes.

Optimization

To begin with, I reduced the rotation to 7 days, since it was the last week that was used. It is worth noting here that the step is very easy (in the source code I changed 10 to 7), it immediately reduces the RAM size by 30%.

, , , name, url, offerId 50%.

{"EventName":"DELIVERY_CHANGED","DateTime":"2021-02-22T00:04:00.112593982+03:00","OfferId":"109703","OfferFrom":{"Price":99990,"DeliveryAvailable":true,"DeliveryCost":0,"DeliveryDate":"2021-02-24T23:49:00+03:00"},"OfferTo":{"Price":99990,"DeliveryAvailable":true,"DeliveryCost":0,"DeliveryDate":"2021-02-23T00:04:00.112593982+03:00"}}

334 .

json protobuf.

, protobuf proto - :

syntax = "proto3";

import "google/protobuf/timestamp.proto";

message OfferEvent {
  enum EventType {
    PRICE_CHANGED = 0;
    DELIVERY_CHANGED = 1;
    DELIVERY_SWITCHED = 2;
    APPEARED = 3;
    DISAPPEARED = 4;
  }
  EventType event_name = 1;
  google.protobuf.Timestamp date_time = 2;

  string offer_id = 3;

  message Offer {
    int32 price = 1;
    bool delivery_available = 2;
    int32 delivery_cost = 3;
    google.protobuf.Timestamp  delivery_date = 4;
  }

  Offer offer_from = 4;
  Offer offer_to = 5;
}

protobuf

event_name: DELIVERY_CHANGED
date_time {
  seconds: 1613941440
}
offer_id: "109703"
offer_from {
  price: 99990
  delivery_available: true
  delivery_date {
    seconds: 1614199740
  }
}
offer_to {
  price: 99990
  delivery_available: true
  delivery_date {
    seconds: 1614027840
  }
}

protobuf

echo '
event_name: DELIVERY_CHANGED
date_time {
  seconds: 1613941440
}
offer_id: "109703"
offer_from {
  price: 99990
  delivery_available: true
  delivery_date {
    seconds: 1614199740
  }
}
offer_to {
  price: 99990
  delivery_available: true
  delivery_date {
    seconds: 1614027840
  }
}
' | protoc --encode=OfferEvent offerevent.proto | xxd -p | tr -d "\n"
0801120608c095cb81061a06313039373033220e08968d061001220608bcf7da81062a0e08968d061001220608c0b8d08106

50 . 85%.

proto- - https://protogen.marcgravell.com/

, 14 (50 706 ), 93%.

Optimization of stored data by 93% (Redis)

Optimization

More articles: