How to deal with over 2 billion records in a SQL database

Within the group enrollment on course " Highload Architect " to prepare a translation of an interesting article.

We also invite you to attend the webinar on "Choosing an Architectural Style" . In this open lesson, participants, together with an expert, will consider the differences between microservice and monolithic approaches, the advantages and disadvantages of the approaches, and discuss the principles of choosing an architectural style.

, , MySQL 2 . , . : , , , - , . .

, , : . , , .

- . Google Big Query. , Google, , (low latency). BigQuery, , (. ).

, (Andrzej Ludwikowski), , Big Query , . , , , , BigQuery , BigQuery - .

https://cloud.google.com/solutions/infrastructure-options-for-data-pipelines-in-advertising#storing_data

, . Apache Kafka , . Kafka — Kafka , , MySQL. BigQuery, , . , , - .

MySQL

, MySQL Kafka, , , Debezium Kafka Connect. — , . MySQL , Debezium , MySQL . Kafka Connect - , - . , timestamp-, - , timestamp, .

, , , Kafka. Kafka Producer, - Kafka. Kafka Consumer, BigQuery, .

Sending data to BigQuery — BigQuery

, Kafka ( ), — BigQuery. . . , . . serial id . (truncate) / (drop) , , . , Kafka . , INSERT , . , , , , .

Passing Data to a Partitioned Table

, BigQuery , , , .

, . , 90% . , Kafka Consumer, . «» (compacted table), .

( "A" "B" type ).

Passing data to a compacted table — compacted-

(compacted table), (partitioned table), Kafka (compacted table).

, , , . . , , , .

BigQuery , , , - MySQL, , . — , .

, . Kafka BigQuery. Kafka, , .

"Highload Architect".

« ».

All Articles