Hi, my name is Gennady, I work at Ozon, I develop backend services.
Redundancy of components, clustering or balancing is no longer a surprise to anyone these days. These are very important and necessary mechanisms. But are they really that good? How much do they protect us from possible rejections?
In Ozon, all of the above is used, but we are faced with problems that go beyond the capabilities of standard solutions and we need different approaches and tools. I am sure that you have clustering and it also does not help one hundred percent.
In this article, I want to touch on some of these issues and show how we improve the reliability of services using adaptive balancing.
Initially, the problem of client balancing of queries to PostgreSQL and Redis was solved, but the solution described below is not limited to this - it can be applied for other cases.
The algorithm is quite simple, not tied to any technology or programming language, and can be adapted for different platforms.
But first things first.
Let's start with the problematic.
Scaling bottlenecks
There is a certain service that should respond to requests. It looks like this:
, , PostgreSQL .
. ( , ..), . .
( ). , – . - Kubernetes ( – ).
:
.
. , , PostgreSQL Redis.
, — .
, .
. - , , — , . , .
, , , «» , - . eventual consistency (), , .
, . .
? ? , .
: , , !
, .
, : (), , , , , .
«» .
– - ( ).
:
1) , , ;
2) , , ;
3) , ( ) , .
, - . ? - . «» , .
.
. «», – ( , ). , .
. 1--1, 50% . , ?
( ) : , . , . , . ( - ), , . . , ?
( ) – . . , ; «» . , . , , .
: . , I/O-.
.NET
. .NET- (, , ).
. , PostgreSQL Redis. , . : 5%, 10%, 15% . . , : «FATAL: connection limit exceeded for non-superusers». , Redis !
. - , 80% – .
? (Exceptions).
.NET: , - . , «» , : , , , , stack trace. ? , ( CPU RAM).
, , «», . «», . stack trace.
( ) . , . CPU, throttling, «» . , , . .
, «» 200 Exceptions , .
, soft-exception – . .
, . ? . , , NpgSql, - ?
. :
;
();
( .NET).
.
- , , , , - «» . , .
.
response time – . .
? , . , «» .
– . , 100 . .
. – , , ( , ).
, response time 20ms .
– , «» , 2 ( ).
– .
, . :
F – ; , «»: 2/(n+1), n – ;
RT – ;
i – ;
EMA(i-1) – EMA .
, RT ( ) . , , , «» – .
. .
30, , , «»:
2ms, 10ms . , «» (SMA) . . EMA , , . EMA , «» .
: AMA, Double EMA, Triple EMA, Fractal AMA. , .
, .
(permanent exceptions). . – . , .
, response time? .
RT. , timeout – 100ms, – 200ms . . .
– , 1--1. RT - .
, response time? . Timeout , 1.5, – 2 . .
, «» . , !
, , – .
:
/// - ///
// , :
const EmaPeriod = 250
const EmaFactor = 2 / (EmaPeriod + 1)
// «» :
const BusyMultFactor = 1.5
const TimeoutMultFactor = 2
const ErrorMultFactor = 4
// , , N .
// , , N :
const InitialReponseTime = 2 //
// :
var firstNodeResponseTime = InitialResponseTimeMs
var secondNodeReponseTime = InitialResponseTimeMs
//
var firstNodeRatio = 1
var secondNodeRatio = 1
// , response time .
// ? RT , , .
// , , - , «» .
//MaxRatio – , . 1 200 0.5% .
// 0.5% , .
//, – . , , , .
// 0.5% .
const MaxRatio = 200
var RequestTotal = 0 //
// ,
const ThrottlingResponseRt = 1000;
const MaxResponseRt = 250 * MaxRatio
// , .
// – 0 1, :
func GetNext()
//
if firstNodeRatio >= secondNodeRatio then
fastestNode = 0
else
fastestNode = 1
fi
RequestTotal++; //
requestId = RequestTotal mod (firstNodeRatio + secondNodeRatio) // 1.
// MaxRatio ( 200).
// 201 , , – .
// , .
// highload-,
//
if (requestId = 0) then
return not fastestNode //
else
return fastestNode //
fi
end func
// . .
func updateStatistics(nodeId, responseTime, status)
if nodeId = 0 then
prevNodeResponseTime = firstNodeResponseTime // . , , .
fallbackNodeRatio = secondNodeRatio // , «.
else
prevNodeResponseTime = secondNodeResponseTime
fallbackNodeRatio = firstNodeRatio
fi
currentResponseTime = getResponseTime(prevNodeResponseTime, responseTime, status, fallbackNodeRatio)
newNodeResponseTime = calcEma(prevNodeResponseTime, currentResponseTime)
//
if nodeid = 0 then
firstNodeResponseTime = newNodeResposneTime
else
secondNodeResponseTime = newNodeResponseTIme
fi
updateRequestRatio();
end func
// – ,
func getResponseTime(prevNodeResponseTime, responseTime, status, fallbackNodeRatio)
if status != OK AND prevNodeResponseTime > ThrottlingResponseRt && fallbackNodeRatio == MaxRatio then
return prevNodeResponseTime;
// ThrottlingResponseRt «» ,
// .
// , responseTime ,
// 1 MaxRatio ( 1 200).
//TrottlingResponseRt – , , MaxRatio, .
// , , 2ms, , 1 200 400ms.
// .
// . , «»
// .
// .
end fi
//
If status = busy then
nodeResponseTime = BusyMultFactor * prevNodeResponseTime
else if status = … then // «»
….
else // –
return responseTime
fi
if nodeResponseTime > MaxResponseRt then
return MaxResponseRt // , .
// , , , , .
// , TrottlingResponseRt. , ?
// MaxResponseRt.
// , 250ms ( deadline_timeout).
// MaxResponseRt, deadline_timeout MaxRatio.
else
return nodeResponseTime
fi
end func
// :
func UpdateRequestRatio()
if firstNodeResponseTime <= secondNodeResponseTime then
fastestNode = 0
else
fastestNode = 1
fi
overallRt = firstNodeResponseTime + secondNodeResponseTime
if fastestNode = 0 then
fastestNodePercent = firstNodeResponseTime / overallRt
else
fastestNodePercent = secondNodeResponseTime / overallRt
fi
slowestNodePercent = 1 - fastestNodePercent
slowNodeRatio = slowestNodePercent / fastestNodePercent
if slowNodeRatio > MaxNodeRatio then
slowNodeRatio = MaxNodeRatio
fi
if fastestNode = 0 then
firstNodeRatio = slowNodeRatio
secondNodeRatio = 1
else
firstNodeRatio = 1
secondNodeRatio = slowNodeRatio
fi
end func
// EMA:
func calcEma(prevNodeResponseTime, currentResponseTime)
return currentResponseTime*EmaFactor + prevNodeResponseTime * (1-EmaFactor)
end func
.
:
nodeId = GetNext()
cluster = getClusterByNodeId(nodeId) // , .
responseInfo = getData(cluster, request) // ,
updateStatistics(nodeId, responseInfo.responseTimeMs, responseInfo.status) //
if responseInfo.status != OK then // ,
cluster = getClusterByNodeId(not nodeId)
responseInfo = getData(cluster, request)
updateStatistics(not nodeId, responseInfo.responseTimeMs, responseInfo.status) //
…
fi
, « », . , , .
? , . , .
(, ) , .
, ( GetNext, RequestTotal++). , # – Interlocked.
(firstNodeResponseTime secondNodeResponseTime)? – . .
, , . , .
«» ( «») , . .
(nodeRatio) . 1--X, . , . Interlocked.
? highload- . 50 .
. .
.
, PostgreSQL Redis. - .
PostgreSQL.
1 ( , ). 16:44 ( , , ). , , 0.5% fallback Redis. C 2 – .
, .
Error rate response time – - postgres, fallback Redis. PostgreSQL exception ( , ), response time . , , «» !
. Redis ( DEBUG SEGFAULT).
, . .
, error rate 2.51% ( – 0.009%). response time 90 99 . , .
-.
«»?
Redis ( ).
, PostgreSQL exception.
«» – PostgreSQL.
, 99 . 11:30 12:30.
, – .
,
, , . .
, , . - .
.
, , !