Maximizing the number of incoming connections
"Administrator Linux. Professional" .
- « Ansible ». , ansible . , , , . — Linux. - , - . 2.2 , , , 5.5.
, , Linux Kernel Tuning for High Performance Networking: Configuring Kernel Settings.
netdev_max_backlog
. , TCP- , (dropped packet counter). net.core.netdev_max_backlog
.
net.core.netdev_max_backlog
— .
tcp_max_syn_backlog
SYN- (SYN Backlog Queue). "SYN_RECV" "SYN+ACK". accept()
(accept queue) , ACK. net.ipv4.tcp_max_syn_backlog
.
netstat
. 1. (SYN Backlog):
# netstat -an | grep SYN_RECV | wc -l
"SYN_RECV" , SYN- .
SYN Cookie
SYN cookie , SYN-. (net.ipv4.tcp_syncookies), SYN backlog, SYN+ACK , . SYN cookie , , . SYN cookie (Graeme Cole) SYN cookies ate my dog (SYN cookie ), , SYN cookie .
SYN+ACK
, SYN+ACK , ACK ? SYN+ACK. , . SYN SYN+ACK, ACK, - (Exponental Backoff) , , . SYN+ACK net.ipv4.tcp_synack_retries
( 5). : 1, 3, 7, 15, 31. 63 . SYN- 60 . SYN backlog , , , . SYN+ACK 0 1, .
SYN
SYN- SYN+ACK, , -. , nginx, - -, - , , (SYN backlog). , , . SYN net.ipv4.tcp_syn_retries
( 5 6 ). SYN 0 1, 63–130 .
- Linux Kernel Tuning for High Performance Networking: Ephemeral Ports.
(accept queue) somaxconn
, listen()
"backlog". 2.2 , , , , . , net.ipv4.tcp_max_syn_backlog
.
somaxconn backlog listen()
, , . : 1) backlog listen()
2) net.core.somaxconn
, .
net.core.somaxconn SOMAXCONN, Linux 5.3 128, 5.4 4096. , , 5.4 , 128, net.core.somaxconn.
SOMAXCONN, . . , nginx 511, 128 Linux 5.3.
, backlog
listen()
. listen()
backlog
, net.core.somaxconn, SOMAXCONN.
, , . , nginx 20480 HTTP-listener worker_connections
, .
. , , fs.file-max. :
# cat /proc/sys/fs/file-nr
1976 12 2048
, 1976 . , 12 ( 2.6+), — 2048. , , .
, . limits.conf (nofile) , systemd, unit- systemd (LimitNOFILE). :
$ ulimit -n
1024
systemd ( nginx):
$ systemctl show nginx | grep LimitNOFILE 4096
fs.max-file , ( ). :
fs.file-max = 3261780
, ( ). /etc/security/limits.conf, conf- /etc/security/limits.d/ unit- systemd. :
# cat /etc/security/limits.d/nginx.conf
nginx soft nofile 64000
nginx hard nofile 64000
# cat /lib/systemd/system/nginx.service
[Unit]
Description=OpenResty Nginx - high performance web server
Documentation=https://www.nginx.org/en/docs/
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target
[Service]
Type=forking
LimitNOFILE=64000
PIDFile=/var/run/nginx.pid
ExecStart=/usr/local/openresty/nginx/sbin/nginx -c /usr/local/openresty/nginx/conf/nginx.conf
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID
[Install]
WantedBy=multi-user.target
worker'
, worker' , , , .
. , , kernel.threads-max
. , , :
# cat /proc/sys/kernel/threads-max
257083
# ps -eo nlwp | tail -n +2 | \
awk '{ num_threads += $1 } END { print num_threads }'
576
. limits.conf (nproc) unit- systemd (LimitNPROC). , :
$ ulimit -u
4096
systemd ( nginx):
$ systemctl show nginx | grep LimitNPROC
4096
, . kernel.threads-max
. , , . :
kernel.threads-max = 3261780
nofile
, (nproc
) /etc/security/limits.conf, conf- /etc/security/limits.d/ unit- systemd. nproc
nofile
:
# cat /etc/security/limits.d/nginx.conf
nginx soft nofile 64000
nginx hard nofile 64000
nginx soft nproc 64000
nginx hard nproc 64000
# cat /lib/systemd/system/nginx.service
[Unit]
Description=OpenResty Nginx - high performance web server
Documentation=https://www.nginx.org/en/docs/
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target
[Service]
Type=forking
LimitNOFILE=64000
LimitNPROC=64000
PIDFile=/var/run/nginx.pid
ExecStart=/usr/local/openresty/nginx/sbin/nginx -c /usr/local/openresty/nginx/conf/nginx.conf
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID
[Install]
WantedBy=multi-user.target
TIME_WAIT
-, "TIME_WAIT", . , FIN- ( worker') . "TIME_WAIT" 2 x MSL (Maximum Segment Length — ), 2 x 60 . , 120 . "TIME_WAIT" , . FIN -.
- net.ipv4.tcp_fin_timeout
. 5 7 .
(receive queue) , , . , , . softnet_stat . — tcp_max_syn_backlog, SYN-, . , , , .
SYN (SYN backlog queue) . , , , , , , . SYN cookie, .
, (accept queue) , . — 20–25% .
:
# /etc/sysctl.d/00-network.conf
# Receive Queue Size per CPU Core, number of packets
# Example server: 8 cores
net.core.netdev_max_backlog = 4096
# SYN Backlog Queue, number of half-open connections
net.ipv4.tcp_max_syn_backlog = 32768
# Accept Queue Limit, maximum number of established
# connections waiting for accept() per listener.
net.core.somaxconn = 65535
# Maximum number of SYN and SYN+ACK retries before
# packet expires.
net.ipv4.tcp_syn_retries = 1
net.ipv4.tcp_synack_retries = 1
# Timeout in seconds to close client connections in
# TIME_WAIT after receiving FIN packet.
net.ipv4.tcp_fin_timeout = 5
# Disable SYN cookie flood protection
net.ipv4.tcp_syncookies = 0
# Maximum number of threads system can have, total.
# Commented, may not be needed. See user limits.
#kernel.threads-max = 3261780
# Maximum number of file descriptors system can have, total.
# Commented, may not be needed. See user limits.
#fs.file-max = 3261780
:
# /etc/security/limits.d/nginx.conf
nginx soft nofile 64000
nginx hard nofile 64000
nginx soft nproc 64000
nginx hard nproc 64000
- . , . , , .
"Administrator Linux. Professional".