Networking at bitly: Linux tc to keep costs low and for fun

, , , bitly — . , 150 , . , distcp hadoop — , . , , - API- — , , - . DevOps , , , bitly, .






bitly , , . , .





bitly Data Science (Ops/Infra) hadoop — , , . , .





, , , . , , :





  • bitly : hadoop 150 , , . ;





  • bitly .   (, ), . Ethernet- ( VLAN) — , ( ). - 10 Gb ;





  • bitly ( ), . , , . 





.





distcp, hadoop. , distcp mapreduce ( ) hdfs " ". distcp , .





bitly, Ops/Infra.





- API- . , , DNS- . , , , . , , , : , .





: , hadoop, — , .





mapreduce, distcp , - . , , .





, , , hadoop .





, , Data Science.





, , hadoop , mapreduce , :





bitly , . Ops/Infra .





hadoop.





Data Science.





hadoop — ( bitly), 1995 , 100 / ( 1 /) ethtool -s eth1 speed 100 duplex full autoneg on. hadoop, !





Data Science - .





, "", , ETL (, ) , , Ops/Infra.





, Ops/Infra!





, . - , hadoop , , bitly .





:





bitly?





  • roles.json : (app01, app02, userdb01, hadoop01 . .), (userdb, app, web, monitoring, hadoop_node ..), (app01,02 -> app, hadoop01,02 -> hadoop_node . .);





  • $datacenter/jsons/* : , json- , , — IP-, , , , .;





  • Linux : Linux.





, , Linux. , , , .





Ops/Infra .





(Traffic Control, tc) Linux, . ( ) , , tc. , , , , :





$ tc class show dev eth1
class htb 1:100 root prio 0 rate 204800Kbit ceil 204800Kbit burst 1561b
    cburst 1561b
class htb 1:10 root prio 0 rate 819200Kbit ceil 819200Kbit burst 1433b 
    cburst 1433b
class htb 1:20 root prio 0 rate 204800Kbit ceil 204800Kbit burst 1561b 
    cburst 1561b

$ tc filter show dev eth1
filter parent 1: protocol ip pref 49128 u32 
filter parent 1: protocol ip pref 49128 u32 fh 818: ht divisor 1 
filter parent 1: protocol ip pref 49128 u32 fh 818::800 order 2048 key 
    ht 818 bkt 0 flowid 1:20 
    match 7f000001/ffffffff at 16
filter parent 1: protocol ip pref 49129 u32 
filter parent 1: protocol ip pref 49129 u32 fh 817: ht divisor 1 
filter parent 1: protocol ip pref 49129 u32 fh 817::800 order 2048 key 
    ht 817 bkt 0 flowid 1:10 
    match 7f000002/ffffffff at 16
filter parent 1: protocol ip pref 49130 u32 
filter parent 1: protocol ip pref 49130 u32 fh 816: ht divisor 1 
filter parent 1: protocol ip pref 49130 u32 fh 816::800 order 2048 key 
    ht 816 bkt 0 flowid 1:20 
    match 7f000003/ffffffff at 16
<snipped>

$ tc qdisc show
qdisc mq 0: dev eth2 root 
qdisc mq 0: dev eth0 root 
qdisc htb 1: dev eth1 root refcnt 9 r2q 10 default 100 
    direct_packets_stat 24
      
      



, . — , , :





class htb 1:100 root prio 0 rate 204800Kbit ceil 204800Kbit burst 1561b cburst 1561b
      
      



— , .





— IP ( , IP ), :





filter parent 1: protocol ip pref 49128 u32 
filter parent 1: protocol ip pref 49128 u32 fh 818: ht divisor 1 
filter parent 1: protocol ip pref 49128 u32 fh 818::800 order 2048 key 
    ht 818 bkt 0 flowid 1:20 
    match 7f000001/ffffffff at 16
      
      



"subscribe hadoop14 to the class 1:20", "7f000001" IP hadoop14, "flowid 1:20" — . qdisc, eth1. , , 1:100:





qdisc htb 1: dev eth1 root refcnt 9 r2q 10 default 100 direct_packets_stat 24
      
      



(hadoop ), , , "1:10", ~800 / . , " ", "1:100". , hadoop namenode jobtracker, . hadoop, , "1:20", ~200 /.





, , , eth1 qdisc, "1:100". ? , "1:100":





[root@hadoop27 ~]# iperf -t 30 -c NONHADOOPHOST
------------------------------------------------------------
Client connecting to NONHADOOPHOST, TCP port 5001
TCP window size: 23.2 KByte (default)
------------------------------------------------------------
[  3] local hadoop27 port 35897 connected with NONHADOOPHOST port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-30.1 sec   735 MBytes   205 Mbits/sec
      
      



, "1:10":





[root@hadoop27 ~]# iperf -t 30 -c CABINETPEER
------------------------------------------------------------
Client connecting to CABINETPEER, TCP port 5001
TCP window size: 23.2 KByte (default)
------------------------------------------------------------
[  3] local hadoop27 port 39016 connected with CABINETPEER port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-30.0 sec  2.86 GBytes   820 Mbits/sec
      
      



, "1:10"?





[root@hadoop27 ~]# iperf -t 30 -c CABINETPEER1
------------------------------------------------------------
Client connecting to CABINETPEER1, TCP port 5001
TCP window size: 23.2 KByte (default)
------------------------------------------------------------
[  3] local hadoop27 port 39648 connected with CABINETPEER1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-30.0 sec  1.47 GBytes   421 Mbits/sec

[root@hadoop27 ~]# iperf -t 30 -c CABINETPEER2
------------------------------------------------------------
Client connecting to 10.241.28.160, TCP port 5001
TCP window size: 23.2 KByte (default)
------------------------------------------------------------
[  3] local hadoop27 port 38218 connected with CABINETPEER2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-30.0 sec  1.43 GBytes   408 Mbits/sec
      
      



? . — , , :





$ /sbin/tc -s class show dev eth1 classid 1:100
class htb 1:100 root prio 0 rate 204800Kbit ceil 204800Kbit 
    burst 1561b cburst 1561b 
Sent 5876292240 bytes 41184081 pkt (dropped 0, overlimits 0 requeues 0) 
rate 3456bit 2pps backlog 0b 0p requeues 0 
lended: 40130273 borrowed: 0 giants: 0
tokens: 906 ctokens: 906
      
      



hadoop, 1Gb traffic control. hadoop .





Data Science.





Ops/Infra , , , bitly .





Ops/Infra.





:

  • , : , . , , .





  • : , . hadoop .





  • Linux TC — , . , . — .





  • Linux: Linux





— " ":





: " - , , - , Nagios ".





, , , . hadoop, , , , . .





, , , . , , DevOps, , , . , DevOps, DevOps.





, :





  • Data Scientist





  • Data Analyst





  • Data Engineering









  • Fullstack- Python





  • Java-





  • QA- JAVA





  • Frontend-









  • C++





  • Unity





  • -





  • iOS-





  • Android-









  • Machine Learning





  • "Machine Learning Deep Learning"





  • " Data Science"





  • " Machine Learning Data Science"





  • "Python -"





  • " "









  • DevOps








All Articles