Let network engineers laugh at me in large integrators who have access to top-end equipment for a bunch of kilo-bucks. I want to talk about how you can get your home grid that is fast and not go broke.
tl; dr If anyone is interested, now the situation is such that a couple of desktops can be connected with a 56Gb grid for <$ 100 + shipping from the states. If you want more computers, then it's <$ 100 for a 12-port 40Gb switch and ~ $ 60 for a desktop, plus shipping. But this is all second-hand and zero guarantees.
I apologize in advance, but there will be many abbreviations. And this is not advertising - all this equipment has been around for years, it is no longer supported by the manufacturer and is easily searched for used on eBay for very inexpensive, compared to the prices for new from the manufacturer. But I must say that the new one (announced at the end of 2020) is almost an order of magnitude faster. So this is all about home experimentation and self-study.
Copper Age
, 2013, - , Ethernet 1Gb - , . , . 10GbE , Mellanox. Ethernet, InfiniBand ( ). Wiki, , - IB IP over IB, , . , / eBay - ($20-30). ( ) Mellanox- MHEH28-XTC, InfiniBand SDR Ethernet 10GbE, CX4. 3 ( 0.5, - ).
, 600-700MB/s . RAM , SSD 250-300MB/s .
, - , . , , . . , .
, , . MHGH28-XTC, InfiniBand DDR, 16Gbps. , , , . , 8 , , Cat 5/6. 1600MB/s, . , - .
, , MHJH29-XTC, QDR - 32Gbps. , Firmware - . , Win . , Linux . QDR , QDR - .
- , . . Enterprise , , . - . , - . .
, 2018. , - ( ) , eBay / . ConnectX-3, , . , .
SIDENOTE: , - 10/40GbE 25/50/100+GbE. , .
, . , Mellanox', HP/Dell/IBM, . Win, Linux, . QDR/FDR10/FDR14, 40/56Gbps. 4700+MB/sec . NVMe (PCI 3 -), .
- (<$100 8/12 32/40Gb). IS5022, - QDR, - , 8 , - , .
, Noctua, , , . 26074-84 8486-86.
WARNING: - - , . - , !
SX6005 - FDR10 12 , , . . DIY, -.
, - (Brocade ICX ).
- , - InfiniBand . Ethernet, IPoIB. - Win/Linux , . - - , IB level 3 (, , ). , (SR-IOV VF ), , .
Unfortunately, there are currently no cheap 40 / 56GbE Ethernet switches on eBay (you can look for the SX1012 if you're interested) with which to experiment. We'll have to wait a few more years. And there, you see, and up to 25 / 100GbE you can get in your home lab.
PS: With IB, there are still all sorts of nuances, such as the need for OpenSM somewhere, if switch is non-managed, but this is still not an article about setting up IB.