I'm giving the Marvell ConnectX cards a second go, as in, a second round of purchases.
-
I'm giving the Marvell ConnectX cards a second go, as in, a second round of purchases. I triple checked these are Ethernet capable. All I want for now is to feel the speed of 40gbps.
Maybe later I'll fool around with InfiniBand. I'm looking for a cheap managed switch < $100.
-
I'm giving the Marvell ConnectX cards a second go, as in, a second round of purchases. I triple checked these are Ethernet capable. All I want for now is to feel the speed of 40gbps.
Maybe later I'll fool around with InfiniBand. I'm looking for a cheap managed switch < $100.
@lymenzies hmmm, Mellanox ConnectX that is... Marvel also makes RDMA NICs but they are not InfiniBand. All Mellanox CX series (don't bother with CX-3, driver is ancient and power efficiency is bad, CX-4 and newer are good)... those are EN compat (either native or via slowness with encapsulated IPoIB).
I still have eight CX3 QDR-IB Nics — sitting new in a box, same with CX4 100G... they gotta go. I've moved my homelab mostly over to CX5, Qlogic with RDMA, and some Intel (meh) to keep up with specific features (really, CX-6/7 is ideal, but that's a tangent).
40G switch... various options under $100
- Dell PowerSwitch S6000 32-Port 40GbE QSFP+
- Cisco Nexus N9K-C9396PX 48x 1/10G SFP+ 6x40GB QSFP
- MELLANOX 851-0167-01 8-PORT 40GBPS -
@lymenzies hmmm, Mellanox ConnectX that is... Marvel also makes RDMA NICs but they are not InfiniBand. All Mellanox CX series (don't bother with CX-3, driver is ancient and power efficiency is bad, CX-4 and newer are good)... those are EN compat (either native or via slowness with encapsulated IPoIB).
I still have eight CX3 QDR-IB Nics — sitting new in a box, same with CX4 100G... they gotta go. I've moved my homelab mostly over to CX5, Qlogic with RDMA, and some Intel (meh) to keep up with specific features (really, CX-6/7 is ideal, but that's a tangent).
40G switch... various options under $100
- Dell PowerSwitch S6000 32-Port 40GbE QSFP+
- Cisco Nexus N9K-C9396PX 48x 1/10G SFP+ 6x40GB QSFP
- MELLANOX 851-0167-01 8-PORT 40GBPSI don't know why I wrote Marvell, must have had them on my mind.
The price point on the ConnectX-3 cards is too tempting, $25 for the pair. I'd like to see if it's worth the trouble, then upgrade from there.
While I probably do need to raise the minimum I'm willing to pay to get something useful, I have learned a lot from cast-off hardware over the years
I appreciate your recommendations. I'll look into the Mellanox since it's not large and not power hungry.
-
I don't know why I wrote Marvell, must have had them on my mind.
The price point on the ConnectX-3 cards is too tempting, $25 for the pair. I'd like to see if it's worth the trouble, then upgrade from there.
While I probably do need to raise the minimum I'm willing to pay to get something useful, I have learned a lot from cast-off hardware over the years
I appreciate your recommendations. I'll look into the Mellanox since it's not large and not power hungry.
@lymenzies ack, the cx3 prices are very good. however, if you want somewhere between 6-8 ConnectX-3 VPI (ie: they do 40G IB and 10G IPoIB) which are like-new in boxes with their cable sets, for free — just cover USPS priority shipping cost (~$15 approx, continental USA).
I learned a lot with used hardware, and really wouldn't have had the opportunity to get to where I am now without that approach. so to spread the sharing-is-caring ethical approach, you are welcome to own them! right now they're hanging out in my hardware stockpile inventory with no intended use. originally I had two of those Mellanox switches for a mini cluster, but I left them at my desk at Fastly just before covid — then they disappeared. cute little blue boxes, very loud fans.
in similar news, I am also giving away other hardware, so if you need other stuff just let me know. if you want low latency more so than RDMA protocols, I have a triple box of Solarflare 10G (see attached), which are formerly very popular with the HFT and distributed computing architectures where PTP time-code syncing is critical for transaction processing without collisions.