I'm giving the Marvell ConnectX cards a second go, as in, a second round of purchases. I triple checked these are Ethernet capable. All I want for now is to feel the speed of 40gbps.
Maybe later I'll fool around with InfiniBand. I'm looking for a cheap managed switch < $100.