@jwz When I last experienced something similar, it turned out to be a bad router somewhere that broke path MTU discovery. Symptoms were SSH connections dying after some time, but immediately after running a TUI app like top or vim. Lowering the MTU on my interface from 1500 to around 1200 "fixed" that for me.
You could try to ping your Mac with successively larger packets (ping -l), see if that's your issue.