There’s a right mixed bag of stuff going on here… hard to pinpoint problems of this nature. A thought did occur though as I was reading… what are the network interface cards on client end, and server end for that matter… and do you use any form of traffic management.
As a test, take a client machine and disable the onboard nic if thats what your using, and place in an alternative nic. Try to find one thats not the same chipset, purely so we can eliminate the nic as a cause. If the card in the server is the same you’ll need to change it also to test, and you’ll have to mess about with ip addresses etc in linux… but anyhow, should at very least eliminate it from possible causes. I have my suspicions that its not actually a software issue at least. The NICs is where I’d go looking first though.