By connecting to the router, you have the router in the way. If you really want to know, grab a cable and connect it to the server. Sometimes AC is right and it is the network.
Do you have a networking computer place who can come over with a very expensive fluke meter and check the speeds.
Check your network card and make sure it is autonegotiating with the switch.
If there is a problem with your network card, you can plug it into anything and it will be slow.
