Ok, to answer my own question, I actually extensively tested both now, a USB serial adapter and a PCIe serial card. The USB serial adapter which had a FTDI based chipset gave me driver problems on a few occasions, so it was not optimal.
The PCIe card worked well without any problems once the drivers were installed correctly, so I guess that makes the PCIe card the better choice.
Another option I also tested was using a serial Ethernet server
which actually worked pretty well too, however it was harder to get configured correctly.