I had a similar problem when I was first trying to get the Realport software running on Fedora Core 5.
Some of the things I learned was:
1. Check and make certain that the /dev/ttyXXYY devices are actually there. If they aren't then the driver isn't loaded or running properly. You won't get very far until this is resolved.
2. When you change the port setting on the PortServer to "realport", try rebooting the PortServer. This may not always be the case (and I do have an older version), but at one point I did this and things started to work.
My observations for Fedora are here:
How did you build from the src rpm? What command line did you use?
Also, are you sure the daemons are running?
Finally, check /var/log/messages, as there might be some useful info there.
Wish I had a solution for you, but hopefully this will give you a few pointers.
The RealPort software is great, and Digi has done a fine job IMHO. But getting every support working for every different distro out there strikes me as a pain. Which I suppose is part of what the general Linux community is for.