When selecting the option to "send messages to device connected to a serial port", you should see a drop down box for selecting the serial ports right below the Protocol selection.
Also, make sure your unit has the latest firmware applied, which is available from Digi's web site.