If you connect directly to the tty port using cu, minicom, etc...and issue those AT commands are you successful or do you see errors.
Based on the garbage characters seen, you might need to train the modem speed:
If the Modem has DIP switches, set them to the async default position.
The command sequence for the modem should be:
at&w1z -- reset modem to factory defaults
and read DIP switches if present
at$sbs0=1&w -- locks DTE speed at port speed
and stores to NVRAM
If you are using 8 wire cabling, make sure the altpin setting is applied to the port on the PortServer unit.