What firmware were you running?
5236 had this in the release notes:
RS485 support has been updated to allow a 120 Ohm termination resistor to be enabled via configuration. To enable the termination resistor, set bit 3 of SREGs register S16 or via the Serial Interface configuration Web page. (WR21-63)
Perhaps that will help clean up the signal.