The Digi One SP is an ethernet-to-serial converter. As far as "mode", this would be more relevant to your serial application, and the way it needs to access the serial data.
Does the application talk to a Windows comm port? If so, you'd want to set the DOSP up for a Realport profile, then install the relevant driver to your Windows Server to create the comm port itself.
Does your app instead need to talk to a particular IP/TCP port instead? If so, you'd likely want to setup the DOSP for TCP Sockets, in a "Server" mode.
Further info can be found in the user manual and/or CLI reference:
http://ftp1.digi.com/support/documentation/90000583_L.pdf
http://ftp1.digi.com/support/documentation/92000304_N.pdf