Digi Python Programmer's Guide

From Digi Developer

Jump to: navigation, search

Contents

Purpose of this Guide

This guide introduces the Python programming language by showing how to create and run a simple Python program. It describes how to load and run Python programs onto Digi devices, either through the command-line or Web user interfaces. It reviews Python modules, particularly those modules with Digi-specific behavior. Several sample Python programs are included on the Software and Documentation CD. This guide describes how to run the executable programs and describes program files.

What Is Python?

Python is a dynamic, object-oriented language that can be used for developing a wide range of software applications, from simple programs to more complex embedded applications. It includes extensive libraries and works well with other languages. A true open-source language, Python runs on a wide range of operating systems, such as Windows, Linux/Unix, Mac OS X, OS/2, Amiga, Palm Handhelds, and Nokia mobile phones. Python has also been ported to Java and .NET virtual machines.

Additional Python Documentation

For more information on the Python Programming Language, go to http://www.python.org/ and click the Documentation link.

Getting Started

This section shows how to create a very simple Python program, named hello.py, and the steps necessary to run that program on a Digi device.

First Program: “Hello, World!”

Step 1: Write the program.

In a text editor, create a file named hello.py, with the following contents:

# hello.py – Simple demonstration program
 
print “Hello Digi World!

Step 2: Test the program locally.

Digi has attempted to expose all the currently available Python functionality using subsets of standard Python APIs. This means you should be able to port programs between a Digi device and other Python running systems with a minimum of modifications. The tools of a real PC provide a friendlier environment in which to check for issues in the program and debug it. While the hello.py program is simple, it is a good practice to run programs locally before attempting to move them to the Digi device.

Step 3: Move the program onto the Digi device.

  1. In a web browser, access the web interface of the Digi device.
  2. Log in to the device.
  3. Using the menu, navigate to the Applications > Python page.
  4. In the Upload Files section of the Python page, type in the location or browse to select the hello.py file created earlier.
  5. Once selected, click the Upload button to place the file into the file system of the device.

Later, when creating more substantial programs, this same mechanism is used to load modules and ZIP files containing modules and packages on the Digi device’s file system.

Step 4: Run the program.

  1. Telnet or SSH to the Digi device and run this command:
    python hello.py 
  2. The program should output Hello Digi World and then exit.

Congratulations! You have just successfully run a Python program with the interpreter embedded on your Digi device.

Python Commands in the Digi Command-Line Interface

The Digi command-line interface has two commands for configuring and executing Python programs on Digi devices:

  • python: Manually executes a Python program.
  • set python: Configure Python programs to execute when a Digi device boots.

Detailed descriptions of the commands follow.

python

Purpose

Manually executes a Python program from a Digi device’s command line.

The python command is similar to a command executed on a PC. However, other than a program name and arguments for the program, the command takes no arguments itself.

Syntax
python [TFTP server ip:]filename [program args...] 

[TFTP server ip:]filename

The main file to be executed. This file can be either a file on the file system accessed through the Web UI, or a file accessible through a TFTP server on the network. This TFTP functionality reduces the number of times that you may need to place a program on the file system while developing and refining functionality. However, the TFTP behavior only works for the main program. Modules and packages must still be present on the file system to be used.

If no filename is given, an interactive shell is spawned. To exit the interactive shell, you must import sys and call sys.exit(), because exit() is not implemented in DigiPython.

program args

The arguments for the program.

set python

Purpose

Configures Python programs to execute on device boot.

Syntax
set python [range=1-4] [state={on|off}] [command=filename,args] 
Options

range=1-4

Range specifies the index or indices to view or modify with the command.

state={on|off}

When the state is set to on, the specified command is run when the device boots.

command=filename,args

The program filename to execute and any arguments to pass with the program, similar to the arguments for the python command. The command option allows for programs to be run from a TFTP server; however, this usage is not recommended. If there are spaces to provide arguments, make sure to wrap the entire command in quotation marks.

Loading Python Programs onto a Digi Device

As demonstrated in the hello.py example, Python programs are loaded onto the Digi device using the Digi web interface page Applications > Python. Program files and modules can be uploaded directly to the Digi device’s file system through this page. However, files can be uploaded one at a time only, and there is no support for directory manipulation. This makes adding entire libraries of modules difficult and adding packages impossible. To address this issue, Digi provides a module called zipimport, allowing collections of modules and packages to be included in a single upload.

To decide which modules work well on the device, or to share information on your experiences, please check out the Module Notes page.

To use zipimport, add all the files to be moved onto the Digi device to a file named python.zip. By default, Digi’s embedded copy of the Python interpreter checks for the presence of this file, and examines it when performing an import. Digi provides a python.zip file which includes several useful Python standard library modules that are known to work well on our device. This file comes pre-loaded on your Python enabled device, or the latest version will always be available at Digi's support website as part number 40002643.

Besides python.zip, the zipimport module can be used with additional .zip files, provided the Python program knows of their presence and modifies its environment accordingly. Internally, the files accessed through the Applications -> Python web page are stored in a directory called WEB/python/. To use additional .zip files, add them to the sys.path variable. This causes zipimport to search that file for .zip files as well. See the GPS sample application for an example.

To add mymodules.zip to the search path, you will need to following in your application.

import sys
 
sys.path.append("WEB/python/mymodules.zip")

This is perfectly adequate and recommended for a production environment. However, during development it is often desirable to be able to load a modified version of a zipfile onto the device without rebooting. In order to do this, we must remove our zipfile from zipimport's directory cache.

The following code should allow you to load and reload mymodules.zip without rebooting.

import sys, zipimport
 
zip = "WEB/python/mymodules.zip"
if zip in zipimport._zip_directory_cache:
    del zipimport._zip_directory_cache[zip]
 
sys.path.append(zip)

Using modulefinder.py to Determine Files to Load

For most programs, determining which files should be moved onto the Digi device should be fairly simple, most likely you will be writing most program modules and content yourself. However, when using third party modules or those provided by the standard distribution, a tree of dependencies may exist, making it difficult to determine which files must be placed on the device.

The standard Python distribution provides a tool called modulefinder.py that is useful in this scenario. This tool examines imports in Python programs to build a list of modules that may be used.

Using digi_build_zip.py to Automatically Build a zip File

digi_build_zip.py is an experimental utility to automatically build a zip file containing modules and packages required to run a Python application on the Digi platform. At its heart, digi_build_zip.py uses the modulefinder module to analyze a given script to build the zip file with some added intelligence. However, this analysis method is not perfect and errs on the side of inclusion. For example, many Python standard libraries perform dynamic feature support detection and import further packages if the OS feature is supported. modulefinder parses these import statements and includes packages and code that will never be used by the application script running on the Digi platform.

Using digi_build_zip.py is simple: execute the script with the name of the application script to be loaded on the Digi device as an argument. For example:

C:\My Project> python digi_build_zip.py my_project.py 

By default, digi_build_zip.py creates a zip file using the base name of the script. For example, my_project.py becomes my_project.zip. digi_build_zip.py will normally act silently until the zip file is written, unless any errors or warnings occur. Additional script options are available by specifying the --help option.

For example, here is output for the regular expression module:

Name File 
--------
m __main__ re.py
m _sre 
m array /usr/lib/python2.4/lib
dynload/array.so
m copy_reg copy_reg.py
m sre sre.py
m sre_compile sre_compile.py
m sre_constants sre_constants.py
m sre_parse sre_parse.py
m sys
m types types.py 

When using modulefinder.py, the list needs to be trimmed manually to minimize the size of the python.zip file and keep dependencies to a minimum. modulefinder.py lists files even if they exist in an execution path that the program will never use and would therefore not need to be present. It also lists modules that are built-in or may be dynamically loaded in your environment. Choosing which programs to load on the Digi device requires care, because some files may require functionality not present on the Digi device. Further, because Python is a dynamic run-time interpreted language, this missing functionality may not cause errors in program execution until the code is interpreted.

Recommended Distribution of Python Interpreter

The current version of the Python interpreter embedded in Digi devices is 2.4.3. Please use modules known to be compatible with this version of the Python language only.

To obtain python 2.4.3 for your Windows or Linux computer, it can be downloaded directly from this page: http://www.python.org/download/releases/2.4.3/

Python Module Reference

Fully Supported Python Built-In Modules

These modules are fully supported in the Digi implementation of Python:

  • array
  • binascii
  • cStringIO
  • cmath
  • collections
  • errno
  • itertools
  • math
  • operator
  • pyexpat
  • select
  • strop
  • struct
  • zipimport
  • zlib

Python Standard Modules with Digi-Specific Behavior

These Python modules have some Digi-specific behavior and limitations that are important to be aware of when designing an application:

os

Use of the os module in Digi devices is currently very limited. The primary purpose in exposing it is to allow access to the serial ports, which are presented as nodes in the file system. Serial ports are available as files with the path in the form /com/0 with the zero replaced by the zero-based index of the serial port to control.

Please see also Digi Connect Port Serial Port Access for a fuller discussion of using the os module to access serial ports.

The file system currently does not allow directory traversal or manipulation, and all file specification must be performed using absolute file names only. All files accessible through the Applications > Python web page are placed in the directory prefix WEB/python/.

When importing the os.py module, the module provides some functionality that will not work on Digi systems. These calls should work completely:

  • open
  • close
  • read
  • write
  • lseek
  • remove
  • unlink
  • isatty

However, not all of the above calls may be available on devices running SarOS. You can check if the above calls exist by connecting via SSH or telnet to a device and running import os followed by accessing the desired function as an attribute (such as running os.open)

These calls will not return complete information, and only the attributes st_size, st_blocks, and st_mode will contain correct information:

  • stat
  • fstat

Rather than deal explicitly with the limitations of the os modules for files, it is recommended to use Python’s built-in file objects, which have full functionality directly.

All files necessary to use the os module are included in the python.zip provided by Digi.

random

The random class or built-in calls to the module-level instance functions work using a time-based seed. However, the SystemRandom class cannot be used, because there is no OS support for a /dev/urandom device.

The random.py file necessary to use the built-in random module is included in the python.zip provided by Digi.

re

The re module works correctly. All files necessary to use regular expressions are included in the python.zip file provided by Digi.

socket

Most of the standard Python socket API is available for use on Digi devices. However, there are some exceptions and limitations in functionality.

Although gethostname() is supported, gethostname_ex() is not.

Normally, socket implementations make the assumption that one cannot bind two sockets to the same host and port. This is expected, and Linux- and NDS-based devices will raise an error message upon the attempt to bind the second socket to a host and port with a pre-existing socket. SarOS devices follow the non-standard behavior of silently letting the second socket replace the first socket. This makes socket-related logical errors much harder to debug on SarOS devices. In addition, SarOS devices have very few sockets available, while Linux and NDS have socket implementations that are largely constrained by the number of concurrent files can be opened.

Service and protocol lookup functions and the related name resolution functionality are not available. The Digi device does not currently maintain a database of symbolic service names. This means that the getservbyname(), getservbyport() and getprotobyname() functions are not available.

Digi has extended the standard socket API to provide access to ZigBee mesh networks. Please see Zigbee Extensions to the Python socket API for details

For examples of socket error handling, as well as TCP Keepalive usage see: Handling_Socket_Error_and_Keepalive

termios

A limited-functionality version of the termios library is present in the Digi device. It is not expected that the library will need to be used often for configuration, because the Digi device provides a mechanism for lasting configuration of the serial port using the user interfaces provided. Any configuration of the serial port using the termios module in Python will possibly interfere with other ways in which the serial port can be used in the system. If using the termios module, make sure your Python code is the only element in the system using the port.

Please see also Digi Connect Port Serial Port Access for simple Python examples of using the termios module to set baud rate and other common port settings.

Supported flags

Iflags
IXON Enable recognition of software flow control characters for output flow control.
IXOFF Enable generation of software flow control characters for input flow control.
IXANY Un-pause output flow control on reception of any character.
INPCK Check the parity bit on incoming data.
IGNBRK Ignore breaks in the data stream.
IGNPAR Ignore parity errors in the data stream.
PARMRK Encode errors in the data stream as 0xff 0x00 <ch>.
DOSMODE When used with PARMRK, encode the second byte of the error string as:

0x10 – A break was received. 0x08 – A framing error was received. 0x02 – An overrun occurred.

ISTRIP Strip incoming characters to 7-bits wide.
Oflags
ONLCR Insert a carriage return before every outgoing newline in the data stream.
OCRNL Insert a newline after every outgoing carriage return in the data stream.
ONOCR Do not transmit carriage return if terminal state is already at column 0.
ONLRET Newline characters perform carriage return on attached terminal.
TABDLY Expand tabs in outgoing data stream to the number of spaces that will

bring the terminal column to an 8 character tab stop.

Cflags
CSIZE Number of data bits CS5, CS6, CS7, CS8.
CSTOPB Number of stop bits: 2 when CSTOPB is set, 1.5 if CS5 is also set.
PARENB Enable parity generation.
PARODD Odd parity when PARENB is set; even if PARODD is clear.
CRTSCTS Hardware flow control.
C_cc
VSTART Software flow control start character.
VSTOP Software flow control stop character.
VLNEXT Next character is not interpreted with any special meaning.

Notably, no local modes are supported. Also, VMIN and VTIME are not supported; all os.reads() return immediately if any received data is available. However, there are oflags; the OPOST flag is not recognized. Setting an oflag is sufficient to turn on processing for that flag. If configured, the LNEXT character will be recognized despite the fact that canonical mode is not supported. The receiver is always enabled, so CREAD is not supported.

The ispeed and ospeed members of the attribute list passed to tcsetattr() must be identical. However, it is not required that they be specified using the existing symbolic defines. They may be specified numerically, and the tcsetattr call will succeed if the system can provide that baud rate within an error of 5%.

Non-standard routines

Two non-standard routines have also been added to the termios module:

  • tcgetstats()
  • tcsetsignals()
tcgetstats()
Purpose

Get modem signal and event status.

Syntax
tcgetstats(fd) -> [estat, mstat] 
Description

The estat variable is a bit-mask that reports status using the following constants:

EV_OPU Output paused unconditionally (using tcflow() )
EV_OPS Output paused by software flow control
EV_OPH Output paused by hardware flow control
EV_IPU Input paused unconditionally
EV_IPS Input paused by driver logic

The mstat variable is a bit-mask that reports signal status using the following constants:

MS_RTS, MS_CTS, MS_DTR, MS_DSR, MS_RI, MS_DCD

tcsetsignals()
Purpose

Set the output signals of the serial port.

Syntax
tcsetsignals(fd, sigmask) -> None 
Description

When the outgoing signals RTS and DTR are not being used for flow control, they can be controlled with this function. Sigmask is a bit-mask that should be set using the MS_RTS and MS_DTR constants.

thread and threading

The built-in thread and helper threading modules are available for use as needed in the system. They operate normally. However, because of the manner in which the Python interpreter is embedded in the system, error handling and thread exiting need to be done with care. Take all possible measures to ensure that threads started from a main thread of execution exit prior to the main thread exiting. Failure to do so will cause undefined behavior.

Consider whether threads are required in the application. If they are used, the main thread should include a top level try-except or try-finally construct that captures all exceptions and performs a loop that waits on a join() call or equivalent for all child threads. In addition, the main thread should be as simple as possible to minimize possible unforeseen termination.

time

The time module is believed to work well. As Digi does not yet have time-zone support, the tzset() function is not available. The clock() function has a resolution of milliseconds only, not microseconds, and tracks the uptime of the device. If the system you are using has a configured real-time clock, the time() routine will return the correct value; otherwise it too will return the system uptime, like the clock() function.

File System Access

Digi devices may support several varieties of file systems. Each distinct file system in the device is prefixed in the pathname with a volume specifier. All Python-enabled Digi devices provide the WEB volume, which contains the WEB/python directory from which programs and modules are managed. Some devices provide support for attached USB mass storage devices such as flash drives, and mini-hard drives. If USB mass storage devices are supported, when attached they will be added as volumes lettered A-Z based on their order of enumeration in the device. For more information see: How to use a USB Flash drive in Python.

File systems on Digi devices are subject to all of the limitations mentioned for the os module. Namely, directory operations are not supported and files must be specified with complete absolute path names including volume. For an example using an attached USB mass storage device, see the port sharing demo. All file system calls provided including write operations are fully blocking and will not complete until all data provided has been committed to the storage medium.

Sample Programs

Several sample Python programs are included on the Software and Documentation CD for your Digi device. This section describes the sample programs and files.

GPS Demo

The GPS demo in the Python/Samples/gps directory on the Software and Documentation CD (Media:Gps_demo.zip‎) communicates serially with a GPS receiver using the NMEA 0183 data format and presents the location information retrieved through that data stream as a web page redirection to Google Maps as well as through remote procedure calls using the XML-RPC protocol. It is not necessary to have a GPS receiver to run and view results from the application. In the absence of GPS data, it reports the location of the Greenwich Royal Observatory. New releases of Digi firmware provide support for GPS devices as part of the built-in functionality. If using a product with an embedded GPS, you should modify the script to open the /gps/0 file for the NMEA data stream.

Run the GPS Demo

  1. Load the demo files onto the device by selecting Applications > Python from the Digi device’s main menu, and using the file-loading function.
  2. (Optional) Attach a GPS device to the serial port and configure the port for 4800 8N1 and the GPS for NMEA.
  3. Run the gps_demo.py program.
  4. Point your web browser at port 8080 of the device running the demo. It should generate an HTTP redirect with a query to maps.google.com for its location.
  5. Run the following script on a PC to use the XML-RPC behavior:
# Get position information from gps_demo.py
import xmlrpclib 
 
s = xmlrpclib.Server(“http://<ip of device>:8080)
print s.position()

Files in GPS demo

The GPS demo consists of the following files:

gps_demo.py

This file includes the main application logic, which ties together the GPS communication library and the XML-RPC and HTTP modules from the standard library in a shared framework.

Notable aspects of the main application are:

  • The program modifies the sys.path program variable to add WEB/python/gps.zip to the module search path. This ensures that the additional libraries will be found in the import process for the SimpleXMLRPCServer which follows.
  • The program creates a built in way to cause it to exit using another socket. There is no current signal support provided by the Digi environment, thus no external default means of generating an interrupt in a program. For testing, it is helpful to be able to cause a program to exit. When finished, this logic could be removed or conditionally enabled based on arguments so that a port scan or other network activity can not inadvertently cause program termination.
  • Asynchronous processing of data using select. The built-in select module works for all socket and serial operations and ensures that the program runs only when it has work to perform.
  • sys.argv support. The program allows overriding the default run-time parameters by processing the argv array.
nmea.py

This file contains a library that performs the parsing and extraction of location information from an NMEA sentence stream on the serial port.

Notable aspects of this library:

  • The library uses the re module to perform protocol recognition. Each NMEA sentence is fairly simple consisting of a starting dollar sign, a two character device type identifier, a three character sentence identifier, a comma separated list of sentence specific data elements and an optional two digit checksum proceeded by an asterisk if present.
  • The library does not care what the source is for the provided data. Any endpoint providing NMEA data can be processed through this library through its provided feed function.
  • The core functionality is provided as a class object so that multiple instances of parsing can be generated and exist simultaneously in one program.

The library does a number of things that could be tuned for a GPS application and which require more complex manipulation and knowledge of the data stream. It does not verify that checksums are present on the sentences that are required to have them. The templated sentence lists used to extract data elements do not handle all sentence types; with more sentence types, the elements being added to the class dictionary would become cumbersome. Furthermore, there are possible performance considerations. In testing, with a single 4800 bps data stream, it was far from an issue. However, string manipulation of this form, which is heavy on slicing and splitting, requires several dynamic memory and copy operations, because strings are immutable and each operation creates and populates entirely new strings.

gps.zip

The XML-RPC and HTTP behavior is performed by taking advantage of additional files from the Python 2.4.3 standard library distribution. These files have ways of being used that can attempt to do things such as directory manipulation and hostname lookup. Because of the potential of using these files incorrectly, they have not been included in the base python.zip file. However, the problematic behaviors of the modules are present only when the modules are used in particular ways; if used properly you can still take advantage of most of their power.

This list of files to include in the gps.zip file was generated through manual inspection of the imports performed by the only module from the fine included on the top level; SimpleXMLRPCServer. This process could also be performed using the modulefinder.py module. However, manually scanning the modules imports allows for reviewing each module for possible problems and usages that could appear when running on the Digi device.

Port Sharing Demo

The port sharing demo program in Python/Samples/sharing (Media:sharing.zip) demonstrates an asynchronous socket server that allows multiple socket clients shared access to a single serial port. The demo program does this by using the select module and standard API calls. Much like the GPS demo, it listens on a socket to provide the main application behavior, and provides a socket to ask the application to exit as well for ease of debugging. However, if no sockets are attached, the demo application spools any data received to a file on an attached USB mass storage device.

Run the port sharing demo

  1. In the web interface of the Digi device, go to Applications -> Python and load the demo files onto the Digi device.
  2. Configure the serial port to match any attached serial device.
  3. In the command-line interface for the Digi device, execute the python command, specifying the file sharing.py.
  4. Connect to port 8001 with up to five TCP clients and read/write data to the sockets.
  5. Disconnect all TCP connections and generate data on the serial port. This data will be spooled in a file on any attached USB mass storage device.
Files in port sharing demo
sharing.py

The sharing.py file contains the master logic for the application. The majority of the program logic is in the process() function. The port-sharing logic runs in the main thread, while a child thread is created to spool any queued data to the file system. This is done to ensure that file system blocking will not interrupt processing of any additional data during the write.

Notable aspects:

  • Since most of the application logic is in process, it should be easy to extend this program to create a thread per serial port to run the process routine on multiple serial ports. Because of resource limitations on the Digi device, using this routine or select would be the preferred approach to extend the program to multiple ports, rather than running multiple copies of the program.
  • The program takes care to request termination and perform a join on child threads when exiting, to avoid the main thread exiting before all child threads have exited. This avoids causing undefined behavior and instability when the main thread exits.
  • The program imposes a maximum limit on client connections and I/O request sizes. Once again, this is to place a boundary on resource use rather than allowing a situation where usage could grow to be unbounded.
spooler.py

The Spooler class in the spooler.py file is an extremely simple sub-class of the thread object from threading.py. It retrieves objects from a queue object, and if the objects are strings, writes them to a file specified in the constructor. As mentioned for the sharing.py program, this program runs as a thread to allow file system blocking to not hold off the continued execution of program logic in the main thread. The thread object was designed to allow a mechanism for the main thread to ask it to terminate as well. This provides a way to cleanly exit the entire application, with child threads being terminated before the main thread.

Impact on payload size when using source routing

In large ZigBee networks, many-to-one and source routing can be used to reduce routing overhead and improve network performance. They are supported in XBee RF modules that run ZB and SE firmware. They are enabled when the AR (Aggregate Routing Notification) parameter of an XBee RF module, usually in the ConnectPort X gateway, is set to less than 0xFF. When source routing is enabled, the gateway will automatically capture and use the source route from each remote node. When data is received from a node, the route from the node is stored in the gateway. When transmitting data to the same node, the gateway will insert this route into the packet, causing it to be routed back to the node along the same path. An application does not need to change how it addresses transmissions to the node. When the source route is inserted into a packet, it becomes part of the packet’s data payload. This reduces the maximum size of the payload that is available to applications. The source route size is 2 bytes per hop plus 1 byte. The maximum size is 21 bytes. The NP (Maximum RF Payload Bytes) parameter of the XBee RF module in the ConnectPort X gateway indicates the maximum payload size available, including any source route. To allow space for a source route, the application is recommended to use a maximum data payload size of NP-21. If the application transmits a payload size that is too large, a TX status of 0x74 is returned. The application must be receiving TX status frames to receive this status. The data packet will be dropped.

For more information, see this Wiki article on large ZigBee networks and source routing: http://www.digi.com/wiki/developer/index.php/Large_ZigBee_Networks_and_Source_Routing

Personal tools
Wiki Editing