TCP to Zigbee port binding

Introduction

This page is to document a demonstration application that operates on a Digi Gateway product to tunnel data between a TCP network and a XBee network.

WARNING This script is deprecated! for an up to date script, use xbee_transport.py

Requirements

Overview

The application works to tunnel data from the TCP network and the XBee network by defining a TCP port XBee address pair. The TCP port is bound to the Gateway any any client connection will have its data forwarded to the XBee address specified. Correspondingly, the data received by the Gateway from that XBee address will be forwarded to the TCP client connection.

To help define the TCP port and XBee addresses is a helper script called 'table_generator.py'. After uploading the script to the Gateway, run it. This performs a discovery on the XBee network, identifies the unique addresses, and outputs a TCP port to XBee address mapping to a configuration file. By default the TCP port starts at port 4000, and increments by one per additional XBee address found. The output file is placed into the WEB/Python directory named as 'bind_table.py'. To edit it, download it from the Gateway and manually change the parts desired.

The main application 'tcp_zig_binding.py' takes one parameter optionally. The parameter represents the amount of time to wait before sending a message received from the XBee network to the corresponding TCP port. The usefulness of this is to potentially send fewer messages over the TCP network, or to group related messages into a single TCP packet. The parameter represents the amount of time in seconds, for example, .2 represents 200 ms delay. By default the application has a delay of .3 ms. Use a value of 0 to disable.

Code

This is the code from the main application 'tcp_zig_binding.py'.

"""
This script is intended to act as a data tunneling application between the TCP network and zigbee network.
It assigns multiple TCP sockets to listen on specific ports on the gateway device, this is defined in
bind_table.py, which can be hand made or generated. Each of the TCP sockets corresponds to a specific Zigbee
address on the zigbee network.  This allows an application to send data to port 4000 and always have it be in
turn sent to the same zigbee address.

There are several limitations and liabilities.  First the Series 2 radios have a maximum payload size of 72
bytes of data, with no internal ability to understand fragmented data.  The Series 1 radios have a maximum payload
of 100 bytes, again with no internal ability to understand fragmented data.  While this application can perform
some packet reassmebly on this end, the zigbee endpoint would not be able to do so, and whatever device or
application attached to said zigbee device will have to reassemble the data as needed.

"""

import socket
import select
import bind_table
import sys
import zigbee
import errno
import struct
import traceback
from time import clock

tcp_conn_dict = {}
tcp_port_dict = {}
tcp_data_dict = {}  ##Data associated with tcp port

zig_port_dict = bind_table.node_list  ##Where we obtain the zigbee to address lookup
zig_data_queue = []

listen_list = []
client_list = []

MAX_ZIGBEE_PACKET_SIZE = 100
MAX_TCP_PACKET_SIZE    = 8192
segment    = 0
end_point  = 0
profile_id = 0
cluster_id = 0

"""
This value should be changed to suite the data tunneling needs.  Factors such as the remote device's baud rate,
packetization time, and size of data all come into play, test and retest as needed.
"""

TCP_BUFFERING_TIME = .300 #300 milliseconds is out timeout period

###################################################################################################
# Parse Args
###################################################################################################

#We only have one argument, the TCP_BUFFERING_TIME
try:
  if len(sys.argv) > 1:
    TCP_BUFFERING_TIME = float(sys.argv[1])
except Exception, e:
  print get_exception_info()
  print "TCP_BUFFERING_TIME requires a floating number, it's default is .3 (300 ms)"
  sys.exit(0)

###################################################################################################
# Detect our radio type and assign address information accordingly
###################################################################################################

"""
Due to the differences between series 1 and series 2 radios, we much be able to determine which type of radio
is currently onboard our gateway, and set the end_point, profile_id, and cluster_id appropriately.  To do this,
we use the ddo_get_param(local_radio, 'HV'), which is the AT command to retrieve the hardware version.

Once we have the hardware version, we have to unpack the binary string into a tuple, and take the first
(and only) element from that tuple, and assign it to hw_version.  We test that number returned, if it's
over 6400, it must be a Series 2 radio, under 6400, series 1.
"""

try:
  hw_version = zigbee.ddo_get_param(None, "HV")
  hw_version = struct.unpack("=H", hw_version)[0]
except Exception, e:
  hw_version = None
  print get_exception_info()
  print "Failed to retrieve hardware version from local radio"
  print "Assuming it's a series 1 device"

if hw_version != None:
  if hw_version > 6400: #If the hardware version is greater then 6400, it must be a series 2 radio
    print "Detected Series 2 radio in gateway, configuring zigbee socket appropriately"
    end_point  = 0xE8 #232
    profile_id = 0xC105
    cluster_id = 0x11 #Out the UART
    MAX_ZIGBEE_PACKET_SIZE = 72   #Packet sizes are at maximum 72 bytes

  else:
    print "Detected Series 1 radio in gateway, configuring zigbee socket appropriately"

###################################################################################################
#  cleanUpConn()
###################################################################################################
def cleanUpConn(conn):
  try:
    client_list.remove(conn)   #We remove it from our listening list
    sock = tcp_conn_dict[conn] #We locate the associated listener
    tcp_conn_dict[sock] = None #We set the listener's lookup to None
    tcp_data_dict[sock] = []   #We reset the data queue
    del tcp_conn_dict[conn]    #We delete the key value pair for the reverse lookup
    conn.close()               #We close the instance
    conn = None                #We set the instance to None
  except Exception, e:
    print get_exception_info()

def _format_exception_info(max_tb_level=20):

    e_type, e_value, e_traceback = sys.exc_info()
    e_name = e_type.__name__
    e_args = []

    try:
        for item in e_value.args:
          e_args.append(str(item))
    except AttributeError:
        pass

    try:
        for item in e_value.message:
          e_args.append(str(item))
    except AttributeError:
        pass

    tb_string = traceback.format_tb(e_traceback, max_tb_level)
    return (str(e_name), e_args, str(tb_string))

def get_exception_info():
    """Formats the last raised exception, and returns a string that
       contains the name of the exception, any info associated with
       the exception, and a call trace for the exception."""

    e_name, e_args, e_tb = _format_exception_info()
    e_string = "".join(["Exception ", e_name, ":\n",
                        "".join(e_args), "\n",
                        "Traceback:\n", "".join(e_tb)])

    return e_string

###################################################################################################
# Init
###################################################################################################

"""
We loop through the current zigbee address to tcp port dictionary, create a socket per entry, bind
the socket to the specified tcp port, and map the various dictionaries to provide fast lookups
"""

##Note we are doing keys(), so we only generate the list once at the start, otherwise
##It will change due to our assignment of port -> address lookups

for addr, port in zig_port_dict.items():

    ##Create socket, bind to localhost:port, set it to listen and append it to the listening socket list
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.bind(("", port))
    sock.listen(1)
    listen_list.append(sock)

    tcp_conn_dict[sock] = None  ##Init the listener to connection dictionary

    tcp_port_dict[sock] = port  ##Assign listener to port lookup
    tcp_port_dict[port] = sock  ##Assign port to listener lookup

    tcp_data_dict[sock] = []    ##Assign listener to tcp_data_queue lookup

    zig_port_dict[port] = addr  ##Assign port to zigbee_address lookup
                                ##The lookup between address and port already exists by this point

#end for key in zig_port_dict

## Creating the zigbee socket, binding it to a local endpoint, set it to not block
zig_sock = socket.socket(socket.AF_ZIGBEE, socket.SOCK_DGRAM, socket.ZBS_PROT_TRANSPORT)
zig_sock.bind(('', end_point, profile_id, cluster_id))
zig_sock.setblocking(0)

###################################################################################################
# Mainloop
###################################################################################################
print "starting mainloop"
while True:

  """
  We listen for reads on the listening sockets for new connections the client sockets for new data
  from the tcp side and the zigbee socket for incoming zigbee data

  We listen for write ability on the client sockets so we can write the received and queued zigbee
  data on them, and we listen for write ability on the zigbee socket so we can write the received
  and queued tcp data on it.
  """

rl, wl, el = select.select(listen_list + client_list + [zig_sock],  #Reading sockets
                           client_list + [zig_sock],                #Writing sockets
                           [],
                           .5)                                      #Not used, (error lists, see man select)

###################################################################################################
# Handle zigbee reading
###################################################################################################

if zig_sock in rl:
  data, addr = zig_sock.recvfrom(MAX_ZIGBEE_PACKET_SIZE)
  print "Zigbee read: %d bytes from address " %(len(data)), addr

  if len(data) != 0:                 #if we've received data
    try:
      port = zig_port_dict[addr[0]]    #We find the matching tcp port
      sock = tcp_port_dict[port]       #we find the socket matching our port

      if len(tcp_data_dict[sock]) != 0:  #If the queue isn't empty
        (old_data, packet_time) = tcp_data_dict[sock][-1]  #Copy the last packet off

        if clock() - packet_time >= TCP_BUFFERING_TIME:  #Check the time of the last packet

          #If the difference in time is greater then our buffer time
          #We should just append the data to the queue
          tcp_data_dict[sock].append((data, clock()))
      else:
        #The difference in time ISN'T greater then our buffer time
        #We should append our new data to the old data

        lump_data = old_data + data
        tcp_data_dict[sock][-1] = (lump_data, packet_time)

        # We have to be careful of how we append the data, and to assign the older
        # time back into the tuple.  Otherwise we may continue to receive data
        # append it out of order, and never send it anyways because the buffer time
        # may keep reseting

      else:
        tcp_data_dict[sock].append((data, clock()))

    # We can have a valid key error here, in this case, if we receive a zigbee packet from an
    # Address that is not in the bind table, we will throw a key error.  An alternative is
    # Receiving a packet from the local zigbee radio, which will have the 0000 address, something
    # That is not normally in the bind table if automatically generated.

    except KeyError, e:
      pass

    except Exception, e:
      print "We received an unknown error, ", e
      print "Please report this!"
      sys.exit(0)

#end if zig_sock in rl

###################################################################################################
# Handle zigbee writing
###################################################################################################

if zig_sock in wl:
  if len(zig_data_queue) != 0:
    (addr, data) = zig_data_queue[0]

    if len(data) < MAX_ZIGBEE_PACKET_SIZE:   
      segment = len(data)
    else:
      segment = MAX_ZIGBEE_PACKET_SIZE
    try:
      sent_data = zig_sock.sendto(data[:segment], 0, (addr, end_point, profile_id, cluster_id))

      data = data[sent_data:]

      if len(data) == 0:  ##If empty, pop element, if not, return modified element to position in queue
        zig_data_queue.pop(0)
      else:
        zig_data_queue[0] = (addr, data)

    except socket.error, e:          #If its a socket error
      if e.args[0] == errno.EAGAIN:  #If its a blocking error
        pass                         #Do nothing, come again next select call
      else:
        print get_exception_info()                      #Not a blocking error, print and exit
        sys.exit(0)

    except Exception, e:             #Unknown error, print and exit
      print get_exception_info()
      sys.exit(0)

#end if zig_sock in wl

###################################################################################################
# Handle listener sockets
###################################################################################################

for sock in listen_list:
  if sock in rl:  #if it is receiving
    conn, addr = sock.accept()      # accept conn
    print "New connection from: ", addr

    conn.setblocking(0)             # No blocking on connection
    tcp_conn_dict[sock] = conn      # assign sock to conn lookup
    tcp_conn_dict[conn] = sock      # assign conn to sock lookup
    tcp_data_dict[sock] = []        # Clean any queued data for new connection
    client_list.append(conn)        # append conn to client_list for select statement

#endfor sock in listen_sock

###################################################################################################
# Handle read client sockets
###################################################################################################

for conn in client_list:
  if conn in rl:
    try:
      data = conn.recv(MAX_TCP_PACKET_SIZE)          #Receive our data
      print "Receiving %d bytes from tcp socket" %len(data)
    except socket.error, e:           #We have a socket exception
      if (e.args[0] == errno.EAGAIN): #If it's a blocking related exception
        continue                      #come back again next select call
      else:                           #If it's not a block related exception
        print get_exception_info()                       #Clean it up
        cleanUpConn(conn)
        continue

      except Exception, e:              #An unknown exception that's not socket related
        print get_exception_info()                         #clean it up
        cleanUpConn(conn)
        continue

      if len(data) == 0:                #Did we receive a clean break?
        cleanUpConn(conn)               #Clean it up
        continue
      else:
        sock = tcp_conn_dict[conn]   #Find the associated socket with connection
        port = tcp_port_dict[sock]   #Find the associated port with socket
        addr = zig_port_dict[port]   #Find the associated zigbee address with port
        zig_data_queue.append((addr, data))

#endfor conn in client_sock

###################################################################################################
# Handle write client sockets
###################################################################################################

for conn in client_list: 
  if conn in wl: ##We can write
    sock = tcp_conn_dict[conn] ##Find the associated socket with connection

    if len(tcp_data_dict[sock]) != 0:  ##Find the associated data list with socket, check if length of list is 0

      (data, packet_time) = tcp_data_dict[sock][0]
      if clock() - packet_time >= TCP_BUFFERING_TIME:
        #This packet has exceeded the time buffering, it is ready to send!
        try:
          if len(data) < MAX_TCP_PACKET_SIZE:
            segment = len(data)
          else:
            segment = MAX_TCP_PACKET_SIZE

          sent_data = conn.send(data[:segment])
          print "TCP: Sent %d bytes of %d total" %(sent_data, len(data))
          data = data[sent_data:]

          #If all data was sent, pop element out of queue, else restore modified element to queue
          if len(data) == 0:
            tcp_data_dict[sock].pop(0)
          else:
            tcp_data_dict[sock][0] = (data, packet_time)

          except socket.error, e:
            if (e.args[0] == errno.EAGAIN):
              continue
            else:
              print get_exception_info()
              cleanUpConn(conn)
              continue

            except Exception, e:
              print get_exception_info()
              cleanUpConn(conn)
              continue

  #end for conn in client_list

#end main loop

Downloading the source

Tcp_zig_binding.zip

 

© 2018 Digi International Inc. All rights reserved.
TCP to Zigbee port binding updated on 23 Aug 2018 02:02 PM