Archiving Data Files from Device Cloud

From Digi Developer

Jump to: navigation, search


Archiving Data Files from Device Cloud

When experimenting with XBee sensors, solar power and other fun technologies, it may be useful to archive your old data files on your own server in raw XML form. For example, someone testing use of solar power for a remote sensor could archive data for a full year, then run a report comparing the solar panel voltage verse battery voltage verse temperature verse month of the year.

Using Device Cloud presentations such as idigi_db (included in DIA) or idigi_upload (on this page Dia_Event_Uploader), you'll have a circular collection of N files which eventually overwrite themselves. For example, if you upload every hour and have at most 25 files, then your Device Cloud account will hold only 1 days worth of data.

Running by CRON

The Python script below can be run by a task scheduler every few hours to copy the data into a local archive. For example CRON can be used on a Unix/Linux server.

Assuming the user which runs this job runs at low permission level, you should create a shell script which runs this python script, saving the output into a log location. User ownership and permissions can be tricky, but with care it will work. For example, you can archive the files to a SAMBA share drive under a special SAMBA user.

Python Code

This Python application runs under Windows or Linux - it is not designed to run on a Digi gateway.

ZIP Archive

Here is the Python code as a ZIP file:

Full Code

If you wish to browse the file for ideas, it is included inline below

# - download the XML files from the Device Cloud server
import os
import sys
import time
import urllib, urllib2
from urllib2 import URLError
# put your idigi account info here
USERNAME = "my_nama"
PASSWORD = "my_passa"
# your archive builds below this - if you use CRON to run this script,
# then you'll want the full path name
ARCHIVEBASE = '/home/idigi/archive'
# archive files are named with the modified-date from the Device Cloud storage
# in a form 'YYYYMMDD_HHMM' such as '20100812_1045'
# set to True to append the seconds, False to truncate them
ADD_SECS = False
# set True to force the seconds to be '00', ignored if ADD_SECS = False
           # tuple of device id and storage directory tag name
DEV_LIST = [ ("00010000-00000000-03560210-13650987", 'x3_aio'),
             ("00000000-00000000-00409DFF-FF36EE2C", 'demo_solar'),
# In this example, the archive will store files as as:
#   /home/idigi/archive/x3_aio/x3_aio_20100812_1045.xml
#   /home/idigi/archive/x3_aio/x3_aio_20100812_1100.xml
#   /home/idigi/archive/demo_solar/demo_solar_20100812_1045.xml
#   /home/idigi/archive/demo_solar/demo_solar_20100812_1100.xml
# and so on
def log_into_idigi(username, password, idigiurl):
    '''Open the HTTP socket with login to'''
    pwdManager = urllib2.HTTPPasswordMgrWithDefaultRealm()
    pwdManager.add_password(None, idigiurl, username, password)
    authHandler = urllib2.HTTPBasicAuthHandler(pwdManager)
    opener = urllib2.build_opener(authHandler)
def get_file_list(idigiurl, devid):
    '''Fetch a list of files in this device storage, which
    are returned as a list of tuples such as:
     [('data02.xml', '20100812_1045'),
      ('data03.xml', '20100812_1100')]
    These represent the file name in idigi and modified time
    dataurl = "%s/ws/data/~/%s/" % (idigiurl, devid)
        f = urllib2.urlopen(dataurl)
        data =
        # print data
    except URLError, e:
        print e.code
    data = data.split('<')
    lst = []
    for ln in data:
        x = parse_fileinfo( ln)
        if x is not None:
    return lst
def parse_fileinfo( ln):
    '''Return a tuple of file name and string of modified date,
    such as ('data02.xml', '20100812_1045')
    # you could expand this to use SAX or another XML parser, but
    # since we just extract 2 pieces of info per line, this brute
    # force method works and is likely faster
    if ln.startswith('exist:resource'):
        # print 'ln:%s' % ln
        x = ln.find('name=')
        if x >= 0:
            st = ln[x+6:].split('"')
            name = st[0]
            x = ln.find('modified=')
            if x >= 0:
                st = ln[x+10:].split('"')
                date = st[0]
                date = date[0:4] + date[5:7] + date[8:10] + '_' + \
                       date[11:13] + date[14:16]
                if ADD_SECS:
                    # then user wants the seconds in the name
                    if ZERO_SECS:
                        # then just treat as seconds
                        date += '00'
                    else: # else save actual
                        date += date[17:19]
                return (name, date)
    return None
def get_one_file(idigiurl, devid, filnam):
    '''Download one file as a string'''
    dataurl = "%s/ws/data/~/%s/%s" % (idigiurl, devid, filnam)
        f = urllib2.urlopen(dataurl)
        data =
    except URLError, e:
        print e.code
    return data
def main():
    for dev_id, dev_nam in DEV_LIST:
        # verify the archive sub-directory exists
        arch = os.path.join(ARCHIVEBASE, dev_nam)
        if not os.path.exists(arch):
            print "Creating archive dir <%s>" % arch
            print "Device <%s> archive dir <%s> exists" % (dev_id,arch)
        # get the file list for this device        
        log_into_idigi(USERNAME, PASSWORD, IDIGI_URL)
        files = get_file_list(IDIGI_URL, dev_id)
        # print files                
        for fil_nam, fil_tim in files:
            save_file = os.path.join(arch, dev_nam + '_' + fil_tim + '.xml')
            if os.path.exists(save_file):
                print "Archive file <%s> already exists, skip download" % arch
                data = get_one_file(IDIGI_URL, dev_id, fil_nam)
                print data
                    print 'Saving file %s ...' % save_file,
                    f = open(save_file,'wb')
                    print 'done.'
                except URLError, e:
                    print e.code
if __name__ == '__main__':
Personal tools
Wiki Editing