DIA Event Uploader
From Digi Developer
Uploading Events or discrete data samples to DIA
Comparing iDigi_DB to iDigi_Upload
The stock iDigi upload presentation (named iDigi_DB) assumes you only want a sampling (or subset or snap-shot) of data samples handled by DIA. For example, if 20 tanks send new level reading to Dia once per minute, this amounts to 1200 samples per hour. Yet the Device Cloud host may not desire any more than 20 samples per hour - just one per tank per hour. Therefore the stock iDigi_DB presentation is configured with an interval setting (for example: 3600 seconds or 1 hour), then all configured channels are sampled and uploaded to Device Cloud once per hour.
This design does not support uploading real time events. For example, if a door sensor is monitored, then Device Cloud will only see the door status on the hour (the sample interval). Device Cloud will not record that the door was opened three times during the hour.
iDigi_Upload is a custom adaptation of the original idigi_db.py file, modified to save a copy of every new channel sample for upload. These are saved in a cache and only cleared after successful upload.
|Are discrete events uploaded to Device Cloud?||No - only the I/O status at the sample interval||Yes - every new sample is uploaded|
|If upload fails, are samples saved and retried?||No - new samples will be taken at the next sample interval||Yes - copies are made of every sample and only deleted after successful upload (unless cache grows beyond configured limit)|
|If a channel doesn't change, is it uploaded every interval?||Yes - all channels are sampled at each sample interval||No - copies are made only when sample change|
|Is the upload size limited per interval?||Yes - at most one sample of every channels is sent||No - every changed sample is uploaded|
|Is memory usage limited?||Yes - existing samples are used to create upload||No - a duplicate copy of every changed sample is held until upload|
iDigi_Upload supports the original iDigi_DB settings, plus adds two new settings:
Is the maximum number of bytes of flash to use buffering for upload. It defaults as 0, which disables this drastic changes of behavior.
- When cache_size = 0, idigi_upload holds all data in memory only. Restarting the gateway will discard your data.
- When cache_size is set > 0, then idigi_upload queues a deep-copy of every new sample first in memory, then saves this to flash approximately once a minute. You will see files such as 'data_cache.txt' appear and disappear from your Python directory. These are cleared only after upload or when the caches fills, then in FIFO design the oldest data is lost. NOTE that the design keeps a backup when deleting the cache, so the actual size of flash required will be twice the configured amount.
Is an alternative to the base interval setting. It accepts only minutes in the set (0, 1, 2, 3, 4, 5, 6, 10, 12, 15, 20, 30, 60)! It causes upload ON THE HOUR, then minute intervals in a predictable pattern. So setting 10 cause uploads at 00:00:00, 00:10:00, 00:20:00 and so on.
- Setting clean_minute_interval to zero (0) causes the base interval setting to be used.
- Setting clean_minute_interval to non-zero over-rides the interval setting.
- Python Code
This version requires Dia 1.4.14 or newer. The files includes are: