DIA Device - Sample Rate Reducer Filter

From Digi Developer

Jump to: navigation, search


Sample Rate Reduction/Filter Device

Consider a DIA system with 10 tanks and sleeping level sensors which wake once per hour to upload new data to X4 and DIA. Do you really need to push 240 data samples per day over your cost-sensitive cellular link into the Device Cloud? What if on the average day, only 1 or 2 of those tank levels even change?

The DIA 'Filter Device' allows you to add very simple intelligence around the forwarding of new data. In this tank example, it would allow the DIA to apply the logic "update Device Cloud only when the tank level changes by at least 1%, plus never update more often than once per hour, but always update at least once per day." This could reduce the total data samples moved over cellular to only 15 or 20 samples per day - while still allowing the central host to see tank level changes within an hour of the change.

The filter_device acts as a simple, time-aware filter (or transform) which applies process knowledge to reduce the frequency of sample updates. In the above example, the DIA configuration would start with ten (10) sensor device objects which blindly produce new data samples every hour - so 240 changes per day. These might have names like ['TNK01in', 'TNK02in' ...] and so on, which means 'TNK01.channel1_value' might be the level in the first tank.

Ten (10) new filter_device objects would be created to selectively copy or mirror the output of those sensor device objects, having names like ['tank01', 'tank02' ...] and so on. They might apply a logic with the Python dictionary: { 'delta':1.0, 'atleast':'24 hr', 'atmost':'59 min' } They might also rename their 'input' to become output as 'level'. So they create new channels such as 'tank01.level' which change at most 24 times per day, but at least 1 time per day. If idigi_db uploads the channels from the filter_device instead of from the sensors, then you will have less cellular traffic.

Filter Conditions / Configuration

By default each filter_device supports at most 4 channels - this is hard-coded within the filter_device.py since it creates database items EVEN for channels which are NOT configured. The syntax of the filter_device is closer to Python to reduce the system overhead in created such 'mirror channels' on a large DIA system.

The filter_device is configured with:

  • a tag name ['tag1', 'tag2', ... 'tagN'] with a source channel name such as template.counter or m3_01.temperature. By default, the outgoing filter_device channel has the same name as the source. Duplicate names causes exceptions to be thrown.
  • an optional rename ['rename1', 'rename2', ... 'renameN'] which can over-ride the source channel name. For example, instead of a name like 'distance', you might want 'level'.
  • a filter condition dictionary ['filter1', 'filter2', ... 'filterN'] which defines the list of conditions to suppress or cause the output channel to be updated.

Filter Conditions

  • all channels will produce an initial sample even if the conditions are not met.
  • an empty dictionary such as "filter1: {}" causes the data to always propagate, otherwise by default the data is NOT propagated if conditions exist.
  • the "atleast" clause over-rides all others and produces a new data sample AT LEAST as often as configured time in SECONDS. Examples are { 'atleast':'24 hr'} or { 'atleast':86400}, where the string '24 hr' or '1 day' would be internally converted to 86,400 seconds.
  • the "atmost" clause also over-rides others and suppresses new data samples to happen no more that the configured time in SECONDS. Using filter conditions to track 'events' might result in data loss if 'atmost' suppresses samples. Examples are { 'atmost':'15 min'} or { 'atmost':900}
  • the "delta" clause produces a new data sample if the value changes up or down by at least the value configured. The value depends on the units, so for a level measured in inches the delta-value would be inches. Examples are { 'delta':0.5} or { 'delta':25}
  • the "below" clause produces a new data sample if the value is less-than (below) the value configured. There is no hysteresis, so an 'atmost' clause may be required to slow down new data samples.
  • the "above" clause produces a new data sample if the value is greater-than (above) the value configured. There is no hysteresis, so an 'atmost' clause may be required to slow down new data samples.
  • the "equal" clause produces a new data sample if the value is the same as the value configured. Be warned that sensor inputs treated as floats will rare 'equal' a constant such as 2.5 or 100. (It is a TODO item to offer a solution for this)
  • the "round" clause is processed ONLY if the sample is being produced, and it forces the sample to become a float. It is primarily of use when data uploads are handles as ASCII text. For example, it makes no sense to upload a temperature as "21.094400000000007" if the accuracy is +/- 1 degree. Adding the condition { 'round':1 } would round the float to 21.1000000, which normally will be ASCII encoded as only "21.1"

Example YML File for TemplateDevice

This example produces two new channels with the following characteristics:

  • m3_01.counter only produces a new sample (SETs a new output) if the data sample is greater than 6. However it produces atleast 1 sample every 5 minutes (300 seconds) and never produces more than 1 new sample every 1 minute and 10 seconds (70 seconds). This channel inherited its name from the source channel template.counter
  • m3_01.other produces a new sample (SETs a new output) after the source channel has changed at least by a value of 3 (example: 1, 4, 7 etc). It has no time restrictions.

  - name: template
    driver: devices.template_device:TemplateDevice
        count_init: 0
        update_rate: 1.0

  - name: m3_01a
    driver: devices.filter_device:FilterBlockDevice
        tag1: template.counter
        filter1: { 'above':6, 'atleast':'5 min', 'atmost':'1 min 10 sec' }
        tag2: template.counter
        rename2: 'other'
        filter2: { 'delta':3 }

Example YML File for Massa M3 Ultrasonic Sensor

The Massa M3 battery powered device wakes on schedule - for example once per hour - and pushes new data into the Device Cloud/DIA channel database. However, this does NOT mean that a new tank level has been seen. Also the Massa M3 driver produces about 12 channels, all which are 'refreshed' with a new sample at this rate. So just uploading all changed channels wastes a lot of cellular bandwidth.

This example produces four new channels with the following characteristics:

  • m3_01.distance produces a new sample if the level changed by at least 0.5 inch, but produces atleast 1 sample every 3 hours and never more than 1 new sample per hour.
  • m3_01.temperature produces a new sample if the temperature changed by at least 2 degrees, but produces atleast 1 sample every day and never more than 1 per 2 hours.
  • m3_01.target_strength only produces a new sample if the ultra-sonic signal quality is less than 50%, never more than 1 per hour.
  • m3_01.battery produces a 1 new sample every day.

  - name: m3_01
    driver: devices.vendors.massa.massa_m3:MassaM3
        xbee_device_manager: xbee_device_manager
        extended_address: "00:13:a2:00:40:4b:ae:b0!"
        poll_rate_sec: 3600
        sample_rate_sec: 3600

  - name: m3_01a
    driver: devices.filter_device:FilterBlockDevice
        tag1: m3_01.distance
        filter1: { 'delta':0.5, 'atleast':'3 hr', 'atmost':'1 hr' }
        tag2: m3_01.temperature
        filter2: { 'delta':2.0, 'atleast':'24 hr', 'atmost':'2 hr' }
        tag3: m3_01.target_strength
        filter3: { 'below':50, 'atmost':'1 hr' }
        tag4: m3_01.battery
        filter4: { 'atleast':'24 hr' }

Download Info

Files in the filter_device.zip

  • readme_filter_device.txt is a summary similar to this Wiki page.
  • src\common\helpers\parse_duration.py is a Python routine to convert strings like '24 hr' into msec or sec. It supports the tags ['ms','sec','min','hr','day']
  • src\common\helpers\_test_parse_duration.py is a Python regression test routine.
  • src\devices\filter_device.py is the DIA device driver to be referenced in your YML file.
Python Code
  • the tag interface is cumbersome and wasteful - having tags named tag1/filter1 and tag2/filter2 is not ideal. A method should be added to accept them all as tag/filter and be handled properly.
  • since 'equal' doesn't work with floats, adding a 'nearly' tag to match when a float is within some tolerance of a target would be good.
  • enable inverting boolean conditions, so a sample of 'True' can be forwarded as 'False'.
Personal tools
Wiki Editing