Read and Write Flash USB

From Digi Developer

Digi products supporting Python allow you to read or write files from FLASH - both the main system flash and (if available) USB flash drives with standard FAT formatting.

Big Warning

We start with the warnings, because a bad (stupid) program design on your part will destroy your product in a few months. Flash memory is NOT a hard drive, so do not plan on using it like one.

Do not plan to write to flash every second!

Many programmers start with the idea "I'll save my status (or rewrite a web page) every second." You cannot do this, so instead:

• Dynamic Web pages should be handled using the Module:digiweb, which allows the product's main web server to pass unknown URL to your python program. This allows you to have dynamic web pages which are always up-to-date without writing to flash at all.
• For data logs - be reasonable. Assuming your product isn't crashing and rebooting every few seconds, then the primary reason for RAM-based data to be lost is when the product loses power. So stop and think - if the power is out for 30 minutes or 7 hours hours, how earth-shattering is losing 35 minutes or 7 hours & 5 minutes worth of data instead of 30 minutes and 7 hours respectively?
• If you really require constant update, do so ONLY in external USB flash drives because they are much faster to update and not critical to the survival of the product.

Do not plan to maintain a complex database system!

While there are no seek times under Flash, the slowness of the updates means a power glitch or reboot will corrupt your data. The best designs are either flat files which are only appended to until they need to be trimmed to reduce size, or flat files with simple fixed sized records. Any indexing or sort orders should be maintained ONLY in RAM and be manually rebuilt anytime the product restarts.

Do not use write() to build large data blocks!

While it is temping to use a for-loop to write 20 strings to a file in 20 write() calls, the proper solution is to assemble all 20 strings into one larger string in RAM, then use a single write() to update Flash. This will cause a noticeable improvement in both performance and Flash wear.

Always use Try/Except around file operation

Complex operating systems (Windows/Linux/MAC) do complex things to enable graceful multi-user access of the shared filesystems; The Digi embedded OS is not a complex multi-user system. Reading/Writing a USB drive will fail if the user pulls out the drive while you are updating. Reading/Writing the main flash might fail if the user deletes or replaces the files through the Web interface. Multiple threads affecting the filesystem might impact each other, and certainly two threads CANNOT share the same file unless the file is protected by a semaphore.

In general do NOT expect files to remain open forever

Assuming you are (correctly) updating your file only once per few minutes, then you should close and later reopen the file.

Code Examples

Now that the warnings have been issued, here are code fragments used in a successful data logging application. In general, the file I/O routines like open/read/write/close work. The functions like os.access() do NOT work.

Testing File Size

This routine (which we assume is part of a log-manager class) allows a thread to test the file size. Note the blocking semaphore which protects the flash file if multiple threads share the log, which in this example is true because one thread adds to the file, and another reads data from it for serving out to remote TCP clients.

    def get_bytes_in_file(self):
        """Return Byte count in FLASH file"""
 
        self.disklog_acquire_blocking()
        try:
            fileHan = file(self.get_disklog_filename(),'rb')
            fileHan.seek(0,2)
            self.bytes_in_file = fileHan.tell()
            # print 'Discover %d bytes in file' % self.bytes_in_file
            fileHan.close()
 
        except:
            # traceback.print_exc() - in case this should never happen
            # print "File not found"
            self.bytes_in_file = -1
 
        self.disklog_release()
        return self.bytes_in_file

Append Data to the File

This routine (which we assume is part of a log-manager class) takes a LIST of text or binary strings and appends them to a Flash file. In this application the data is being appended to a list of strings held only in RAM, then every five (5) minutes an event causes the disklog manager to dump the accumulated data to the Flash file.

A number of the functions here are not explicitly shown. For example, before appending the data the code needs to confirm if the file size has grown to the point it should be trimmed (cut in half for example). It also assumes a simple file "header" exists, which might be little more than some text showing the time/date the file was started (or was last compacted).

One simple method of "compaction" with low Flash overhead is to maintain two files, each one-half the maximum desired size. When the active one fills up, the older file (older half) is deleted and a new file with time-stamp in the name is used to start a new half. The actually application being shown in this example has both data-item and time-stamp info in fixed size records, thus its compaction is rather complex since it wants to delete the OLDEST records, yet to maintain some minimum number of records for each data-item.

    def disklog_append_rec_list(self,rec):
        """Store/Append LIST of bin-string to log file"""
 
        # pack the list into a single bin-string, we do NOT want to
        # issue any more than one (1) write() call
        rec = "".join(rec)
 
        # psuedo-code here, you'll need to consider this action
        # - test if the existing file will be TOO large if we add this
        # - if it is, then we need to COMPACT the file first
 
        bResult = False
        self.disklog_acquire_blocking()
        try:
            fileHan = file( self.get_disklog_filename(),'ab') # append
            if(fileHan.tell()==0): # then is NEW file
                print "Starting new log file %s" % self.get_disklog_filename()
                fileHan.write(self.make_diskfile_header())
            fileHan.write(rec)
            self.bytes_in_file = fileHan.tell()
            fileHan.close()
            bResult = True
 
        except:
            # else just make as EMPTY
            traceback.print_exc()
            self.bytes_in_file = -1
 
        self.disklog_release()
        return bResult

Delete the File

This routine (which we assume is part of a log-manager class) deletes a file by resetting it to zero bytes.

    def disklog_delete(self):
        """Delete FLASH file"""
 
        bResult = False
        self.disklog_acquire_blocking()
 
        try:
            fileHan = file( self.get_disklog_filename(),'wb') # clobber, don't append
            fileHan.write(self.make_diskfile_header())
            fileHan.close()
            bResult = True
 
        except:
            traceback.print_exc()
 
        self.disklog_release()
        return bResult

Using Main Flash

In general you only want to use the internal Main Flash for configuration files, or for data records which are NOT updated any faster than once per five (5) minutes. This is because:

• The internal Flash is fairly slow to update, which impacts your Python program
• Digi's main flash is NOT wear-leveled; certain critical sectors are used extensively and will fail fastest!
• Updating this flash too often will render the product inoperable (meaning it breaks and you must BUY A NEW Digi device)
• The size is limited, and future firmware upgrades may unexpectedly squeeze the size even smaller.
• Storing data here consumes space desired for Python code

How to Access

The main FLASH appears as the directory "WEB/python". Creating a file as "WEB/python/data.csv" puts the file into Main Flash. The really nice thing about this main flash is the files show up within the Web Interface as the Python files for upload. For example, if you upload a file named "run.py" via the Web Interface, it can be accessed as "WEB/python/run.py"

Using USB Flash Drives

In general using external USB Flash for Log and Event files is safest because:

• It has no impact on the critical internal Flash
• The FAT file system supported is faster, more standard, and of course allows you to move the files to a normal computer
• Larger sizes are possible - as of Oct-2008 once can easily buy 2GB to 4GB for from us$9 to $19.

How to Access

Any supported USB Flash drives appear as mounted drives, so the first USB port is "A/", the second is "B/" and so on. There is no drive ':', so it is NOT A:/. Creating a file as "A/data.csv" puts the file out in the USB Flash Device.

Questions: Does mkdir work? How to create/delete subdirectories? Questions: How to see USB files from Web UI or CLI/Telnet Questions: How to read space usage - how much free space exists

Products known NOT to work

• In general, many of the U3-style Flash sticks which support PASSWORD or ENCRYPTION do not work. To a Windows computer they look NOT like a FAT file system, but like a small CD-ROM (read-only) drive, which runs a Windows application, which asks the user for password and only mounts the normal FAT filesystem once the password has been accepted.
• Sony MicroVault Tiny (example: USM2GH, tested Jun-2008) does not work because it has built-in hardware compression which assumes use in Windows computers, so is NOT a simple FAT file system. Since these USB drives are literally about the size of cell-phone SIM, this is a big disappointment!