Concept and Implementation of a data logging system with Python

At the beginning my experience with python was not very solid. So I didn’t trust my own scripts to run stable. When I start a script that is supposed to run all day, but it crashes at some time because of a bug, I would loose a lot of data until I realize the script doesn’t run anymore. Therefore I implemented a scheme where the script would collect data over 5 minutes and then terminate. If it crashes, I would loose data of a 5 minutes period only. A cron job can start the same script every 5 minutes.

concept

a python script is started by a cron job every 5 minutes. The script will collect data from the inverter 10 times every 30 seconds, so its total run time is 4 minutes 30 seconds. At the end of the script the collected data are processed, and average, min and max values are stored. I store  energy production, grid voltage, grid frequency, PV solar panels voltage and Raspberry Pi temperature in a data base.

The crontab entry looks like this:

crontab_a

*/5     means every 5 minutes

5-22  means between 5:00 am and 10:00 pm

* * *  means every day / every month / every day of week

>/dev/null means all output (stdout) is diverted into the trash bin

2>&1 means error messages are diverted to stdout so go into the trash bin

implementation of the data logging system

data are stored in csv  format in text files, i.e parameters  are plain text, separated by commata. The set of data recorded each 5-minute period is stored in a file. This file has the format [week-of-year]_[year].csv. 
As its name suggests, a new file will be generated for each week.

Continue with Fan control

Advertisements