weeWX Calduino Top

Creating a WeeWX extension to integrate Calduino

weeWX Calduino Top

Introduction

In the two previous posts we commented the main characteristics of the EMS Bus and described Calduino, the hardware and software used to connect and exchange information with a Buderus boiler though this bus. In this post you will see how to create a weeWX extension to import the information gathered and how to control the boiler configurations from an HTTP web page.

Creating a WeeWX Extension

WeeWX has a data/event driven architecture. It uses different drivers to communicate with the connected weather stations (hardware). When a sensor spits some data out, weeWX does something. This action can be either generate an HTML report or perform some calculations with the data gathered. These actions are called services.

The drivers are Python code that reads from serial port/USB/network interface the bytes sent by the hardware and decodes them, putting the resulting data into LOOP packets and archive RECORDs. In my case, I use the WMR89 driver to recieve the meteorological conditions. A weeWX instance can only have one driver, so in order to incorporate a new source of information, one or more services should be incorporated.

WeeWX services are bound to events and there can be more than one service bounded with the same event. There are different types of services, listed in the following table:

Service list Function
prep_services Performs any actions before the main loop is run.
data_services Augments data, before it is processed.
process_services Processes, filters, and massages the data.
archive_services Records the data in a database.
restful_services Uploads processed data to an external RESTful
service.
report_services Runs any reports.

WeeWX Services

In general terms we want to augment the data included in the archive RECORD and save it in a database. To so do, two data services will be needed:

  • First of all, a data service (called CalduinoMonitor) will include Calduino variables in every archive RECORD generated. In my case, the archives are generated in a five minutes pace (archive_interval parameter in weewx.conf). This can be achieved in two ways. The easiest one would be to send a Get All Monitors command to calduino (operation 29) and parse the XML results. However this get operation takes on average 18 seconds to be performed, blocking weeWX engine during this time. To avoid this I have created a Linux service that downloads the output of this get command every five minutes, before being imported by weeWX. By doing this, the data service will only have to access the last XML file and import the values.
  • At the end, an archive service (called CalduinoArchive) will store the data in a new database that will only contain the information gathered by Calduino. I have decided to create a new database and not to extend the default database following WeeWX recommendation: extensions should not modify the database schemas.

WeeWX Database

As it has been commented before, a new database will be used to save Calduino data. It will just contain a number of the parameters included in the Get All Monitors command. Not all the parameters contained in this command will be saved. Some of them are just current configurations/status of the boiler. It may be of interest to show them in the HTML report in real time but it makes no sense to save them for the future.

The following table summarizes Calduino’s database schema. Note that the field names coincides with the used in the XML tags:

Field name Type Comments
dateTime Integer Not null, primary key (weeWX default field)
usUnits Integer Not null (weeWX default field)
interval Integer Not null (weeWX default field)
extTemp Real
selImpTemp Integer
curImpTemp Real
retTemp Real
boilTemp Real
selTempDHW Integer
curTempDHW Real
selRoomTempHC1 Integer
selRoomTempHC2 Integer
selRoomTempHC3 Integer
selRoomTempHC4 Integer
selImpTempMM10 Integer
curImpTempMM10 Real
selBurnPow Integer
curBurnPow Integer
pumpMod Integer
modMM10 Integer
flameCurr Real
sysPress Real
UBAWorkMin Integer
burnWorkMin Integer
burnWorkMinH Integer
burnWorkMinDHW Integer
burnStarts Integer
burnStartsDHW Integer

WeeWX Skin

The last service being run in every weeWX iteration is weewx.engine.StdReport. This is the standard service for creating reports. Each report has a skin associated with it, which contains the templates, any auxiliary files such as background GIFs or CSS style sheets, and a skin configuration file, named skin.conf. In order to see the data gathered and manage remotely the boiler, a new skin, named calduino and packed in this extension, has been developed.

WeeWX Generators

To create their output skins rely on one or more generators, which are those that do the actual work, such as creating HTML files or plot images. The Calduino Skin included in this extension is just a proof of concept and uses only two generators:

  • The CheetahGenerator will generate the HTML web page files from templates, using the Cheetah template engine. A template is a text file, ended with the .tmpl extension by convention, that is processed by a template engine to create a new file. The Cheetah template engine finds tags by scanning a search list, a Python list of objects. To include Calduino tags in the search list, I have created a search list extension called CalduinoSearchList. This will make some of the data imported from Calduino available in the report but not saved in the database, such as Calduino’s uptime, number of operations failed, etc.
  • The CopyGenerator controls which files are to be copied over from the skin directory to the destination directory. It is used to copy CSS, as well as Javascript files and icons. This files are static so they are copied only once on the first invocation of the report engine service.

Additionally, you may also use other generators such as the GaugeGenerator to show, in a friendly way, the information gathered by Calduino. I have developed, based on the work of Nick Dajda’s Gauge Generator, a type that can include up to three observations from different data sources. More information can be found in this project.

Finally, you will surely want to use the ImageGenerator to create graphics regarding the boiler status. It is up to you decide which generators you want to run and what information you want them to provide.

Installing Calduino Extension

To install the extension using the wee_extension utility follow this steps:

  1. First of all download the last calduino extension from the releases tab on the project GitHub site and save the downloaded file to a directory accessible from the machine running weeWX. In these instructions the /tmp directory will be used.
  2. Run the installer with a user with privileges. This command assumes that wee_extension is included in user’s path, otherwise the path to wee_extension will be required.

    This will result in output similar to the following:
  3. Adapt the configuration files to your installation. Update, if required, the weewx.conf to change the location of the XML files to import in the [Calduino] section. Default configuration presumes an sqlite database. Change the database_type attribute in [Databases] section to mysql if needed.
  4. The tarball file contains an example of service to obtain from the boiler periodically its status in XML format. It is located in /tmp/pollCalduino folder. It is highly tailored to my needs, so most likely you will have to update it. More information about this issue in this section.
  5. Restart weeWX:

First run

In the first run WeeWX will create a new database named calduino.sdb with the previously commented schema. As stated above, the Calduino Skin and templates included in this extension is just a proof of concept with very few options. Edit the template files to include graphics and/or gauges of your boiler as desired.

The file calduinoControlPanel.html.tmpl contains an AJAX form/panel that interacts synchronously with the Calduino device connected to the EMS Bus. It has been designed using Bootstrap. Javascript is used for calculations and user interaction. In the server side, a PHP file calduinoWrapper.php.tmpl performs the communication with the Calduino device using libcurl library.

The next image is an example of Calduino Extension running on my weeWX.

weeWX extension Calduino Full Screen

Poll Calduino Service

Included in the extension, in /tmp/pollCalduino folder you will find a bash script called pollCalduino.sh. It will be in charge of polling Calduino device connected to the EMS Bus and updating the XML file that will be imported by weeWX’s extension. To do so, it will launch the php operation getGeneralMonitor  every minute ended in 3 or 8 (as explained in this post). To work properly it should include the IP address of Calduino in  \etc\hosts.

When a communication problem is detected (i.e. the operation fails), an IFTTT notification is sent to my telephone. The steps done to configure your IFTTT account were explained in this entry. You should just create a service named calduino_error and include your IFTTT key in the script. Moreover, I use a 433 MHZ transmitter module installed in the Raspberry to remotely turn off and on the calduino device, as it is connected to a radio frequency controlled socket. More information about this can be found here. If you are not going to use this functions just comment these lines.

Some comments

  • It happens infrequently, but sometimes the pollCalduino operation fails due to collisions in the EMS Bus. The operation is retried up to 5 times to avoid this issue. In case the problem is related to the WiFi network, a restart (switching off and on the RF socket) is performed as it will probably solve the problem.
  • It should be possible to pass the logic of the Poll Calduino Service to weeWX. Calling directly calduino from python is not a good idea, since this code sits right in the main loop of the weeWX engine. Considering that it takes up to 18 seconds to obtain a general monitor file, a different approach should be used. For instance, this code could be included in a separate thread and feed the results to Calduino Monitor through a queue. I left this improvement to the future.

2 thoughts on “Creating a WeeWX extension to integrate Calduino”

  1. Pingback: Calduino: Connecting Arduino with the EMS Bus - Domotic Project
  2. Pingback: Understanding the EMS Bus of Buderus / Nefit boilers - Domotic Project

Leave a Comment