{"id":1453,"date":"2024-11-03T13:37:41","date_gmt":"2024-11-03T11:37:41","guid":{"rendered":"https:\/\/www.rocworks.at\/wordpress\/?p=1453"},"modified":"2024-11-03T21:18:47","modified_gmt":"2024-11-03T19:18:47","slug":"exploring-my-home-automation-architecture","status":"publish","type":"post","link":"https:\/\/www.rocworks.at\/wordpress\/?p=1453","title":{"rendered":"Exploring My Home Automation Architecture"},"content":{"rendered":"\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-2024-11-03-at-20.10.55.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"569\" src=\"https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-2024-11-03-at-20.10.55-1024x569.png\" alt=\"\" class=\"wp-image-1459\" srcset=\"https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-2024-11-03-at-20.10.55-1024x569.png 1024w, https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-2024-11-03-at-20.10.55-300x167.png 300w, https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-2024-11-03-at-20.10.55-768x427.png 768w, https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-2024-11-03-at-20.10.55-1536x853.png 1536w, https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-2024-11-03-at-20.10.55-2048x1137.png 2048w, https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-2024-11-03-at-20.10.55-672x372.png 672w, https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-2024-11-03-at-20.10.55-1038x576.png 1038w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/a><\/figure>\n\n\n\n<p>My home automation setup has been an ongoing project, evolving over the last 10\u201315 years. It started with a simple goal: to track data from my photovoltaic (PV) system, power meters, and temperature sensors, all connected through Raspberry Pi devices. It started with Oracle and over time, it\u2019s grown into a more complex architecture that incorporates multiple layers of data collection, processing, and visualization.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Bildschirmfoto-2024-11-03-um-12.27.56.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"718\" src=\"https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Bildschirmfoto-2024-11-03-um-12.27.56-1024x718.png\" alt=\"\" class=\"wp-image-1455\" srcset=\"https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Bildschirmfoto-2024-11-03-um-12.27.56-1024x718.png 1024w, https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Bildschirmfoto-2024-11-03-um-12.27.56-300x210.png 300w, https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Bildschirmfoto-2024-11-03-um-12.27.56-768x538.png 768w, https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Bildschirmfoto-2024-11-03-um-12.27.56.png 1462w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/a><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">1. Data Collection with Raspberry Pi and MQTT<\/h3>\n\n\n\n<p>At the core of my setup are <strong>Raspberry Pi devices<\/strong> that connect various sensors, including those for monitoring power generation (PV) by bluetooth, power consumption by meters giving digital signals, and temperature sensors. These Pi devices act as data collectors, feeding data into a <strong>local<\/strong> <strong>Mosquitto <\/strong>broker. A local broker on the device can serve as a short-term buffer, before it\u2019s synchronized to my <strong>central<\/strong> <strong>MonsterMQ <\/strong>broker, by using a persistant session and QoS&gt;0.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">2. MonsterMQ Broker as the Central Hub<\/h3>\n\n\n\n<p>The <strong>MonsterMQ broker<\/strong> is the central point where data from all sources is collected. It serves as a bridge, collecting data from the local Mosquitto broker and preparing it for further processing and storage. Before building my own broker, MonsterMQ, I used Mosquitto. Now that I have my own broker, I use MonsterMQ, both to ensure it gets thoroughly tested and to leverage its features. Additionally, in the future, I can use MonsterMQ to store incoming values directly in Apache Kafka. As a database engineer, I appreciate MonsterMQ because it allows me to view the broker&#8217;s current state by querying a PostgreSQL database. This lets me see connected clients, their connection details, source IP addresses, and all subscriptions with their parameters..<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">3. Automation-Gateway for Data Flexibility<\/h3>\n\n\n\n<p>To expand the possibilities of what I can do with the data, I use the <strong>Automation-Gateway<\/strong>. This tool collects values from MonsterMQ and serves two primary functions:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Integration with Apache Kafka<\/strong>: By publishing data to <strong>Apache Kafka<\/strong>, I maintain a reliable stream that acts as an intermediary between my data sources and the storage databases. This setup provides resilience, allowing me to manage and maintain the databases independently while keeping the data history intact in Kafka.<\/li>\n\n\n\n<li><strong>OPC UA Server Exposure<\/strong>: The Automation-Gateway also exposes data as an <strong>OPC UA server<\/strong>, making it accessible to industrial platforms and clients that communicate over OPC UA. This can be achieved just with a simple YAML configuration file.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">4. Experimental Integrations: Ignition and WinCC Unified<\/h3>\n\n\n\n<p>On top of this setup, I\u2019ve added experimental connections to <strong>Ignition<\/strong> and <strong>WinCC Unified<\/strong>. Both of these platforms connect to the Automation-Gateway OPC UA Server. Just for testing those systems are publishing values to my public MQTT broker at <a href=\"http:\/\/test.monstermq.com\/\"><strong>test.monstermq.com<\/strong><\/a>. While these integrations aren\u2019t necessary, they\u2019re helpful for testing and exploring new capabilities.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">5. Long-Term Data Storage with TimescaleDB, QuestDB, and InfluxDB<\/h3>\n\n\n\n<p>Data from Kafka is stored in multiple databases:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>InfluxDB: <\/strong>My home-automation started with Oracle and then moved to InfluxDB<\/li>\n\n\n\n<li><strong>TimescaleDB<\/strong>: Since I am still an advanced SQL user, I needed a database with strong SQL capabilities. Therefore, I added TimescaleDB and imported historical data into it.<\/li>\n<\/ul>\n\n\n\n<p>Amount of records as of today: 1_773_659_197<\/p>\n\n\n\n<p>Additinally the Automation-Gateway is writing the data now to <strong>QuestDB<\/strong>. It is used for experimental data logging and alternative time-series database exploration. Potentially will replace the other databases. I was blown away by how quickly I was able to import 1.5 billion historical data points into QuestDB.<\/p>\n\n\n\n<p>These databases serve as long-term storage solutions, allowing me to create detailed dashboards in <strong>Grafana<\/strong>. By keeping Kafka as the layer between my data sources and the databases, I ensure flexibility for database maintenance, as <strong>Kafka<\/strong> retains the historical data.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Bildschirmfoto-2024-11-03-um-11.55.49.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"557\" src=\"https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Bildschirmfoto-2024-11-03-um-11.55.49-1024x557.png\" alt=\"\" class=\"wp-image-1456\" srcset=\"https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Bildschirmfoto-2024-11-03-um-11.55.49-1024x557.png 1024w, https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Bildschirmfoto-2024-11-03-um-11.55.49-300x163.png 300w, https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Bildschirmfoto-2024-11-03-um-11.55.49-768x418.png 768w, https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Bildschirmfoto-2024-11-03-um-11.55.49-1536x836.png 1536w, https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Bildschirmfoto-2024-11-03-um-11.55.49.png 1538w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/a><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">6. Data Logging with MonsterMQ<\/h3>\n\n\n\n<p>The public MonsterMQ broker is configured to write data of topics below &#8220;grafana\/#&#8221; directly into a TimescaleDB table. This setup allows you to see updates in Grafana whenever new data is published. In this specific Grafana dashboard configuration, if you publish a JSON object with a key &#8216;value&#8217; and a numeric value, such as {&#8220;value&#8221;: 42}, it will appear on the dashboard almost instantly. Here is a public <a href=\"http:\/\/test.monstermq.com:3000\/public-dashboards\/4f91fe583e5349e9a17c2b581dc0d448\">dashboard<\/a>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>select \n  time, array_to_string(topic,'\/') as topic, \n  (payload_json-&gt;&gt;'value')::numeric as value \nfrom grafanaarchive\nwhere $__timeFilter(time)\nand payload_json-&gt;&gt;'value' is not null\nand payload_json-&gt;&gt;'value' ~ '^&#91;0-9]+(\\.&#91;0-9]+)?$'\norder by time asc<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">7. SparkplugB Decoding with MonsterMQ<\/h3>\n\n\n\n<p>The public <strong>MonsterMQ broker<\/strong> is configured to decode and expand <strong>SparkplugB<\/strong> messages. Expanded messages can be found under the topic &#8220;spBv1.0<strong>e<\/strong>&#8220;. Ignition publishes some of my home automation data via SparkplugB to the public broker, and you\u2019re welcome to publish your own SparkplugB messages here as well.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><a href=\"https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Bildschirmfoto-2024-11-03-um-12.11.59.png\"><img loading=\"lazy\" decoding=\"async\" width=\"690\" height=\"621\" src=\"https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Bildschirmfoto-2024-11-03-um-12.11.59.png\" alt=\"\" class=\"wp-image-1457\" srcset=\"https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Bildschirmfoto-2024-11-03-um-12.11.59.png 690w, https:\/\/www.rocworks.at\/wordpress\/wp-content\/uploads\/2024\/11\/Bildschirmfoto-2024-11-03-um-12.11.59-300x270.png 300w\" sizes=\"auto, (max-width: 690px) 100vw, 690px\" \/><\/a><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">Final Thoughts<\/h3>\n\n\n\n<p>This setup is the result of years of experimentation and adaptation to new tools. In theory, I could simplify parts of it, for example, by replacing more components with the Automation-Gateway. But I appreciate having Kafka as the buffer between data sources and databases &#8211; it offers flexibility for maintenance and helps preserve historical data.<\/p>\n\n\n\n<p>Feel free to test the public <strong>MonsterMQ broker<\/strong> at <a href=\"http:\/\/test.monstermq.com\/\">test.monstermq.com<\/a>. And if you\u2019re curious, publish a JSON object with a &#8220;value&#8221; key to grafana\/something to see it immediately reflected in the Grafana dashboard!<\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>My home automation setup has been an ongoing project, evolving over the last 10\u201315 years. It started with a simple goal: to track data from my photovoltaic (PV) system, power meters, and temperature sensors, all connected through Raspberry Pi devices. It started with Oracle and over time, it\u2019s grown into a more complex architecture that &hellip; <a href=\"https:\/\/www.rocworks.at\/wordpress\/?p=1453\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Exploring My Home Automation Architecture<\/span> <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[39,67],"tags":[],"class_list":["post-1453","post","type-post","status-publish","format-standard","hentry","category-frankenstein","category-mqtt"],"_links":{"self":[{"href":"https:\/\/www.rocworks.at\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/1453","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rocworks.at\/wordpress\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rocworks.at\/wordpress\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rocworks.at\/wordpress\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rocworks.at\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1453"}],"version-history":[{"count":4,"href":"https:\/\/www.rocworks.at\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/1453\/revisions"}],"predecessor-version":[{"id":1462,"href":"https:\/\/www.rocworks.at\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/1453\/revisions\/1462"}],"wp:attachment":[{"href":"https:\/\/www.rocworks.at\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1453"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rocworks.at\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1453"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rocworks.at\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1453"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}