My first tests with QuestDB on 10 years of home automation data (1.4 billion rows) are promising.
👉 Fast ingestion of parquet files (~1 hour on an old Intel NUC i5)
I have stored my data in parquet files, one per month, and imported it with a simple Python script. The import on my really old Intel NUC i5 took only about one hour. I’ve never been able to do this so quickly with any other database.
🤔 Btw.: I think storing data in #parquet files, or any other open table format, like Apache Iceberg, is one of the best choices to keep data. Because it’s independent of a database engine.
👉 Familiar SQL syntax. I experienced that QuestDB has a powerful SQL engine. I converted some Postgres SQL statements to QuestDB without big issues or changes. And I love SQL 💚
👉 Great query response times – see image. Not a representative query, but still impressive speed.
👉 By using ZFS with compression the used disk space can be reduced to a good value.
Do you want to log your OPCUA data to QuestDB? I have added this option to the automation-gateway.com seven days ago.
For the OpenSearch and Elasticsearch fans: the automation-gateway.com can now log OPC UA, MQTT, PLC4X values to OpenSearch!
Makes sense if you have lot of text values, to leverage the full text search capabilities. But for timeseries it is nice as well, the Kibana tool is great to create dashboards.
In this tutorial, I will guide you through the essential steps to set up the Automation Gateway, harness the power of YAML extensions in Visual Studio Code for configuration, and connect various devices, including OPC UA, MQTT, and PLC4X devices. I will show how to integrate the values from the devices to the Gateway’s OPC UA server and how to use the MQTT interface to get the values from the devices via a MQTT client. Additionally values from the connected devices will be logged to a Influx database.
I wanted to get my Home-Automation values to SCADA, it’s a “self-made” JSON message format. I tried it with Ignition and the MQTT Module. Btw.: it’s great that they have the Makers Edition for non-commercial use at home 👍. But I don’t know why, it only got one topic and one value from my MQTT Broker, and it did not receive any updates. Don’t know what went wrong…
Anyhow, I decided to add a custom JSON format to the Automation-Gateway.com. It’s simple, just define the JSON-Path to the value and optionally to a timestamp in milliseconds since epoch or to an ISO 8601 format.
Now I can use the Automation-Gateway’s OPC UA server in any SCADA system to visualize my MQTT values…
Here is the config.yaml configuration file for the Automation-Gateway.
🤔 I thought about if it could make sense to connect to a MQTT broker and write the topic-path in a structured and connected way to a graph database. It could reflect a companies UNS with ISA structure in a graph with query possibilities.
🤝 The UNS could be enriched by adding additional meta information to the database and be linked to the MQTT nodes. Graph queries could be used to combine the enriched data with the current values of the machines…
👉 I reactivated the Neo4J Logger in the Automation-Gateway.com, and enabled MQTT for it. It creates the nodes in a neo4j graph database based on the incoming MQTT messages.
Did a test with one gateway and collected 250000 😲 value changes per second(!) from 10 OPC UA servers and wrote it to 10 topics on a single Kafka Broker… Kafka was bored 😴… one and a half core of an i7 PC. The gateway had a lot of work to do, collect and bulk all the single values and then send it to Kafka … roughly 10 cores of an i7 PC.
The gateway has a GraphQL and MQTT interface where you can read and write OPC UA values. On an internal MQTT topic the logger publishes how many values are getting in and out…
In a simple setup I have tested to send 2000 value changes per second from an OPC UA server to Unity with GraphQL, the Open-Source Frankenstein Automation Gateway, and the GraphQL for Unity Asset. And it could go up to 10000 value changes per second…
I had one DotNet OPC UA server with a lot of simulated tags with random data. The .Net OPC UA server is the DotNet reference implementation from the OPCFoundation, which can be found here.
On top of that the Open-Source Frankenstein Automation Gateway for GraphQL was running. It is connected to the OPC UA server. It offers a GraphQL interface to the tags of one or more connected OPC UA servers.
In Unity I had used the GraphQL for OPC UA Asset to easily connect to the Gateway, browse the tags, and subscribe to the value changes of 100 tags.
Each tag was changed by the OPC UA server every 45ms. This ends up in a bit more than 2000 value changes per seconds, which were sent from the OPC UA server to the Unity Application.
Here we see the Application running 3 times on my Laptop with an Intel(R) Core(TM) i7-9850H CPU @ 2.60GHz. In the center you see the amount of incoming value changes per second (red number). Around that number, we see some of the values coming in from OPC UA. At the bottom we see the CPU load of the Frankenstein Automation Gateway (java program) and the OPC UA DotNet Server. Both were running on an old Intel(R) Core(TM) i3-6100U CPU @ 2.30GHz.
It was also possible to increase to load up to 10000 value changes per seconds! Sending 100 values every 10ms from OPC UA to Unity…