I have packed the Automation-Gateway.com into an Edge App and successfully deployed it on a SIMATIC HMI Unified Comfort Panel.
👉 The app is currently publishing tags to a MQTT broker and logging tag data into QuestDB.
In a quick test with a vPLC and about 4000 Tags it can easily write ~6400 value changes per second to QuestDB. Panel CPU Load ~22%.
My first tests with QuestDB on 10 years of home automation data (1.4 billion rows) are promising.
👉 Fast ingestion of parquet files (~1 hour on an old Intel NUC i5)
I have stored my data in parquet files, one per month, and imported it with a simple Python script. The import on my really old Intel NUC i5 took only about one hour. I’ve never been able to do this so quickly with any other database.
🤔 Btw.: I think storing data in #parquet files, or any other open table format, like Apache Iceberg, is one of the best choices to keep data. Because it’s independent of a database engine.
👉 Familiar SQL syntax. I experienced that QuestDB has a powerful SQL engine. I converted some Postgres SQL statements to QuestDB without big issues or changes. And I love SQL 💚
👉 Great query response times – see image. Not a representative query, but still impressive speed.
👉 By using ZFS with compression the used disk space can be reduced to a good value.
Do you want to log your OPCUA data to QuestDB? I have added this option to the automation-gateway.com seven days ago.
For the OpenSearch and Elasticsearch fans: the automation-gateway.com can now log OPC UA, MQTT, PLC4X values to OpenSearch!
Makes sense if you have lot of text values, to leverage the full text search capabilities. But for timeseries it is nice as well, the Kibana tool is great to create dashboards.
With just 20 lines of configuration you can publish a OPCUA tree of values to MQTT …
in this example to the HiveMQ cloud … 👉 with the automation-gateway.com
It also supports PLC4x connected devices/plcs…
It can also publish values to Kafka or SQL databases…
git clone https://github.com/vogler75/automation-gateway.git
cd source\app
set GATEWAY_CONFIG=configs/config-opcua-mqtt.yaml
gradle run
Drivers:
OpcUa:
- Id: "demo"
Enabled: true
LogLevel: INFO
EndpointUrl: "opc.tcp://192.168.1.3:62541"
UpdateEndpointUrl: true
SecurityPolicyUri: http://opcfoundation.org/UA/SecurityPolicy#None
Loggers:
Mqtt:
- Id: mqtt1
Enabled: true
Host: linux0.rocworks.local
Port: 1883
Ssl: false
Topic: Enterprise/Site
Logging:
- Topic: opc/demo/path/Objects/Demo/SimulationMass/#With just some lines of configuration you can bring PLC values to OPC UA, MQTT and GraphQL. And to a variety of databases for tag logging…
💡 In that example with ModBus, but thanks to #plc4x this should work in the same way also for the other protocols supported by PLC4X.
📺 See the video, ModBus values are brought to OPC UA and MQTT.
💣 In MQTT the topic name is enriched with a UNS ISA95 topic path.
⚡ On MQTT SparkplugB encoded messages could be used.
😎 100% GUI free and Open Source.
Servers:
GraphQL:
- Port: 4000
LogLevel: INFO
GraphiQL: true
OpcUa:
- Port: 4841
Enabled: true
LogLevel: INFO
Topics:
- Topic: plc/demo/node/holding-register:1:INT
- Topic: plc/demo/node/holding-register:2:INT
- Topic: plc/demo/node/holding-register:3:INT
Drivers:
Plc4x:
- Id: "demo"
Enabled: true
Url: "modbus://localhost:502"
Polling:
Time: 100
OldNew: true
WriteTimeout: 100
ReadTimeout: 100
LogLevel: INFO
Loggers:
Mqtt:
- Id: mqtt1
Enabled: true
Host: 192.168.1.4
Port: 1883
Topic: modbus
Format: Raw
Logging:
- Topic: plc/demo/node/holding-register:1:INT
Target: enterprise/area1/line1/cell1/speed
- Topic: plc/demo/node/holding-register:2:INT
Target: enterprise/area1/line1/cell1/power
- Topic: plc/demo/node/holding-register:3:INT
Target: enterprise/area1/line1/cell1/torque
In this tutorial, I will guide you through the essential steps to set up the Automation Gateway, harness the power of YAML extensions in Visual Studio Code for configuration, and connect various devices, including OPC UA, MQTT, and PLC4X devices. I will show how to integrate the values from the devices to the Gateway’s OPC UA server and how to use the MQTT interface to get the values from the devices via a MQTT client. Additionally values from the connected devices will be logged to a Influx database.
I wanted to get my Home-Automation values to SCADA, it’s a “self-made” JSON message format. I tried it with Ignition and the MQTT Module. Btw.: it’s great that they have the Makers Edition for non-commercial use at home 👍. But I don’t know why, it only got one topic and one value from my MQTT Broker, and it did not receive any updates. Don’t know what went wrong…
Anyhow, I decided to add a custom JSON format to the Automation-Gateway.com. It’s simple, just define the JSON-Path to the value and optionally to a timestamp in milliseconds since epoch or to an ISO 8601 format.
Now I can use the Automation-Gateway’s OPC UA server in any SCADA system to visualize my MQTT values…
Here is the config.yaml configuration file for the Automation-Gateway.
Servers:
OpcUa:
- Port: 4841
Enabled: true
LogLevel: INFO
Topics:
- Topic: mqtt/home/path/Original/#
Drivers:
Mqtt:
- Id: "home"
LogLevel: INFO
Host: 192.168.1.3
Port: 1883
Format: Json
CustomJson:
Value: "Value"
TimestampMs: "TimeMS"
🤔 I thought about if it could make sense to connect to a MQTT broker and write the topic-path in a structured and connected way to a graph database. It could reflect a companies UNS with ISA structure in a graph with query possibilities.
🤝 The UNS could be enriched by adding additional meta information to the database and be linked to the MQTT nodes. Graph queries could be used to combine the enriched data with the current values of the machines…
👉 I reactivated the Neo4J Logger in the Automation-Gateway.com, and enabled MQTT for it. It creates the nodes in a neo4j graph database based on the incoming MQTT messages.
Also interesting to explore the graph of the test.mosquitto.org hashtag#MQTT broker…
saw some go-eChargers, some solax inverters, some json scada demos (never heard about it before), and also a “Atomkraftwerk1234567890” 😀
Did a test with one gateway and collected 250000 😲 value changes per second(!) from 10 OPC UA servers and wrote it to 10 topics on a single Kafka Broker… Kafka was bored 😴… one and a half core of an i7 PC. The gateway had a lot of work to do, collect and bulk all the single values and then send it to Kafka … roughly 10 cores of an i7 PC.
The gateway has a GraphQL and MQTT interface where you can read and write OPC UA values. On an internal MQTT topic the logger publishes how many values are getting in and out…
In a simple setup I have tested to send 2000 value changes per second from an OPC UA server to Unity with GraphQL, the Open-Source Frankenstein Automation Gateway, and the GraphQL for Unity Asset. And it could go up to 10000 value changes per second…
I had one DotNet OPC UA server with a lot of simulated tags with random data. The .Net OPC UA server is the DotNet reference implementation from the OPCFoundation, which can be found here.
On top of that the Open-Source Frankenstein Automation Gateway for GraphQL was running. It is connected to the OPC UA server. It offers a GraphQL interface to the tags of one or more connected OPC UA servers.
In Unity I had used the GraphQL for OPC UA Asset to easily connect to the Gateway, browse the tags, and subscribe to the value changes of 100 tags.
Each tag was changed by the OPC UA server every 45ms. This ends up in a bit more than 2000 value changes per seconds, which were sent from the OPC UA server to the Unity Application.
Here we see the Application running 3 times on my Laptop with an Intel(R) Core(TM) i7-9850H CPU @ 2.60GHz. In the center you see the amount of incoming value changes per second (red number). Around that number, we see some of the values coming in from OPC UA. At the bottom we see the CPU load of the Frankenstein Automation Gateway (java program) and the OPC UA DotNet Server. Both were running on an old Intel(R) Core(TM) i3-6100U CPU @ 2.30GHz.
It was also possible to increase to load up to 10000 value changes per seconds! Sending 100 values every 10ms from OPC UA to Unity…
