Branch	Github Actions Linux/Mac/Windows x86/x64/arm builds
develop
master

• Quickstart
• Introduction
• Core features
• Building and Installing
• Basic components
  • Ports
  • Connectors
    • Connections
    • Transforms
  • Plugins
  • Extensibility
• Internal data structure
• Configuration files and syntax
  • Main configuration
  • Keys
  • Port configuration
  • Keys
  • Connector configuration
  • Keys
  • Config file keys
  • Connection configuration
  • Keys
  • Transform configuration
  • Keys
    • IndexOffset "Parameters"
    • Threshold "Parameters"
    • Rand "Parameters"
• Extensions
  • DNP3 Port Library
  • Features
  • Configuration
    • DNP3 Master
    • Example
    • Config file keys
    • DNP3 Outstation
    • Example
    • Config file keys
    • Common keys to DNP3 Master and Outstation
  • JSON Port Library
  • Features
  • Configuration
    • JSON Client
    • Example
    • Config file keys
  • Elasticsearch
• API
  • Port
  • Transform
  • User interface

• Download an installer from the Releases page.

  • Look for the following in the package name to choose the right package - (OS)-(Arch)-(OS_ver):
    • OS: Operating System - either Linux, Darwin (for OSX), or Windows
    • Arch: Architecture - processor family, bits and/or extensions - x86/i386, x64/x86_64/AMD64, arm(hf|el) etc.
    • OS_ver: target platform details - this doesn't have to match your system exactly because most dependencies are bundled.
      • el7: for Linux systems with older system libraries (glibc), like RHEL/CentOS/OEL 7
      • rpi: Raspbian linux
      • version numbers for OSX and Windows system/SDK.

• Windows builds require the the x86 MSVC redistributable or x64 MSVC redistributable

• Run the installer, accept the license conditions, choose where to install

• Optionally install as a windows service (--help on the commandline and look for -i)

• Optionally install the included init.d script (in the init dir where you installed) using your linux distro's tools

opendatacon is designed to be a flexible, highly configurable, modular application. It isn't intended to solve just a singular problem, but rather, provide a tool that can be configured and extended to solve a multitude of problems. This manual will explain core features, inbuilt features and packaged extensions.

The minimal requirement of a data concentrator is to connect multiple data channels to single destination. opendatacon meets this requirement, but in a more generic way. It extends the definition to not only data 'concentration' but arbitrary data routing.

• Combine or pass through any number (within operating system limits) of data interfaces (channels) to any number of other data interfaces. These interfaces are called ports in the context of opendatacon.
• Ports translate from/to external protocols.
• Data can be manipulated en-route. These manipulations are called transforms in the context of opendatacon.

These core features are abstract by design, to allow for extension. Some extensions are included in opendatacon, the features of which will be covered in corresponding sections of this manual.

The only real constraint to whether a particular data stream can be handled by opendatacon, is whether the data can be meaningfully 'wrapped' or translated to the internal data structure used by opendatacon. This is discussed further in the internal data structure section. Suffice to say that most SCADA data, irrespective of protocol, should not pose a problem due to the common nature of SCADA applications, and even arbitray binary data can be wrapped in the more general data types.

opendatacon uses the CMake build system.

• Modern C++17 compiler (currently developed under msvc, g++ and clang)

Core dependencies are set up as git submodules and the default cmake config will auto download and use:

• ASIO: http://think-async.com/
• TCLAP: https://github.com/eile/tclap
• spdlog: https://github.com/gabime/spdlog

Plugin specific dependencies:

• DNP3Port: opendnp3 fork : https://github.com/neilstephens/opendnp3
  • default cmake config will auto download and build
• WebUI: https://gitlab.com/eidheim/Simple-Web-Server.git
  • default cmake config will auto download and build
• ModbusPort: libmodbus http://libmodbus.org/
  • available in most linux repos (try install libmodbus-dev or libmodbus-devel)
  • download and build yourself for windows - or search the net for binaries
• LuaPort, LuaTransform, LuaLogSink, LuaUICommander
  • Lua! did you guess? http://www.lua.org

• TESTS - Build tests
• WEBUI - Build the http(s) web user interface
• DNP3PORT - Build DNP3 Port
• JSONPORT - Build JSON Port
• PYPORT - Build Python Port
• MODBUSPORT - Build Modbus Port
• SIMPORT - Build Simulation Port
• MD3PORT - Build MD3 Port
• CBPORT - Build Conitel-Baker Port
• FILETRANSFERPORT - Build File Transfer Port
• LUAPORT - Build Lua Port
• LUATRANSFORM - Build Lua Transform
• LUALOGSINK - Build Lua log sink plugin
• LUAUICOMMANDER - Build Lua UI command script plugin
• CONSOLEUI - Build the console user interface

• STATIC_LIBSTDC++ - link in libstdc++ statically
• PACKAGE_LIBSTDC++ - optionally include libstdc++ shared library in installation package
• PACKAGE_LIBMODBUS - include in the installation package

If you don't want cmake to download/build for you, or you have manually installed in weird locations:

• DNP3_HOME
• ASIO_HOME
• TCLAP_HOME
• MODBUS_HOME

Cmake cross compiler toolchain files are supported. See the examples in the root of the source tree.

The following examples all create out-of-source build systems which is recommended.

cmake . "-Bbuild-xcode" -G "Xcode" "-DFULL=ON"

The following demonstrates some custom dependency paths

cmake -B D:\odc\build -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=D:\odc/build/install -G"Visual Studio 17 2022" -Ax64 -DCMAKE_CONFIGURATION_TYPES=Release -DMODBUS_HOME=c:\libmodbus\windows64 -DPYTHON_HOME=C:\Python\3.7.9\x64  -DFULL=ON

1. Install build system and dependencies:

sudo apt-get install libmodbus-dev libssl-dev

2. Create build system for desired environment

cmake . "-Bbuild-make" -G "Unix Makefiles" "-DFULL=ON"

3. Build

cd build-make
make

4. Install

sudo make install

5. Optionally generate an installation package

make package

Here are some examples utilising the basic components for configuring opendatacon, to show how they fit together.

Port 1 and port 2 are subscribed to each other by a single connection.

Port 1 and port 2 are subscribed to each other. Port 1 and port 3 are also subscribed. Each has its own connection.

Port 1 and port 2 are subscribed to each other by a single connection. There is a unidirectional transform affecting the data sent from port 1 to port 2.

Port 1 and port 2 are subscribed to each other. Port 1 and port 3 are also subscribed. There is a unidirectional transform affecting the data sent from port 1 to port 2 and port 3. There is also a unidirectional transform affecting the data sent from port 2 to port 1.

Ports are the interface between opendatacon and the outside world. It is a Port's job to translate from/to the internal data structures used by opendatacon to/from an external protocol. For example, as of opendatacon 1.9.0, there are built in Port types:

• Null port
• DNP3 Master and Outstation ports
• Modbus Master and Outstation ports
• Simulation Port
• JSON Client and Server ports
• Conitel Master and Outstation ports
• MD3 Master and Outstation ports
• File transfer port
• Python port (deprecated in favour of Lua port below)
• Lua port
  • Lua port allows custom run-time port implementations
  • See the ExampleConfig directory to see how you can write your own port in Lua

Connectors are simply a 'container' structure for logically delineating groupings of connections and transforms. They are used to add greater flexibility and permutations to how things can be configured. For example, a largely similar pattern of connections/transforms can be easily duplicated using a common connector configuration and only overriding key attributes. More on that in the details of the configuration syntax...

Connections simply open a bi-directional data path between two ports, by subscribing to two ports and routing data between them.

Transforms filter and transform data on entry to a Connector (ie. they're uni-directional), for a single source of data (sending port). They can be thought of as a subscription filter. For example, a transform could be implemented to add timestamps to untimestamped data, or deadband or throttle a data stream.

Custom Transforms can be implemented in Lua (see example configs in the repo)

Plugins don't form a component of the data/event processing path. They're separate modules that use the user interface API to control the other components.

The basic components, described above, form the basis of an extensible platform. Generally, any combination of ports can be connected together because they share the same abstract API. This also includes custom ports and transforms, so it's quite easy to create an adaptor for any of the supported protocols for your specific use case.

The internal data structures of opendatacon were adopted from the data structures in the dnp3 protocol, because this was one of the first port implementations. But it also happens to be a very comprehensive protocol, which covers a broad range of data types

• Data is encapsulated by events
• Events have four parts:
  • Measurement/Payload
  • Timestamp
  • Index/Address
  • Quality flags
• Ports can send and recieve events, and use the index and type as an address to map data from/to multiple points.

The included components of opendatacon use the data format JSON (JavaScript Object Notation) for configuration. opendatacon doesn't utilise Java or JavaScript in any other capacity, but JSON was chosen for configuration because this usage matches the intention of the format:

"JSON (JavaScript Object Notation) is a lightweight data-interchange format. It is easy for humans to read and write. It is easy for machines to parse and generate." - json.org

In general, the configuration of a component (port, connector, transform, connection) is contained in a JSON object. A JSON object is just a group of key value pairs, where values can be nested objects (or arrays - arrays are just lists of values). These component objects are defined within an overall JSON object. In short - a simple hierarchical configuration, explained below.

For details on JSON syntax, see the handy diagrams at json.org. When a value, string, array, object or number is referred to in the following configuration sections, it is as defined in JSON.

Starting with an example configuration:

{
    "LogName" :   "ODC_Log",
    "LogFileSizekB"   : 50000,
    "NumLogFiles":    1,
    "LogLevel":  "error",
     
    "Ports" :
    [
        {
            "Name" : "InPort_23",
            "Type" : "DNP3Master",
            "Library" : "DNP3Port",
            "ConfFilename" : "DMCDNP3_Master.conf",
            "ConfOverrides" : { "IP" : "192.168.5.6", "Port" : 20000, "MasterAddr" : 0, "OutstationAddr" : 1}
        },
        {
            "Name" : "OutPort_23",
            "Type" : "DNP3Outstation",
            "Library" : "DNP3Port",
            "ConfFilename" : "DMCDNP3_Outstation.conf",
            "ConfOverrides" : { "IP" : "0.0.0.0", "Port" : 20000, "MasterAddr" : 0, "OutstationAddr" : 23}
        }
    ],
    "Connectors" :
    [
        {
            "Name" : "DMC_Connector",
            "ConfFilename" : "DMCConnector.conf"
        }
    ]
}

The main config consists of global settings, a list of ports, a list of connectors and a list of any plugins. The global attributes are discussed in this section, and the hierarchical configuration of ports, connectors and plug-ins are discussed in following sections.

Here are the available keys for configuration of the main opendatacon configuration object:

Here are the available keys for configuration of a port in opendatacon. An object with these keys is used as a member of the "Ports" list in the main configuration.

Here are the available keys for configuration of a connector in opendatacon. An object with these keys is used as a member of the "Connectors" list in the main configuration.

Here is an example of the object in the file referred to by "ConfFilename":

{
	"Connections" :
	[
		{
			"Name" : "Test JSON to DNP3",
			"Port1" : "Test JSON input",
			"Port2" : "Test DNP3 output 1"
		},
		{
			"Name" : "Test DNP3 to Null",
			"Port1" : "Test DNP3 Master",
			"Port2" : "Test DNP3 output 2"
		}
	],


	"Transforms" :
	[
		{
			"Type" : "IndexOffset",
			"Sender": "Test JSON input",
			"Parameters" : {
				"Offset" : 1
			}
		}
	]
}

Here are the available keys for configuration of a connection in opendatacon. An object with these keys is used as a member of the "Connections" list in the connector config file.

Here are the available keys for configuration of a transform in opendatacon. An object with these keys is used as a member of the "Transforms" list in the connector config file.

The IndexOffset transform expects a JSON object with the following keys to be provided as the "Parameters" value:

The IndexMap transform enables point indexes to be remapped. It expects a JSON object with the following keys to be provided as the "Parameters" value:

The below example maps analog point 100-105 to 0-5 (e.g. index 100 becomes index 0)

{
    "Type" : "IndexMap",
    "Sender": "DNP3Master",
    "Parameters" :
    {
        "AnalogMap" :
        {
            "From":
            [100,101,102,103,104,105],
            "To":
            [0,1,2,3,4,5]
        }
    }
}

There are no expected parameters for the Rand transform. Any given will be ignored.

If the inbuilt types, and bundled extensions don't meet your needs, then you opt to use a custom Port, Transform, Log sink, or UI script to do what you want. Each of these base components has an implementation to let you load a Lua script to do what you want. See the examples:

ExampleConfig/SelfDocumentingLuaPort/

ExampleConfig/LuaTransform_n_CommandScript/

A DNP3 port is configured by setting the "Type" of a port to either "DNP3Master" or "DNP3Outstation", the "Library" to "DNP3Port", and the "ConfFilename" to a file containing the JSON object discussed below.

{
    //------- DNP3 Link Configuration--------#
    "IP" : "1.2.3.4",
    "Port" : 20000,
    "MasterAddr" : 0,
    "OutstationAddr" : 1,
    "LinkNumRetry" : 0,
    "LinkTimeoutms" : 1000,
    "LinkUseConfirms" : false,
     
    //-------- DNP3 Common Application Configuration -------------#
    "EnableUnsol": true,
    "UnsolClass1": true,
    "UnsolClass2": true,
    "UnsolClass3": true,
    //-------Master Stack conf--------#
    "MasterResponseTimeoutms" : 30000,
    "MasterRespondTimeSync" : true,
    "DoUnsolOnStartup" : true,
    "StartupIntegrityClass0" : true,
    "StartupIntegrityClass1" : true,
    "StartupIntegrityClass2" : true,
    "StartupIntegrityClass3" : true,
    "IntegrityOnEventOverflowIIN" : true,
    "TaskRetryPeriodms" : 30000,
    "IntegrityScanRatems" : 3600000,
    "EventClass1ScanRatems" : 0,
    "EventClass2ScanRatems" : 0,
    "EventClass3ScanRatems" : 0,
    "OverrideControlCode" : "PULSE",
    "DoAssignClassOnStartup" : false,
     
    //-------Point conf--------#
    "Binaries" : [{"Index": 5}],
    "Analogs" : [{"Range" : {"Start" : 0, "Stop" : 5}}],
    "CommsPoint" : {"Index" : 0, "FailValue" : true}
    //"BinaryControls" : [{"Range" : {"Start" : 0, "Stop" : 5}}],  
}

{
    //------- DNP3 Link Configuration--------#
    "IP" : 0.0.0.0,
    "Port" : 20000,
    "MasterAddr" : 0,
    "OutstationAddr" : 1,
    "LinkNumRetry" : 0,
    "LinkTimeoutms" : 1000,
    "LinkUseConfirms" : false,
     
    //-------- DNP3 Common Application Configuration -------------#
    "EnableUnsol": true,
    "UnsolClass1": false,
    "UnsolClass2": false,
    "UnsolClass3": false,
     
    //-------Outstation Stack conf--------#
    "SelectTimeoutms" : 30000,
    "SolConfirmTimeoutms" : 30000,
    "UnsolConfirmTimeoutms" : 30000,
    "EventBinaryResponse": "Group2Var2",
    "EventAnalogResponse": "Group32Var5",
    "EventCounterResponse": "Group22Var1",
    "StaticBinaryResponse": "Group1Var2",
    "StaticAnalogResponse": "Group30Var5",
    "StaticCounterResponse": "Group20Var1",

    "WaitForCommandResponses": false,
    //-------Point conf--------#
    "Binaries" : [{"Index": 0},{"Index": 5}],
    "Analogs" : [{"Range" : {"Start" : 0, "Stop" : 5}}]
//"BinaryControls" : [{"Range" : {"Start" : 0, "Stop" : 5}}]
     
}

Modbus port placeholder

{
    //-------Point conf--------#
    "Binaries" : 
    [
        {"Index": 0},{"Index": 1},{"Index": 5},{"Index": 6}
    ],

    "Analogs" : 
    [
        {"Range" : {"Start" : 0, "Stop" : 2}, "StartVal" : 50, "UpdateIntervalms" : 0, "StdDev" : 2},
        {"Range" : {"Start" : 3, "Stop" : 5}, "StartVal" : 230, "UpdateIntervalms" : 0, "StdDev" : 5}
    ],

    "BinaryControls" : 
    [
        {"Index" : 0, "FeedbackBinaries" : [{"Index" : 0, "FeedbackMode" : "PULSE"}]}
    ]
}

A JSON port is configured by setting the "Type" of a port to "JSONClient" ("JSONServer" yet to be implemented), the "Library" to "JSONPort", and the "ConfFilename" to a file containing the JSON object discussed below.

{
    "JSONPointConf" :
    [
    {
        // The type of points - Analog or Binary
        "PointType" : "Analog",
        "Points" :
        [
            {
              "Index" : 0,
              "StartVal" : 3.46,
              "JSONPath" : ["Weather Data","Wind Speed"]
            },
            {
              "JSONPath" : ["Weather Data","Humidity"],
              "Index" : 1
            }
        ]
    },
 
    {
        "PointType" : "Binary",
        "Points" :
        [
            {
              "Index" : 0,
              "JSONPath" : ["Device","Status"],
              "TrueVal" : "GOOD",
              "StartVal" : "GOOD"
            },
            {
              "JSONPath" : ["Device","Online"],
              "Index" : 1,
              "TrueVal" : "YES",
              "FalseVal" : "NO",
              "StartVal" : "SOMETHING ELSE" //true and false are both defined - another state will produce bad quality
            }
        ]
    }
    ]
}

It is possible to historise data from opendatacon by simply pointing a JSON port at Logstash and Elastic search.

A basic configuration of the logstash.conf

input {
  tcp {
    port => 2598
    codec => json
    mode => client
    host => "127.0.0.1"
    type => "opendatacon"
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "opendatacon-"
    document_type => "%{[@metadata][type]}"
  }
}

The null port is equivalent of /dev/null as a DataPort and can be used for testing purposes. There is no configuration data required.

The null port is simply created as follows:

{
	"Name" : "Null",
	"Type" : "Null",
	"ConfFilename" : "/dev/null"
}

Custom implementations of ports are supported by allowing dynamic libraries to be loaded at runtime. See the port configuration section of this document for how to configure the loading.

To be successfully loaded, a library only needs to export a single C style construction routine for each type of port it implements, of the form:

extern "C" DataPort* new_PORTNAMEPort(const std::string& aName, const std::string& aConfFilename, const Json::Value& aConfOverrides);
extern "C" void delete_PORTNAMEPort(DataPort*);

PORTNAME in the example above is the name used as "Type" in the port configuration. It must return a pointer to a new heap allocated instance of a descendant of C++ class DataPort. The declaration of DataPort is included here for context. The whole class hierarchy for DataPort from the public header files (include directory in the opendatacon repository) is needed in practice.

See include/opendatacon/DataPort.h 

If you're not keen to write and build a DataPort in C++, check out LuaPort, which allows you to load a Lua script as a port implementation.

Similar to Ports, custom Transforms can be implemented as dynamically loaded modules with specified entry points.

extern "C" Transform* new_SOMETransform(const Json::Value& params);
extern "C" void delete_SOMETransform(Transform* pSometx);

The 'newing' function just has to construct a instance of a class derriving from the Transform base class (see below) on the heap, and return a pointer to the object.

The 'deleting' function just has to destroy the specified object.

See include/opendatacon/Transform.h 

See include/opendatacon/IUI.h