Basic concepts

Before starting, let’s quickly introduce the main concepts used in netjsonconfig:

NetJSON configuration dictionary: python dictionary representing the configuration of a router
Backend: python class used to convert the configuration dictionary to the format used natively by a router firmware and vice versa
Schema: each backend has a JSON-Schema which defines the useful configuration options that the backend is able to process
Validation: the configuration is validated against its JSON-Schema before being processed by the backend
Template: common configuration options shared among routers (eg: VPNs, SSID) which can be passed to backends
Multiple template inheritance: possibility to inherit common configuration options from more than one template
Context (configuration variables): variables that can be referenced from the configuration dictionary

NetJSON configuration dictionary

netjsonconfig is an implementation of the NetJSON format, more specifically the DeviceConfiguration object, therefore to understand the configuration format that the library uses to generate the final router configurations it is essential to read at least the relevant DeviceConfiguration section in the NetJSON RFC.

Here it is a simple NetJSON DeviceConfiguration object represented with a python dictionary:

{
    "type": "DeviceConfiguration",
    "general": {"hostname": "RouterA"},
    "interfaces": [
        {
            "name": "eth0",
            "type": "ethernet",
            "addresses": [
                {
                    "address": "192.168.1.1",
                    "mask": 24,
                    "proto": "static",
                    "family": "ipv4",
                }
            ],
        }
    ],
}

The previous example describes a device named RouterA which has a single network interface named eth0 with a statically assigned ip address 192.168.1.1/24 (CIDR notation).

Because netjsonconfig deals only with DeviceConfiguration objects, the type attribute can be omitted.

The previous configuration object therefore can be shortened to:

{
    "general": {"hostname": "RouterA"},
    "interfaces": [
        {
            "name": "eth0",
            "type": "ethernet",
            "addresses": [
                {
                    "address": "192.168.1.1",
                    "mask": 24,
                    "proto": "static",
                    "family": "ipv4",
                }
            ],
        }
    ],
}

From now on we will use the term configuration dictionary to refer to NetJSON DeviceConfiguration objects.

Backend

A backend is a python class used to convert the configuration dictionary to the format used natively by the router (forward conversion, from NetJSON to native) and vice versa (backward conversion, from native to NetJSON), each supported firmware or opearting system will have its own backend and third parties can write their own custom backends.

The current implemented backends are:

OpenWrt

OpenWisp (based on the OpenWrt backend)

OpenVpn (custom backend implementing only OpenVPN configuration)

WireGuard (custom backend implementing only WireGuard configuration)

VXLAN over WireGuard (custom backend implementing only VXLAN over WireGuard configuration)

ZeroTier (custom backend implementing only ZeroTier configuration)

Example initialization of OpenWrt backend:

from netjsonconfig import OpenWrt

ipv6_router = OpenWrt(
    {
        "interfaces": [
            {
                "name": "eth0.1",
                "type": "ethernet",
                "addresses": [
                    {
                        "address": "fd87::1",
                        "mask": 128,
                        "proto": "static",
                        "family": "ipv6",
                    }
                ],
            }
        ]
    }
)

Each backend will implement parsers, renderers and converters to accomplish its configuration generation or parsing goals.

The process is best explained with the following diagram:

Converters take care of converting between NetJSON and the intermediate data structure (and vice versa).

Renderers take care of rendering the intermediate data structure to the native format.

Parsers perform the opposite operation of Renderers: they take care of parsing native format and build the intermediate data structure.

Schema

Each backend has a JSON-Schema, all the backends have a schema which is derived from the same parent schema, defined in netjsonconfig.backends.schema (view source).

Since different backends may support different features each backend may extend its schema by adding custom definitions.

Validation

All the backends have a validate method which is called automatically before trying to process the configuration.

If the passed configuration violates the schema the validate method will raise a ValidationError.

An instance of validation error has two public attributes:

message: a human readable message explaining the error
details: a reference to the instance of jsonschema.exceptions.ValidationError which contains more details about what has gone wrong; for a complete reference see the python-jsonschema documentation

You may call the validate method in your application arbitrarily, eg: before trying to save the configuration dictionary into a database.

Template

If you have devices with very similar configuration dictionaries you can store the shared blocks in one or more reusable templates which will be used as a base to build the final configuration.

Combining different templates

Let’s illustrate this with a practical example, we have two devices:

Router1

Router2

Both devices have an eth0 interface in DHCP mode; Router2 additionally has an eth1 interface with a statically assigned ipv4 address.

The two routers can be represented with the following code:

from netjsonconfig import OpenWrt

router1 = OpenWrt(
    {
        "general": {"hostname": "Router1"},
        "interfaces": [
            {
                "name": "eth0",
                "type": "ethernet",
                "addresses": [{"proto": "dhcp", "family": "ipv4"}],
            }
        ],
    }
)

router2 = OpenWrt(
    {
        "general": {"hostname": "Router2"},
        "interfaces": [
            {
                "name": "eth0",
                "type": "ethernet",
                "addresses": [{"proto": "dhcp", "family": "ipv4"}],
            },
            {
                "name": "eth1",
                "type": "ethernet",
                "addresses": [
                    {
                        "address": "192.168.1.1",
                        "mask": 24,
                        "proto": "static",
                        "family": "ipv4",
                    }
                ],
            },
        ],
    }
)

The two configuration dictionaries share the same settings for the eth0 interface, therefore we can make the eth0 settings our template and refactor the previous code as follows:

from netjsonconfig import OpenWrt

dhcp_template = {
    "interfaces": [
        {
            "name": "eth0",
            "type": "ethernet",
            "addresses": [{"proto": "dhcp", "family": "ipv4"}],
        }
    ]
}

router1 = OpenWrt(
    config={"general": {"hostname": "Router1"}}, templates=[dhcp_template]
)

router2_config = {
    "general": {"hostname": "Router2"},
    "interfaces": [
        {
            "name": "eth1",
            "type": "ethernet",
            "addresses": [
                {
                    "address": "192.168.1.1",
                    "mask": 24,
                    "proto": "static",
                    "family": "ipv4",
                }
            ],
        }
    ],
}
router2 = OpenWrt(router2_config, templates=[dhcp_template])

Overriding a template

In many occasions you may want to define a general template which can be overridden in some specific occasions.

A common use case is to define a general radio template and override its channel on certain access points:

from netjsonconfig import OpenWrt

general_radio_template = {
    "radios": [
        {
            "name": "radio0",
            "phy": "phy0",
            "protocol": "802.11n",
            "driver": "mac80211",
            "channel": 0,  # zero means "auto"
            "channel_width": 20,
            "country": "US",
            "disabled": False,
        }
    ]
}

specific_radio_config = {
    "radios": [
        {
            "name": "radio0",
            "channel": 10,
        }
    ]
}

router1 = OpenWrt(
    config=specific_radio_config, templates=[general_radio_template]
)

print(router1.render())

Will generater the following output:

package wireless

config wifi-device 'radio0'
        option channel '10'
        option country 'US'
        option disabled '0'
        option htmode 'HT20'
        option band '2g'
        option phy 'phy0'
        option type 'mac80211'

Multiple template inheritance

You might have noticed that the templates argument is a list; that’s because it’s possible to pass multiple templates that will be added one on top of the other to build the resulting configuration dictionary, allowing to reduce or even eliminate repetitions.

Note

When using multiple templates, their order is important. Templates that are specified afterwards override the ones that come first.

To understand this, read the section Multiple overrides.

Multiple overrides

Here’s a more complex example involving multiple overrides:

from netjsonconfig import OpenWrt

general_radio_template = {
    "radios": [
        {
            "name": "radio0",
            "phy": "phy0",
            "protocol": "802.11n",
            "driver": "mac80211",
            "channel": 0,  # zero means "auto"
            "channel_width": 20,
            "country": "00",  # world
            "disabled": False,
        }
    ]
}

united_states_radio_template = {
    "radios": [{"name": "radio0", "country": "US"}]
}

specific_radio_config = {
    "radios": [
        {
            "name": "radio0",
            "channel": 10,
        }
    ]
}

router1 = OpenWrt(
    config=specific_radio_config,
    templates=[general_radio_template, united_states_radio_template],
)

print(router1.render())

Will generater the following output:

package wireless

config wifi-device 'radio0'
        option channel '10'
        option country 'US'
        option disabled '0'
        option htmode 'HT20'
        option band '2g'
        option phy 'phy0'
        option type 'mac80211'

Implementation details

The functions used under the hood to merge configurations and templates are netjsonconfig.utils.merge_config and netjsonconfig.utils.merge_list:

netjsonconfig.utils.merge_config(template, config, list_identifiers=None)[source]

Merges config on top of template.

Conflicting keys are handled in the following way:

simple values (eg: str, int, float, ecc) in config will overwrite the ones in template
values of type list in both config and template will be merged using to the merge_list function
values of type dict will be merged recursively

Parameters:

template – template dict
config – config dict
list_identifiers – list or None

Returns:

merged dict

netjsonconfig.utils.merge_list(list1, list2, identifiers=None)[source]

Merges list2 on top of list1.

If both lists contain dictionaries which have keys specified in identifiers which have equal values, those dicts will be merged (dicts in list2 will override dicts in list1). The remaining elements will be summed in order to create a list which contains elements of both lists.

Parameters:

list1 – list from template
list2 – list from config
identifiers – list or None

Returns:

merged list

Context (configuration variables)

Without variables, many bits of configuration cannot be stored in templates, because some parameters are unique to the device, think about things like a UUID or a public ip address.

With this feature it is possible to reference variables in the configuration dictionary, these variables will be evaluated when the configuration is rendered/generated.

Here’s an example from the real world, pay attention to the two variables, {{ UUID }} and {{ KEY }}:

from netjsonconfig import OpenWrt

openwisp_config_template = {
    "openwisp": [
        {
            "config_name": "controller",
            "config_value": "http",
            "url": "http://controller.examplewifiservice.com",
            "interval": "60",
            "verify_ssl": "1",
            "uuid": "{{ UUID }}",
            "key": "{{ KEY }}",
        }
    ]
}

context = {
    "UUID": "9d9032b2-da18-4d47-a414-1f7f605479e6",
    "KEY": "xk7OzA1qN6h1Ggxy8UH5NI8kQnbuLxsE",
}

router1 = OpenWrt(
    config={"general": {"hostname": "Router1"}},
    templates=[openwisp_config_template],
    context=context,
)

print(router1.render())

Result:

package system

config system
        option hostname 'Router1'
        option timezone 'UTC'
        option zonename 'UTC'

package openwisp

config controller 'http'
        option interval '60'
        option key 'xk7OzA1qN6h1Ggxy8UH5NI8kQnbuLxsE'
        option url 'http://controller.examplewifiservice.com'
        option uuid '9d9032b2-da18-4d47-a414-1f7f605479e6'
        option verify_ssl '1'

Warning

When using variables, keep in mind the following rules:

variables must be written in the form of {{ var_name }} or {{var_name}};

variable names can contain only alphanumeric characters and underscores;

unrecognized variables will be ignored;

Project goals

If you are interested in this topic you can read more about the Goals and Motivations of this project.

Support

See OpenWISP Support Channels.

License

This software is licensed under the terms of the GPLv3 license, for more information, please see full LICENSE file.