Welcome to the Hyrrokkin documentation

Introduction - Basic Concepts

Topologies

Hyrrokkin is a library which manages the execution of computational graphs, termed topologies.

Each topology consists of executable components called nodes.

Topologies are executed using an engine. Two engines are provided, currently:

Python Engine

This engine supports nodes written in python.

Javascript Engine

This engine supports nodes written in Javascript. It can be used if deno (https://deno.com/) is installed or within web-browsers (see https://codeberg.org/visual-topology/skadi).

Nodes, Ports and Links

Nodes are associated with a class which implements the node's behaviour, most notably the class will implement a run method which transforms a set of values received through that node's input ports into a set of values sent out via the node's output ports.

A topology will also contain links, which connect the output port of one node to the input port of another.

Packages and Package Configurations

Node types and link types are bundled together into a package containing related functionality.

A package must define a package configuration class, a single instance of which will be created when a topology is loaded.

Configuration instances can provide useful services to and can be accessed by any node or any other configuration instance. They must also define a factory method called create_node which will be called to create instances of nodes which belong to the package.

Persistent storage

As well as persisting the structure of a topology, Hyrrokkin manages the persistent storage of key-value pairs for each node and configuration within a topology.

There are two types of persistent storage:

• properties - key-value pairs where the values are JSON-serialisable
• data - key-value pairs where the values are binary data

Clients

Clients may be attached to specific nodes and configurations and interact with them by exchanging messages. Messages consist of one or more parts, each part may be binary, text or a JSON-serialisable object.

An Example Package

Consider a simple example package, textgraph, consisting of a configuration class and three node classes:

• TextgraphConfiguration

The configuration class is responsible for creating instances of nodes from this package.

• InputTextNode

Stores input text and outputs this value via its output port data_out.

Clients can send a new value to this node to trigger re-execution of downstream parts of the topology.

• WordFrequencyNode

Expects text input values via its input port data_in, computes the frequencies of each word in the text and outputs this information in table form via output port data_out.

• DisplayTableNode

Receives a tabular via its input port data_in, and communicates those to any attached clients.

Package Schema

Each Package, and the Links and Nodes it contains, is specified in a JSON formatted document that represents the schema of that Package.

{
    "id": "textgraph",
    "metadata": {
        "name": "Text Graph",
        "version": "0.0.1",
        "description": "a toy example package for analysing text"
    },
    "node_types": {
        "text_input_node": {
            "metadata": {
                "name": "Text Input",
                "description": "Input a text document to be analysed"
            },
            "output_ports": {
                "data_out": {
                    "link_type": "textgraph:text",
                    "allow_multiple_connections": true
                }
            }
        },
        "word_frequency_node": {
            "metadata": {
                "name": "Word Frequency",
                "description": "Calculate the frequencies of words in the input text"
            },
            "input_ports": {
                "data_in": {
                    "link_type": "textgraph:text",
                    "allow_multiple_connections": false
                }
            },
            "output_ports": {
                "data_out": {
                    "link_type": "textgraph:table",
                    "allow_multiple_connections": true
                }
            }
        },
        "table_display_node": {
            "metadata": {
                "name": "Table Display",
                "description": "Display tabular data containing the results of analysing text"
            },
            "input_ports": {
                "data_in": {
                    "link_type": "textgraph:table",
                    "allow_multiple_connections": false
                }
            }
        }
    },
    "link_types": {
        "text": {
            "metadata": {
                "name": "Text",
                "description": "This type of link carries text values"
            }
        },
        "table": {
            "metadata": {
                "name": "Table",
                "description": "This type of link carries values that are tables (lists of lists)"
            }
        }
    }
}

When refering to a link type, the package id should be used as a prefix, <package-id>:<link-type-id>. In this example, textgraph:text refers to the link type text defined in the textgraph example package. This allows packages to refer to link types defined in other packages when defining nodes.

The package itself may define:

• metadata provides descriptive information, including name and description attributes
• a configuration and the names of any clients that may attach to the configuration

Each node is associated with the following information:

• input_ports and output_ports specify the names and link types of the ports attached to a node
• ports cannot accept multiple connections unless the allow_multiple_connections is set to true
• metadata provides descriptive information, including name and description attributes
• the names of any clients that may attach to the node

Each Link is associated with the following information:

• metadata provides descriptive information, including name and description attributes

Filesystem layout

All files that comprise a packages are stored under a root directory which contains the package schema, named schema.json

Hyrrokkin currently supports packages which implement nodes and configurations using javascript or python, but the interfaces are the same.

pythonjavascript

schema.json
python.json
python/
   text_input_node.py
   word_frequency_node.py
   table_display_node.py
   textgraph_configuration.py

schema.json
javascript.json
python/
   text_input_node.js
   word_frequency_node.js
   table_display_node.js
   textgraph_configuration.js

The file python.json / javascript.json defines how the engine will load the package configuration.

pythonjavascript

{
   "configuration_class": ".python.textgraph_configuration.TextgraphConfiguration"
}

{
  "source_paths": [
    "javascript/textgraph_configuration.js",
    "javascript/text_input_node.js",
    "javascript/word_frequency_node.js",
    "javascript/table_display_node.js"
  ]
}

TextgraphConfiguration

A package configuration is implemented as a class with a constructor accepting a services object.

The configuration is required to implement a method for creating node instances.

pythonjavascript

#   Hyrrokkin - a library for building and running executable graphs
#
#   MIT License - Copyright (C) 2022-2025  Visual Topology Ltd

from hyrrokkin_engine.configuration_interface import ConfigurationInterface
import json

from .text_input_node import TextInputNode
from .word_frequency_node import WordFrequencyNode
from .table_display_node import TableDisplayNode

class TextgraphConfiguration(ConfigurationInterface):

    def __init__(self, services):
        self.services = services

    async def create_node(self, node_type_id, node_services):
        match node_type_id:
            case "text_input_node": return TextInputNode(node_services)
            case "word_frequency_node": return WordFrequencyNode(node_services)
            case "table_display_node": return TableDisplayNode(node_services)
            case _: return None

    async def encode(self, value, link_type):
        if value is not None:
            if link_type == "text":
                return value.encode("utf-8")
            elif link_type == "table":
                return json.dumps(value).encode("utf-8")
        return None

    async def decode(self, encoded_bytes, link_type):
        if encoded_bytes is not None:
            if link_type == "text":
                return encoded_bytes.decode("utf-8")
            elif link_type == "table":
                return json.loads(encoded_bytes.decode("utf-8"))
        return None

//   Hyrrokkin - a library for building and running executable graphs
//
//   MIT License - Copyright (C) 2022-2025  Visual Topology Ltd

var textgraph = textgraph || {};

textgraph.TextgraphConfiguration = class {

    constructor(services) {
        this.services = services;
    }

    async load() {
        this.properties = await this.services.get_properties();
    }

    async create_node(node_type_id, node_services) {
        switch (node_type_id) {
            case "text_input_node": {
                let text_input_node = new textgraph.TextInputNode(node_services);
                await text_input_node.load();
                return text_input_node;
            }
            case "word_frequency_node": return new textgraph.WordFrequencyNode(node_services);
            case "table_display_node": return new textgraph.TableDisplayNode(node_services);
            default: return null;
        }
    }

    async encode(value, link_type) {
        if (value !== null) {
            if (link_type === "text") {
                return (new TextEncoder()).encode(value);
            } else if (link_type === "table") {
                return (new TextEncoder()).encode(JSON.stringify(value));
            }
        }
        return null;
    }

    async decode(self, encoded_bytes, link_type) {
        if (encoded_bytes !== null) {
            if (link_type === "text") {
                return (new TextDecoder()).decode(encoded_bytes);
            } else if (link_type === "table") {
                return JSON.parse((new TextDecoder()).decode(encoded_bytes));
            }
        }
        return null;
    }
}

hyrrokkin_engine.registry.register_configuration_factory("textgraph",(configuration_services) => new textgraph.TextgraphConfiguration(configuration_services));

TextInputNode

When a node is constructed, the constructor is passed a service API object, providing various useful services. This services API is very similar to that passed to a configuration constructor.

Consider the TextInputNode:

pythonjavascript

#   Hyrrokkin - a library for building and running executable graphs
#
#   MIT License - Copyright (C) 2022-2025  Visual Topology Ltd

import asyncio

from hyrrokkin_engine.node_interface import NodeInterface

class TextInputNode(NodeInterface):

    def __init__(self, services):
        self.services = services
        self.clients = set()
        self.text = ""

    async def load(self):
        data = await self.services.get_data("value")
        if data is None:
            self.text = ""
        else:
            self.text = data.decode()

    async def open_client(self, client):
        self.clients.add(client)
        async def handle_message(value):
            if value != self.text:
                self.text = value
                await self.services.set_data("value", self.text.encode())
                for other_client in self.clients:
                    if other_client != client:
                        other_client.send_message(self.text)
                await self.services.request_run()
        client.set_message_handler(handle_message)
        client.send_message(self.text)

    async def close_client(self, client):
        self.clients.remove(client)

    async def run(self, inputs):
        if self.text:
            return {"data_out":self.text}
        else:
            return {}

//   Hyrrokkin - a library for building and running executable graphs
//
//   MIT License - Copyright (C) 2022-2025  Visual Topology Ltd

var textgraph = textgraph || {};

textgraph.TextInputNode = class {

    constructor(services) {
        this.services = services;
        this.clients = new Set();
        this.text = "";
    }

    async load() {
        let data = await this.services.get_data("value");
        if (data === null) {
            this.text = "";
        } else {
            this.text = (new TextDecoder()).decode(data);
        }
    }

    async open_client(client) {
        this.clients.add(client);
        client.set_message_handler(async (...msg) => await this.handle_message(client, ...msg));
        client.send_message(this.text);
    }

    async close_client(client) {
        this.clients.delete(client);
    }

    async handle_message(from_client, value) {
        if (value !== this.text) {
            this.text = value;
            await this.services.set_data("value", (new TextEncoder()).encode(this.text).buffer);
            await this.services.request_run();
            this.clients.forEach((other_client) => {
                if (other_client !== from_client) {
                    other_client.send_message(this.text);
                }
            });
        }
    }

    async run(inputs) {
        if (this.text) {
            return {"data_out": this.text}
        } else {
            return {};
        }
    }
}

This node stores the text to output in a binary data object named value. Nodes use the service APIs get_data and set_data to read and write these data objects.

To communicate with clients, nodes (or configurations) implement open_client and close_client methods. In the example above, the TextInputNode expects messages consisting of a single string value, used to refresh the text stored by the node.

When the node is run, its stored value is output on port data_out

If the value passed by a client is not an integer, the node will issue a warning via the service api set_status. The following set of service APIs related to status updates:

service API	Purpose
set_status(msg,"info")	sets the status as INFORMATIONAL accompanied by message msg
set_status(msg,"warning")	sets the status as WARNING accompanied by message msg
set_status(msg,"error")	sets the status as ERROR accompanied by message msg
set_status("")	clears the status associated with this node

WordFrequencyNode

This node performs processing on an input text value to produce a simple table data structure containing words and word frequencies. Rows are sorted in order of decreasing frequency.

pythonjavascript

#   Hyrrokkin - a library for building and running executable graphs
#
#   MIT License - Copyright (C) 2022-2025  Visual Topology Ltd


import re

from hyrrokkin_engine.node_interface import NodeInterface


class WordFrequencyNode(NodeInterface):

    def __init__(self, services):
        self.services = services
        self.clients = set()
        self.properties = None

    async def load(self):
        self.properties = await self.services.get_properties()
        if "threshold" not in self.properties:
            self.properties["threshold"] = 1

    async def open_client(self, client):
        self.clients.add(client)

        async def handle_message(value):
            self.properties["threshold"] = value
            await self.services.set_properties(self.properties)
            for other_client in self.clients:
                if other_client != client:
                    other_client.send_message(value)
            await self.services.request_run()

        client.set_message_handler(handle_message)
        client.send_message(self.properties["threshold"])

    async def close_client(self, client):
        self.clients.remove(client)

    async def run(self, inputs):
        if "data_in" in inputs:
            input_text = inputs["data_in"]
            input_text = input_text.replace("'","")
            frequencies = {}
            words = re.sub(r'[^\w\s]', ' ', input_text).split(' ')

            for word in words:
                word = word.strip()
                if word:
                    if word not in frequencies:
                        frequencies[word] = 0
                    frequencies[word] += 1

            table = []
            for word in frequencies:
                if frequencies[word] >= self.properties["threshold"]:
                    table.append([word, frequencies[word]])
            table = sorted(table, key=lambda r:r[1], reverse=True)
            table.insert(0,["word","frequency"])
            return {"data_out": table}
        else:
            return {}

//   Hyrrokkin - a library for building and running executable graphs
//
//   MIT License - Copyright (C) 2022-2025  Visual Topology Ltd

var textgraph = textgraph || {};

textgraph.WordFrequencyNode = class {

    constructor(services) {
        this.services = services;
        this.clients = new Set();
        this.properties = null;
    }

    async load() {
        this.properties = await this.services.get_properties();
        if (!("threshold" in this.properties)) {
            this.properties["threshold"] = 1;
        }
    }

    async open_client(client) {
        this.clients.add(client);
        client.set_message_handler(async (...msg) => await this.handle_message(client, ...msg));
        client.send_message(this.properties["threshold"]);
    }

    async close_client(client) {
        this.clients.delete(client);
    }

    async handle_message(from_client, value) {
        this.properties["threshold"] = value;
        await this.services.set_properties(this.properties);
        this.clients.forEach((other_client) => {
            if (other_client !== from_client) {
                other_client.send_message(value);
            }
        });
        this.services.request_run();
    }

    async run(inputs) {
        if ("data_in" in inputs) {
            let input_text = inputs["data_in"];
            input_text = input_text.replaceAll("'","");
            let words = input_text.replace(/[^\w\s]/g," ").split(" ");
            let frequencies = {};
            words.forEach((word) => {
                word = word.trim()
                if (word) {
                   if (!(word in frequencies)) {
                       frequencies[word] = 0;
                   }
                   frequencies[word] += 1;
                }
            });
            let table = [];
            for (let word in frequencies) {
                if (word) {
                    if (frequencies[word] >= this.properties.threshold) {
                        table.push([word, frequencies[word]]);
                    }
                }
            }
            table.sort(function(r1, r2) {
                return r2[1] - r1[1];
            });
            table.splice(0,0,["word", "frequency"])
            return {"data_out": table};
        } else {
            return {};
        }
    }
}

The node uses an integer stored in a threshold property to ignore low frequency words and the services api get_properties) and set_properties(properties) are used to retrieve and update the properties.

Property names must be strings and values must be JSON-serialisable objects.

TableDisplayNode

The TableDisplayNode implements some code to send an input table data structure and send it to connected clients.

pythonjavascript

#   Hyrrokkin - a library for building and running executable graphs
#
#   MIT License - Copyright (C) 2022-2025  Visual Topology Ltd

import json

from hyrrokkin_engine.node_interface import NodeInterface

class TableDisplayNode(NodeInterface):

    def __init__(self, services):
        self.services = services
        self.clients = set()
        self.input_value = None

    async def reset_run(self):
        self.input_value = None
        for client in self.clients:
            client.send_message(self.input_value)

    async def run(self, inputs):
        self.input_value = None
        if "data_in" in inputs:
            self.input_value = inputs["data_in"]
            self.services.set_status(f"{len(self.input_value) - 1} rows", "info")
        else:
            self.services.set_status("No data", "warning")
        for client in self.clients:
            client.send_message(self.input_value)

    async def open_client(self, client):
        self.clients.add(client)
        client.send_message(self.input_value)

    async def close_client(self, client):
        self.clients.remove(client)

//   Hyrrokkin - a library for building and running executable graphs
//
//   MIT License - Copyright (C) 2022-2025  Visual Topology Ltd

var textgraph = textgraph || {};

textgraph.TableDisplayNode = class {

    constructor(services) {
        this.services = services;
        this.clients = new Set();
        this.input_value = null;
    }

    async reset_run() {
        this.input_value = null;
        this.services.set_status("","info");
        this.clients.forEach((client) => {
            client.send_message(this.input_value);
        });
    }

    async run(inputs) {
        this.input_value = null;
        if ("data_in" in inputs) {
            this.input_value = inputs["data_in"];
            this.services.set_status(this.input_value ? `${this.input_value.length - 1} rows` : "", "info");
        } else {
            this.services.set_status("No data", "warning");
        }
        this.clients.forEach((client) => {
            client.send_message(this.input_value);
        });
    }

    open_client(client) {
        this.clients.add(client);
        client.send_message(this.input_value);
    }

    close_client(client) {
        this.clients.delete(client);
    }
}

Node lifecycle - the load, reset_run, run and close methods.

When a topology is loaded, or when any upstream node in the topology is re-run, the node will be constructed and its load method, if implemented, will be called.

As the topology is executed, the node's inputs will be collected and its run method will be called. But, before this happens, the node's reset_run method will be called, if it is implemented. A node can implement this method to inform any clients that the node's current results are invalid and the node will soon be re-run.

The reset_run method is called as soon as the framework is aware that the node's run method will need to be called.

A node may implement a remove method to receive notifications when the node is removed from a topology

The configuration is then accessed by nodes via the get_configuration service method.

For more details on the methods that a node or configuration can implement, see

• Node abstract base class API documentation
• Configuration abstract base class API documentation

For more details on the services API passed to node or configuration constructors, see:

• Node services API documentation
• Configuration API documentation

Creating, loading and running topologies using the Hyrrokkin API

Hyrrokkin provides a Python API for creating, running and loading and saving topologies

from hyrrokkin.api.topology import Topology

# provide the resource path to the package containing the schema file
textgraph_package = "hyrrokkin.example_packages.textgraph"

t = Topology(execution_folder=tempfile.mkdtemp(),package_list=[textgraph_package])

t.add_node("n0", "textgraph:text_input_node")
t.set_node_data("n0","value","This is some text")
t.add_node("n1", "textgraph:word_frequency_node", properties={"threshold":1})
t.add_node("n2", "textgraph:table_display_node")

t.add_link("l0", "n0", "data_out", "n1", "data_in")
t.add_link("l1", "n1", "data_out", "n2", "data_in")
runner = t.open_runner()
runner.run()

The same topology can be expressed using a YAML file

metadata:
  name: test topology
nodes:
  n0:
    type: textgraph:text_input_node
    data:
      value: path/to/textfile.txt
  n1:
    type: word_frequency_node
    properties:
      threshold: 1
  n2:
    type: textgraph:table_display_node
links:
- n0:data_out => n1:data_in
- n1:data_out => n2:data_in

This YAML file can then be imported using the following API calls

from hyrrokkin.api.topology import Topology
from hyrrokkin.utils.yaml_importer import import_from_yaml

# provide the resource path to the package containing the schema file
textgraph_package = "hyrrokkin.example_packages.textgraph"

t = Topology(execution_folder=tempfile.mkdtemp(),package_list=[textgraph_package])
import_from_yaml(t,"topology.yaml")

Note that in the links section of the YAML file, where nodes have only one input or output port, the port name can be omitted in the links section:

metadata:
  name: test topology
configuration:
  ...
nodes:
  ...
links:
- n0 => n1
- n1 => n2

Saving and loading topologies

A topology including its properties and data can be saved to and loaded from a serialised zip format file, using the following API calls. Saving first:

from hyrrokkin.api.topology import Topology

# provide the resource path to the package containing the schema file
textgraph_package = "hyrrokkin.example_packages.textgraph"

t = Topology(execution_folder=tempfile.mkdtemp(),package_list=[textgraph_package])
t.add_node("n0", "textgraph:text_input_node")
t.set_node_data("n0","value","this is some text")
t.add_node("n1", "textgraph:word_frequency_node",properties={"threshold": 1})
t.add_node("n2", "textgraph:table_display_node")

t.add_link("l0", "n0", "data_out", "n1", "data_in")
t.add_link("l1", "n1", "data_out", "n2", "data_in")

with open("topology.zip","wb") as f:
    t.save_zip(f)

To load from a saved topology:

from hyrrokkin.api.topology import Topology

# provide the resource path to the package containing the schema file
textgraph_package = "hyrrokkin_example_packages.textgraph"

t = Topology(execution_folder=tempfile.mkdtemp(),package_list=[textgraph_package])
with open("topology.zip","rb") as f:
    t.load_zip(f)

A utility function is also provided to export a topology to YAML format. Note that the exported YAML file contains node and configuration properties but does not contain node and configuration data.

from hyrrokkin.api.topology import Topology
from hyrrokkin.utils.yaml_exporter import export_to_yaml

# provide the resource path to the package containing the schema file
textgraph_package = "hyrrokkin_example_packages.textgraph"

t = Topology(execution_folder=tempfile.mkdtemp(),package_list=[textgraph_package])
with open("topology.zip","rb") as f:
    t.load(f)
with open("topology.yaml","w") as f:
    export_to_yaml(t,f)

For full details on the topology API, see:

• Topology API documentation

loading, saving and running topologies using topology_runner CLI

The Hyrrokkin package will install a hyrrokkin CLI command. Some typical usages include:

Import a topology from zip and run it:

hyrrokkin --packages hyrrokkin_example_packages.textgraph \      
                --execution-folder /tmp/execution_test \
                --import-path topology.zip --run

Import a topology from yaml, run it and save the topology (including data) to a zip file:

hyrrokkin --packages hyrrokkin_example_packages.textgraph \
                --execution-folder /tmp/execution_test \
                --import-path topology.yaml \
                --run --export-path topology.zip

Convert a topology from zip format to yaml format, but do not run it:

hyrrokkin --packages hyrrokkin_example_packages.textgraph \
                --execution-folder /tmp/execution_test \
                --import-path topology.zip \
                --export-path topology.yaml

Using the Hyrrokkin expression parser

Often nodes need to work with string-based expressions, for example:

r * sin(theta)

Hyrrokkin provides a simple expression based parser which can be set up to parse simple string based expressions into a parse tree.

from hyrrokkin_engine_utils.expr_parser import ExpressionParser
import json

ep = ExpressionParser()
ep.add_binary_operator("*",1)
print(json.dumps(ep.parse("10 * sin(pi)"),indent=2))

This program will print:

{
   "operator": "*",
   "args": [
     {
        "literal": 10
     },
     {
       "function": "sin",
       "args": [
         {
           "name": "pi"
         }
      ]
    }
  ]
}

Parser limitations

• unary and binary operators must be explicity registered with the parser
• unary operators have higher precedence than binary operators
• binary operators must be registered with a precedence