Pipeline Processing Architecture

Pipeline Syntax

pipeline JSON has 2 components

Nodes

{
  "name": "node-01",
  "type": "source",
  "node_ref": "file_log",
  "config": {
    "include_path": ["/var/log/*.log"]
  },
  "outputs": ["out"],
  "multiprocessor": "multiprocessor-01",
  "supports": ["metrics", "logs", "traces"],
  "active": ["metrics", "logs"],
  "credential_ref": "cred_name"
}

Name	Acceptable Values	Mandatory	Description
name	non-empty string	Yes	userdefined node name which is configured in the UI
type	source, processor, destination, pack, connector	Yes	node category
node_ref	valid node definition identifier	Yes	node reference present in the node definition to classify the node
config	JSON object per node schema	Yes	output of JSONForms data for the node
inputs	array of input port names	Depends (mandatory for destinations)	list of input ports supported by the node
outputs	array of output port names	Depends (mandatory for sources)	list of output ports supported by the node
multiprocessor	string (processor-only)	No	name of the multiprocessor if part of any (only applicable when type is processor and part of multiprocessor in UI)
supports	metrics, logs, traces	Yes	the set of signals the node can handle
active	metrics, logs, traces	Yes	the type of signals the node is configured for
credential_ref	non-empty string	No	optional field only set when the source requires credentials to be set

Connections

{
  "to": "batch:in",
  "from": "devnull:out"
}

connections define the relationship between 2 nodes, where to specifies the source node and from specifies the target node. The connection is always assumed to be directed to -> from

The format for the values in "to" and "from" is

node_name:port_name

Example

{
  "nodes": [
    {
      "name": "user defined node name",
      "type": "source|processor|destination|pack|connector",
      "node_ref": "backend node_ref id for the node",
      "config": {},
      "outputs": ["out"],
      "multiprocessor": "name of multiprocessor if part of any",
      "supports": ["metrics", "logs", "traces"],
      "active": ["metrics", "logs"]
    }
  ],
  "connections": [
    {
      "to": "batch-1756882307193-wactgn:in",
      "from": "windows_events_system:out"
    }
  ]
}

Packs Definition

packs are an logical construct which have a mini pipeline defined inside it.

A pack in a complete pipeline is represented in the following way

Note: Packs are always restricted to a single type of Signal either logs, metrics or traces

{
  "name": "pack-1",
  "type": "pack",
  "pack_ref": "pack-1",
  "pack_version": "2.0.0",
  "inputs": ["in_1", "in_2"],
  "outputs": ["out"],
  "supports": ["metrics"],
  "active": ["metrics"]
}

actual pack definition is as follows

{
  "nodes": [
    {
      "name": "proc-1",
      "type": "processor",
      "node_ref": "batch",
      "config": {},
      "outputs": ["out"],
      "multiprocessor": "name of multiprocessor if part of any",
      "supports": ["metrics", "logs", "traces"],
      "active": ["metrics", "logs"]
    },
    {
      "name": "proc-2",
      "type": "processor",
      "node_ref": "batch",
      "config": {},
      "outputs": ["out"],
      "multiprocessor": "name of multiprocessor if part of any",
      "supports": ["metrics", "logs", "traces"],
      "active": ["metrics", "logs"]
    }
  ],
  "connections": [
    {
      "to": "batch:in",
      "from": "windows_events_system:out"
    }
  ]
}

Predefined Source or Destination nodes

Substituting predefined source or destination node

{
  "nodes": [
    {
      "name": "user defined node name",
      "type": "source|destination",
      "template_ref": "title_of_predefined_source/dest",
      "inputs": ["in"],
      "outputs": ["out"]
    }
  ],
  "connections": [
    {
      "to": "batch-1756882307193-wactgn:in",
      "from": "windows_events_system:out"
    }
  ]
}

The payload for predefined nodes is same as the normal ones but only difference being they will have a template_ref key with the source or dest name which they want to use and no config

Kubernetes-Aware Conversion

For pipelines that use Kubernetes source nodes (k8s_logs, k8s_kubeletstats, k8s_hostmetrics, k8s_cluster, k8s_events, k8s_appmetrics, k8s_otlp), Praxis converts the pipeline into one or more OpenTelemetryCollector custom resources that the in-cluster Praxis Collector supervisor applies to your cluster.

How the conversion works

Build the pipeline — Praxis produces the same OpenTelemetry configuration it would for any pipeline.
Group by workload — The configuration is split into per-workload groups based on the kind of Kubernetes signal each source collects (for example logs, kubelet metrics, cluster events). Each group becomes its own collector workload.
Merge node-level signals — When a pipeline collects two or more node-level signals (logs, kubelet metrics, host metrics), Praxis collapses them into a single node agent group by default, so the fleet runs one DaemonSet pod per node instead of one per signal. This roughly halves per-node overhead and the number of managed agents. The trade-off is a shared rollout cadence and a wider blast radius if that pod restarts, so it can be turned off when per-signal isolation matters more than pod count.
Filter each group — Every group carries only the receivers, processors, exporters, and extensions its own pipelines actually reference.
Package as a custom resource — Each filtered group is wrapped in an OpenTelemetryCollector custom resource. Its workload mode (DaemonSet, Deployment, or StatefulSet) and the host paths it mounts are derived automatically from the sources in that group — for example a log source brings the host paths needed to tail container logs, and a host-metrics source mounts the host filesystem.

Migration safety

On-node file offsets (the record of how far each log file has been read) are keyed to the pipeline, not to the individual signal workload. This means that when a pipeline switches between per-signal collectors and a merged node agent, the new pod re-uses the existing offsets instead of starting over and re-reading every log file from the beginning. This behavior is covered by automated tests.

Rollout posture

Node-agent merging is on by default. When a pipeline changes shape, the supervisor brings the new workload up and confirms it is ready before removing the old one, so signals are not dropped during the transition. Merging can be disabled to fall back to per-signal collectors — useful when you want per-signal blast-radius isolation more than a lower per-node pod count.
The supervisor treats each generated workload as an opaque custom resource and does not special-case any signal, so merged and per-signal layouts are handled the same way.

Placement on tainted nodes

When node-level signals are merged, Praxis takes the union of each source's placement settings so the merged node agent still lands where it should:

Tolerations — if any merged source opts into tolerating node taints, the merged DaemonSet tolerates them, so log collection on tainted nodes does not silently stop after a merge.
Priority class — the first configured priority class is applied.
Log-specific settings (such as log path, container runtime type, and multi-line recombination) come from the log source in the group.

Pipeline Syntax​

Nodes​

Connections​

Example​

Packs Definition​

Predefined Source or Destination nodes​

Kubernetes-Aware Conversion​

How the conversion works​

Migration safety​

Rollout posture​

Placement on tainted nodes​