Skip to main content
Version: Next

Introduction

Intended audience

This document is intended for those who will use Nussknacker Designer to configure processing scenarios. Nussknacker is a low-code platform; prior knowledge of SQL, JSON and concepts like variables and data types will help master authoring of the stream processing scenarios in Nussknacker.

Please try Quickstart to quickly understand how to move around Nussknacker Designer, create a simple scenario and see SpEL in action.

Events

Nussknacker nodes process events; once the node finishes processing of the event it hands it over to the next node in the flow for processing. Filter, Split, Switch nodes behave exactly like this. Typically events processed by Nussknacker come from Kafka topics; Nussknacker source components are used to read events from Kafka topic and inject them into Nussknacker scenarios. There are cases though when a node can produce a new event, a Tumbling-window or and Session-window component being good examples.

Notion of passing time is very important in dealing with real time events processing. Please see following excellent references to learn about basic concepts:

For Streaming-Flink engine Flink documentation applies. Certain Nussknacker components make assumptions and have predefined settings, so that the end users don't have to configure all by themselves.

In general following rules apply:

  • We use event time in scenarios to handle notion of passing time
  • Kafka record timestamps are used to assign event time to Flink events
  • Kafka records produced by Nussknacker sinks have timestamp of event (in the sense of event time) that generated them
  • We use bound of order watermark generator, with configurable amount of lateness (see kafka.kafkaEspProperties.defaultMaxOutOfOrdernessMillis property in Configuration for details).

If a new event is triggered by e.g. tumbling time window, its timestamp is equal to the time of the timer that generated it, not system time of the moment when it happened. See Aggregates in Time Windows for more details.

(Streaming-Lite only) Notion of time

Streaming-Lite is stateless, so many concepts important for windows or aggregations do not apply, but following rules apply for Kafka sources and sinks:

  • Kafka record timestamps are used to determine time of the event
  • Kafka records produced by Nussknacker sinks have timestamp of event that generated them

SpEL

Configuring Nussknacker nodes to a large degree is about using SpEL; knowledge of how to write valid expressions in SpEL is an important part of using Nussknacker.

SpEL Spring Expression Language is a powerful expression language that supports querying and manipulating data objects. What exactly does the term expression mean and why SpEL is an expression language? In programming language terminology, an expression is a combination of values and functions that are combined to create a new value. SpEL allows to write expressions only; therefore it is an expression language. Couple examples:

ExpressionResultType
'Hello World'"Hello World"String
truetrueBoolean
{1,2,3,4}a list of integers from 1 to 4List[Integer]
{john:300, alex:400}a map (name-value collection)Map[String, Integer]
2 > 1trueboolean
2 > 1 ? 'a' : 'b'"a"String
42 + 244Integer
'AA' + 'BB'"AABB"String

SpEL is used in Nussknacker to access data processed by a node and expand node's configuration capabilities. Some examples:

  • create boolean expression (for example in filters) based on logical or relational (equal, greater than, etc) operators
  • access, query and manipulate fields of the incoming data record
  • format records (events) written to data sinks
  • provide helper functions like date and time, access to system variables
  • and many more.

The SpEL Cheat Sheet page provides an exhaustive list of examples of how to write expressions with SpEL.

Data Types

Every SpEL expression returns a value of one of the predefined SpEL data types, like integer, double or boolean, map, etc. Data types in Nussknacker can be a confusing aspect at the beginning, as depending on the context in which data are processed or displayed, different data type schemes are in use - please refer to the SpEL Cheat Sheet page for more information.

In some contexts data type conversions may be necessary - conversion functions are described here.

Variables

Nussknacker uses variables as containers for data; they can be referred to in SpEL expressions. Variables have to be declared; a variable or mapVariable component are used for this. Once declared, a hash sign "#" is used to refer to a variable.

There are three predefined variables: #input, #inputMeta and #meta.

If the event which arrived to some node originally came from the Kafka topic, the data carried in the event record are available in the #input variable. The metadata associated with this event are available in #inputMeta variable. The following meta information fields are available in #inputMeta:

  • headers
  • key
  • leaderEpoch
  • offset
  • partition
  • timestamp
  • timestampType
  • topic. Consult Kafka documentation for the exact meaning of those fields.

The #meta variable carries meta information about the currently executed scenario. The following meta information elements are available:

  • processName - name of the Nussknacker scenario
  • properties

Check Basic Nodes page for examples how to use variables.