Skip to main content

Streaming-Lite engine

Architecture overview

One of the runtime execution engines provided by Nu is called Streaming-Lite it provides a way to deploy a Nussknacker scenario as a Kubernetes microservice.

Architecture layers diagram

This architecture provides significant benefits:

  • We leverage industry standards - Kafka and K8s - to handle tough topics like scaling, resilience, operations, delivery guarantees.
  • Scenario runtimes are independent of each other and the Designer,
  • in particular, each scenario deployment runs and scales independently of other scenarios.

Scenario deployment

K8s deployment diagram

Each Streaming-Lite scenario creates a K8s Deployment that uses a dedicated nu-runtime-image, each new deployment of a scenario creates a new Deployment version. A detailed deployment configuration can be set through deployment manager configuration, both scenario and its configuration are passed to Nu runtime pods as Config Maps where an interpreter can execute scenario logic as JVM code without an additional layer such as Flink.

Scenario execution

Each scenario is composed of three main components:

  • Kafka consumers of input topics
  • Interpreted scenario
  • Output topic Kafka producer
Scenario execution overview

During deployment, runtime POD compiles scenario to interpretable parts. This is when components implementations are extracted. Some implementations are streaming-lite specific but from the designer perspective, most component definitions look the same for every execution engine. After compilation, Streaming-Lite runtime polls on the Kafka consumers to wait for incoming messages, interprets compiled parts and results of scenario logic are produced to the output topic. All of that happens in kafka transactions so messages have exactly once delivery guarantee.