Skip to main content
Version: 1.5

Streaming-Lite components

caution

Streaming-Lite API should not be considered stable at the moment.

Common features of Lite components​

Base API of Lite components is here This is a generic API that is used by Streaming-Lite engine, but can be used to implement own engines, e.g. running in Request-Response way, or embedded in your application. For that reason, this API has a few parameters, that are fixed in StreamingLite:

  • F - the effect monad, in Streaming-Lite it is Scala Future
  • Input - the type of data (e.g. events processed by the given engine). In Streaming-Lite it's ConsumerRecord[Array[Byte], Array[Byte]]
  • Result - the type of data produced by sinks of a given engine. In Streaming-Lite it's ProducerRecord[Array[Byte], Array[Byte]] You can see sample implementation of different Lite engine.

The data can be processed in microbatches, in Streaming-Lite the batch contains all the records that are read with single a consumer.poll.

Processing of a single batch ends with a list of results, each of the results may by either Result in case of success, or NuExceptionInfo in case of an error.

Sources and sinks​

In Streaming-Lite sources and sinks are relatively simple, as all sources and sinks are based on Kafka:

  • A source transforms ConsumerRecord to Nussknacker Context.
  • A sink creates ProducerRecord from its parameters.
  • Both may return error.

Lite custom transformers​

Generic Lite custom component is designed in continuation-passing style. Some helper traits are provided, you can also look at base components.

Capabilities transformer​

Some transformers can work with arbitrary effect F, the examples are basic components like union or for-each. If custom component depends on the specific effect (e.g. in Streaming-Lite you want to invoke a custom service which returns Future[_]), you can use CapabilityTransformer provided by CustomComponentContext to return desired effect. If this component will be used with a different effect, error will be returned by scenario's compiler. See example for the details.