Streaming-Lite API should not be considered stable at the moment.
Common features of Lite components
Base API of Lite components is here This is a generic API that is used by Streaming-Lite engine, but can be used to implement own engines, e.g. running in Request-Response way, or embedded in your application. For that reason, this API has a few parameters, that are fixed in StreamingLite:
F- the effect monad, in Streaming-Lite it is Scala
Input- the type of data (e.g. events processed by the given engine). In Streaming-Lite it's
Result- the type of data produced by sinks of a given engine. In Streaming-Lite it's
ProducerRecord[Array[Byte], Array[Byte]]You can see sample implementation of different Lite engine.
The data can be processed in microbatches,
in Streaming-Lite the batch contains all the records that are read with single a
Processing of a single batch ends with a list of results, each of the results may by either
Result in case of success,
NuExceptionInfo in case of an error.
Sources and sinks
In Streaming-Lite sources and sinks are relatively simple, as all sources and sinks are based on Kafka:
- A source
ConsumerRecordto Nussknacker Context.
- A sink creates
ProducerRecordfrom its parameters.
- Both may return error.
Lite custom transformers
Some transformers can work with arbitrary effect
F, the examples are basic components
like union or for-each. If custom component depends on the specific effect
(e.g. in Streaming-Lite you want to invoke a custom service which returns
you can use
CapabilityTransformer provided by
CustomComponentContext to return desired effect. If this component will be used
with a different effect, error will be returned by scenarion compiler.
See example for the details.