Packages

class KafkaSourceFactory[K, V] extends SourceFactory with SingleInputDynamicComponent[Source] with WithCachedTopicsExistenceValidator with WithExplicitTypesToExtract with UnboundedStreamComponent

Base factory for Kafka sources with additional metadata variable. It is based on pl.touk.nussknacker.engine.api.context.transformation.SingleInputDynamicComponent that allows custom ValidationContext and Context transformations, which are provided by KafkaContextInitializer Can be used for single- or multi- topic sources (as csv, see topicNameSeparator and extractTopics).

Features:

  • fetch latest N records which can be later used to test process in UI Fetching data is defined in source which may extends pl.touk.nussknacker.engine.api.process.TestDataGenerator. See pl.touk.nussknacker.engine.kafka.KafkaUtils#readLastMessages
  • reset Kafka's offset to latest value - forceLatestRead property, see pl.touk.nussknacker.engine.kafka.KafkaUtils#setOffsetToLatest
K

- type of key of kafka event that is generated by raw source (SourceFunction).

V

- type of value of kafka event that is generated by raw source (SourceFunction).

Linear Supertypes
UnboundedStreamComponent, WithExplicitTypesToExtract, WithCachedTopicsExistenceValidator, TopicsExistenceValidator, SingleInputDynamicComponent[Source], DynamicComponent[Source], SourceFactory, Component, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. KafkaSourceFactory
  2. UnboundedStreamComponent
  3. WithExplicitTypesToExtract
  4. WithCachedTopicsExistenceValidator
  5. TopicsExistenceValidator
  6. SingleInputDynamicComponent
  7. DynamicComponent
  8. SourceFactory
  9. Component
  10. Serializable
  11. AnyRef
  12. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new KafkaSourceFactory(deserializationSchemaFactory: KafkaDeserializationSchemaFactory[ConsumerRecord[K, V]], formatterFactory: RecordFormatterFactory, modelDependencies: ProcessObjectDependencies, implProvider: KafkaSourceImplFactory[K, V])(implicit arg0: ClassTag[K], arg1: ClassTag[V])

Type Members

  1. type ContextTransformationDefinition = PartialFunction[TransformationStep, TransformationStepResult]
    Definition Classes
    DynamicComponent
  2. type DefinedParameter = DefinedSingleParameter
    Definition Classes
    SingleInputDynamicComponent → DynamicComponent
  3. case class FinalResults extends TransformationStepResult with Product with Serializable
    Definition Classes
    DynamicComponent
  4. type InputContext = ValidationContext
    Definition Classes
    SingleInputDynamicComponent → DynamicComponent
  5. case class NextParameters extends TransformationStepResult with Product with Serializable
    Definition Classes
    DynamicComponent
  6. type State = KafkaSourceFactoryState[K, V]
    Definition Classes
    KafkaSourceFactory → DynamicComponent
  7. case class TransformationStep extends Product with Serializable
    Definition Classes
    DynamicComponent
  8. sealed trait TransformationStepResult extends AnyRef
    Definition Classes
    DynamicComponent

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def allowedProcessingModes: AllowedProcessingModes
    Definition Classes
    UnboundedStreamComponent
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @HotSpotIntrinsicCandidate() @native()
  7. def contextTransformation(context: ValidationContext, dependencies: List[NodeDependencyValue])(implicit nodeId: NodeId): ContextTransformationDefinition

    contextTransformation should handle exceptions raised by prepareInitialParameters

    contextTransformation should handle exceptions raised by prepareInitialParameters

    Definition Classes
    KafkaSourceFactory → DynamicComponent
  8. val deserializationSchemaFactory: KafkaDeserializationSchemaFactory[ConsumerRecord[K, V]]
    Attributes
    protected
  9. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  10. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  11. def extractTopics(params: Params): NonEmptyList[ForSource]

    Extracts topics from default topic parameter.

    Extracts topics from default topic parameter.

    Attributes
    protected
  12. def fallbackFinalResult(step: TransformationStep, inputContext: InputContext, outputVariable: Option[String])(implicit nodeId: NodeId): FinalResults
    Attributes
    protected
    Definition Classes
    DynamicComponent
  13. val formatterFactory: RecordFormatterFactory
    Attributes
    protected
  14. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @HotSpotIntrinsicCandidate() @native()
  15. def handleExceptionDuringTransformation(step: TransformationStep, inputContext: InputContext, outputVariable: Option[String], ex: Throwable)(implicit nodeId: NodeId): FinalResults
    Definition Classes
    DynamicComponent
  16. def handleUnmatchedTransformationStep(step: TransformationStep, inputContext: InputContext, outputVariable: Option[String])(implicit nodeId: NodeId): FinalResults
    Definition Classes
    DynamicComponent
  17. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @HotSpotIntrinsicCandidate() @native()
  18. val implProvider: KafkaSourceImplFactory[K, V]
    Attributes
    protected
  19. def implementation(params: Params, dependencies: List[NodeDependencyValue], finalState: Option[State]): Source

    Common set of operations required to create basic KafkaSource.

    Common set of operations required to create basic KafkaSource.

    Definition Classes
    KafkaSourceFactory → DynamicComponent
  20. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  21. val kafkaConfig: KafkaConfig
    Attributes
    protected
    Definition Classes
    KafkaSourceFactory → WithCachedTopicsExistenceValidator
  22. lazy val keyTypingResult: TypingResult
    Attributes
    protected
  23. val modelDependencies: ProcessObjectDependencies
    Attributes
    protected
  24. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  25. def nextSteps(context: ValidationContext, dependencies: List[NodeDependencyValue])(implicit nodeId: NodeId): ContextTransformationDefinition
    Attributes
    protected
  26. def nodeDependencies: List[NodeDependency]
    Definition Classes
    KafkaSourceFactory → DynamicComponent
  27. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @HotSpotIntrinsicCandidate() @native()
  28. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @HotSpotIntrinsicCandidate() @native()
  29. def prepareContextInitializer(dependencies: List[NodeDependencyValue], parameters: List[(ParameterName, DefinedParameter)], keyTypingResult: TypingResult, valueTypingResult: TypingResult): ContextInitializer[ConsumerRecord[K, V]]
    Attributes
    protected
  30. final def prepareFinalResultWithOptionalVariable(inputContext: InputContext, outputVariable: Option[(String, TypingResult)], state: Option[State])(implicit nodeId: NodeId): FinalResults
    Attributes
    protected
    Definition Classes
    DynamicComponent
  31. def prepareInitialParameters: List[Parameter]

    Basic implementation of definition of single topic parameter.

    Basic implementation of definition of single topic parameter. In case of fetching topics from external repository: return list of topics or raise exception.

    Attributes
    protected
  32. def prepareSourceFinalErrors(context: ValidationContext, dependencies: List[NodeDependencyValue], parameters: List[(ParameterName, DefinedParameter)], errors: List[ProcessCompilationError])(implicit nodeId: NodeId): FinalResults
    Attributes
    protected
  33. def prepareSourceFinalResults(context: ValidationContext, dependencies: List[NodeDependencyValue], parameters: List[(ParameterName, DefinedParameter)], keyTypingResult: TypingResult, valueTypingResult: TypingResult, errors: List[ProcessCompilationError])(implicit nodeId: NodeId): FinalResults
    Attributes
    protected
  34. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  35. def toString(): String
    Definition Classes
    AnyRef → Any
  36. val topicNameSeparator: String
    Attributes
    protected
  37. val topicParameterDeclaration: ParameterCreatorWithNoDependency with ParameterExtractor[String]
    Attributes
    protected
  38. def topicsValidationErrors(topicsString: String)(implicit nodeId: NodeId): List[CustomNodeError]
    Attributes
    protected
  39. def typesToExtract: List[TypingResult]
    Definition Classes
    KafkaSourceFactory → WithExplicitTypesToExtract
  40. final def validateTopic[T <: TopicName](topic: T)(implicit arg0: TopicValidationType[T]): Validated[TopicExistenceValidationException[T], T]
    Definition Classes
    TopicsExistenceValidator
  41. final def validateTopics[T <: TopicName](topics: NonEmptyList[T])(implicit arg0: TopicValidationType[T]): Validated[TopicExistenceValidationException[T], NonEmptyList[T]]
    Definition Classes
    WithCachedTopicsExistenceValidator → TopicsExistenceValidator
  42. lazy val valueTypingResult: TypingResult
    Attributes
    protected
  43. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  44. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  45. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  46. object FinalResults extends Serializable
    Definition Classes
    DynamicComponent

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from UnboundedStreamComponent

Inherited from WithExplicitTypesToExtract

Inherited from WithCachedTopicsExistenceValidator

Inherited from TopicsExistenceValidator

Inherited from SingleInputDynamicComponent[Source]

Inherited from DynamicComponent[Source]

Inherited from SourceFactory

Inherited from Component

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped