Packages

c

pl.touk.nussknacker.engine.kafka.generic

FlinkKafkaDelayedSourceImplFactory

class FlinkKafkaDelayedSourceImplFactory[K, V] extends FlinkKafkaSourceImplFactory[K, V]

createDelayedKafkaSourceWithFixedDelay is used to create KafkaSource with specified fixed delay (eg 2 hours). createDelayedKafkaSource is used to create KafkaSource that delays events using custom DelayCalculator It is used by experimental delayed sources, that unlike delay component, do not use state to keep track of delayed events. This is the optimization for scenarios that require very long delays, where we don't want to keep big state and/or keeping events in state for such long time can result in memory issues. We assume that event timestamps are monotonous. First event is read and when extractTimestamp + delay < processing time the process sleeps.

Linear Supertypes
FlinkKafkaSourceImplFactory[K, V], Serializable, Serializable, KafkaSourceImplFactory[K, V], AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. FlinkKafkaDelayedSourceImplFactory
  2. FlinkKafkaSourceImplFactory
  3. Serializable
  4. Serializable
  5. KafkaSourceImplFactory
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new FlinkKafkaDelayedSourceImplFactory(timestampAssigner: Option[TimestampWatermarkHandler[ConsumerRecord[K, V]]], extractTimestampFromField: (String) ⇒ SerializableTimestampAssigner[ConsumerRecord[K, V]])

    timestampAssigner

    - provides event timestamp to calculate delay, user is allowed to calculate delay in relation to their selected field that represents "business timestamp".

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @HotSpotIntrinsicCandidate()
  6. def createDelayedKafkaSource(preparedTopics: NonEmptyList[PreparedKafkaTopic[ForSource]], kafkaConfig: KafkaConfig, deserializationSchema: KafkaDeserializationSchema[ConsumerRecord[K, V]], timestampAssigner: Option[TimestampWatermarkHandler[ConsumerRecord[K, V]]], formatter: RecordFormatter, contextInitializer: ContextInitializer[ConsumerRecord[K, V]], testParametersInfo: KafkaTestParametersInfo, delayCalculator: DelayCalculator, namingStrategy: NamingStrategy): FlinkKafkaSource[ConsumerRecord[K, V]]
    Attributes
    protected
  7. def createDelayedKafkaSourceWithFixedDelay(preparedTopics: NonEmptyList[PreparedKafkaTopic[ForSource]], kafkaConfig: KafkaConfig, deserializationSchema: KafkaDeserializationSchema[ConsumerRecord[K, V]], timestampAssigner: Option[TimestampWatermarkHandler[ConsumerRecord[K, V]]], formatter: RecordFormatter, contextInitializer: ContextInitializer[ConsumerRecord[K, V]], testParametersInfo: KafkaTestParametersInfo, delay: Long, namingStrategy: NamingStrategy): FlinkKafkaSource[ConsumerRecord[K, V]]
    Attributes
    protected
  8. def createSource(params: Params, dependencies: List[NodeDependencyValue], finalState: Any, preparedTopics: NonEmptyList[PreparedKafkaTopic[ForSource]], kafkaConfig: KafkaConfig, deserializationSchema: KafkaDeserializationSchema[ConsumerRecord[K, V]], formatter: RecordFormatter, contextInitializer: ContextInitializer[ConsumerRecord[K, V]], testParametersInfo: KafkaTestParametersInfo, namingStrategy: NamingStrategy): Source
    Definition Classes
    FlinkKafkaDelayedSourceImplFactory → FlinkKafkaSourceImplFactory → KafkaSourceImplFactory
  9. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  10. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  11. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  12. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  13. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  14. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  16. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  17. def prepareTimestampAssigner(kafkaConfig: KafkaConfig, extract: SerializableTimestampAssigner[ConsumerRecord[K, V]]): TimestampWatermarkHandler[ConsumerRecord[K, V]]
    Attributes
    protected
  18. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  19. val timestampAssigner: Option[TimestampWatermarkHandler[ConsumerRecord[K, V]]]
    Attributes
    protected
    Definition Classes
    FlinkKafkaSourceImplFactory
  20. def toString(): String
    Definition Classes
    AnyRef → Any
  21. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  22. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  23. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated @deprecated
    Deprecated

    (Since version ) see corresponding Javadoc for more information.

Inherited from FlinkKafkaSourceImplFactory[K, V]

Inherited from Serializable

Inherited from Serializable

Inherited from KafkaSourceImplFactory[K, V]

Inherited from AnyRef

Inherited from Any

Ungrouped