class FlinkKafkaDelayedSourceImplFactory[K, V] extends FlinkKafkaSourceImplFactory[K, V]
createDelayedKafkaSourceWithFixedDelay is used to create KafkaSource with specified fixed delay (eg 2 hours).
createDelayedKafkaSource is used to create KafkaSource that delays events using custom DelayCalculator
It is used by experimental delayed sources, that unlike delay component, do not use state to keep track of delayed events.
This is the optimization for scenarios that require very long delays, where we don't want to keep big state and/or
keeping events in state for such long time can result in memory issues.
We assume that event timestamps are monotonous. First event is read and when extractTimestamp + delay < processing time the process sleeps.
Linear Supertypes
Ordering
- Alphabetic
- By Inheritance
Inherited
- FlinkKafkaDelayedSourceImplFactory
- FlinkKafkaSourceImplFactory
- Serializable
- KafkaSourceImplFactory
- AnyRef
- Any
- Hide All
- Show All
Visibility
- Public
- Protected
Instance Constructors
- new FlinkKafkaDelayedSourceImplFactory(timestampAssigner: Option[TimestampWatermarkHandler[ConsumerRecord[K, V]]], extractTimestampFromField: (String) => SerializableTimestampAssigner[ConsumerRecord[K, V]])
- timestampAssigner
- provides event timestamp to calculate delay, user is allowed to calculate delay in relation to their selected field that represents "business timestamp".
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @HotSpotIntrinsicCandidate() @native()
- def createDelayedKafkaSource(preparedTopics: NonEmptyList[PreparedKafkaTopic[ForSource]], kafkaConfig: KafkaConfig, deserializationSchema: KafkaDeserializationSchema[ConsumerRecord[K, V]], timestampAssigner: Option[TimestampWatermarkHandler[ConsumerRecord[K, V]]], formatter: RecordFormatter, contextInitializer: ContextInitializer[ConsumerRecord[K, V]], testParametersInfo: KafkaTestParametersInfo, delayCalculator: DelayCalculator, namingStrategy: NamingStrategy): FlinkKafkaSource[ConsumerRecord[K, V]]
- Attributes
- protected
- def createDelayedKafkaSourceWithFixedDelay(preparedTopics: NonEmptyList[PreparedKafkaTopic[ForSource]], kafkaConfig: KafkaConfig, deserializationSchema: KafkaDeserializationSchema[ConsumerRecord[K, V]], timestampAssigner: Option[TimestampWatermarkHandler[ConsumerRecord[K, V]]], formatter: RecordFormatter, contextInitializer: ContextInitializer[ConsumerRecord[K, V]], testParametersInfo: KafkaTestParametersInfo, delay: Long, namingStrategy: NamingStrategy): FlinkKafkaSource[ConsumerRecord[K, V]]
- Attributes
- protected
- def createSource(params: Params, dependencies: List[NodeDependencyValue], finalState: Any, preparedTopics: NonEmptyList[PreparedKafkaTopic[ForSource]], kafkaConfig: KafkaConfig, deserializationSchema: KafkaDeserializationSchema[ConsumerRecord[K, V]], formatter: RecordFormatter, contextInitializer: ContextInitializer[ConsumerRecord[K, V]], testParametersInfo: KafkaTestParametersInfo, namingStrategy: NamingStrategy): Source
- Definition Classes
- FlinkKafkaDelayedSourceImplFactory → FlinkKafkaSourceImplFactory → KafkaSourceImplFactory
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @HotSpotIntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @HotSpotIntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @HotSpotIntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @HotSpotIntrinsicCandidate() @native()
- def prepareTimestampAssigner(kafkaConfig: KafkaConfig, extract: SerializableTimestampAssigner[ConsumerRecord[K, V]]): TimestampWatermarkHandler[ConsumerRecord[K, V]]
- Attributes
- protected
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- val timestampAssigner: Option[TimestampWatermarkHandler[ConsumerRecord[K, V]]]
- Attributes
- protected
- Definition Classes
- FlinkKafkaSourceImplFactory
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)