How to continously read data using Flink
How to continously read data using Flink
Flink can read data continuously from non-stream layers. This means that during
the runtime of the Flink execution environment the regular stream layer
processing is ongoing and if there is a change in the data of a non-stream
layer, the data is read and injected into the processing. This gives the
functionality to not just read such layers once at the startup of the Flink
execution environment but during runtime as well.
The behavior slightly differs depending on layer type like this:
-
Versioned layer:
For versioned layer this function is enabled by default. It uses layer
notifications. Therefore it does not need to poll but gets notified if a new
version is published. To disable this function just add the property
olp.connector-refresh-intervalwith value-1to your environment or
application configuration.Note: For versioned layer this feature works only for catalogs with
notificationsEnabled = truein catalog configuration. -
Volatile, Index and Interactive Map layers:
For these layers you can define the refresh (polling) interval in milliseconds
by adding the propertyolp.connector-refresh-intervalto your environment or
application configuration. By default, this interval is60000Lmeaning 60
seconds.