How to continously read data using Flink

How to continously read data using Flink

Flink can read data continuously from non-stream layers. This means that during
the runtime of the Flink execution environment the regular stream layer
processing is ongoing and if there is a change in the data of a non-stream
layer, the data is read and injected into the processing. This gives the
functionality to not just read such layers once at the startup of the Flink
execution environment but during runtime as well.

The behavior slightly differs depending on layer type like this:

  • Versioned layer:

    For versioned layer this function is enabled by default. It uses layer
    notifications. Therefore it does not need to poll but gets notified if a new
    version is published. To disable this function just add the property
    olp.connector-refresh-interval with value -1 to your environment or
    application configuration.

    Note: For versioned layer this feature works only for catalogs with
    notificationsEnabled = true in catalog configuration.

  • Volatile, Index and Interactive Map layers:

    For these layers you can define the refresh (polling) interval in milliseconds
    by adding the property olp.connector-refresh-interval to your environment or
    application configuration. By default, this interval is 60000L meaning 60
    seconds.