Kubernetes
By its popularity on top cloud platforms, Kubernetes is the preferable and most unified way for production deployment.
Deployment modes
As HERE Anonymizer Preprocessor is an Apache Flink application, it can be deployed in all the available modes:
The Application Mode is the recommended deployment mode for most business cases. It's the most stable mode that allows fine-tuning, and deployments on pre-existing clusters.
Basic deployment pattern
Same as for standalone Docker deployments, the basic running pattern is to run single JobManager with one or more TaskManagers.
All examples in the Kubernetes deployment guides use the Flink Application Mode and the following Kubernetes resource maps:
| Resource type | Resource name | Description, essential parameters |
|---|---|---|
| configmap | here-anonymizer-preprocessor-configs | Config files: flink configurations, logging, app.conf, ignite-config |
| configmap | here-anonymizer-indexer-env | Environment variables: SOURCE_URI, SOURCE_FORMAT |
| configmap | here-anonymizer-preprocessor-env | Environment variables: SOURCE_URI, SINK_URI, etc. |
| configmap | here-anonymizer-ignite-env | Environment variables: OPTION_LIBS, CONFIG_URI, etc. |
| deployment | ignite-service | Ignite service for demo purposes |
| job | indexer-jobmanager | Flink JobManager for Indexer |
| job | preprocessor-jobmanager | Flink JobManager for Preprocessor |
| deployment | indexer-taskmanager | Flink TaskManager, replicas: 2 for Indexer |
| deployment | preprocessor-taskmanager | Flink TaskManager, replicas: 2 for Preprocessor |
| pod | ignite-service-xxx | Running pod of Apache Ignite |
| pod | indexer-jobmanager-xxx | Running pod of JobManager for Indexer |
| pod | indexer-taskmanager-xx1 | Running pod of first TaskManager for Indexer |
| pod | indexer-taskmanager-xx2 | Running pod of second TaskManager for Indexer |
| pod | preprocessor-jobmanager-xxx | Running pod of JobManager for Preprocessor |
| pod | preprocessor-taskmanager-xx1 | Running pod of first TaskManager for Preprocessor |
| pod | preprocessor-taskmanager-xx2 | Running pod of second TaskManager for Preprocessor |
| service | indexer-jobmanager | Exposing indexer-jobmanager host into the cluster, ports: 6123, 6124 |
| service | indexer-jobmanager-ui | Exposing indexer JobManager UI outside the cluster, port 8081 |
| service | preprocessor-jobmanager | Exposing preprocessor-jobmanager host into the cluster, ports: 6123, 6124 |
| service | preprocessor-jobmanager-ui | Exposing preprocessor JobManager UI outside the cluster, port 8081 |
| service | ignite-service | Exposing Apache Ignite ignite-service host into the cluster, ports: 8080, 10800 |
Tips
Use the following command to open ports and enable network accessibility for debugging in a Kubernetes cluster.
kubectl create deployment utils --image=docker.io/wbitt/network-multitoolUse this command to check if a resource, for example the preprocessor-jobmanager, is accessible inside the cluster.
kubectl exec -it deployments/utils -- bash
bash-5.1# nc -zv preprocessor-jobmanager 6123
preprocessor-jobmanager (10.99.56.22:6123) openUpdated 26 days ago