Rescaling
Apply the following configuration to enable adaptive rescaling with checkpointing.
Adaptive mode
jobmanager:
image: here-anonymizer:latest
environment:
# ...
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
jobmanager.scheduler: adaptive
execution.checkpointing.mode: EXACTLY_ONCE
execution.checkpointing.dir: file:///tmp/hash-checkpoints
execution.checkpointing.interval: 60s
execution.checkpointing.timeout: 30s
execution.checkpointing.max-concurrent: 1
parallelism.default: 4
taskmanager:
image: here-anonymizer:latest
# ...
scale: 2Example
-
From the application directory, build the
here-anonymizer:latestimage:docker build -t here-anonymizer:latest . -
Run the example:
sh demo-start.sh -
Inspect the Flink console at http://localhost:8081:
- Check the
Running Jobs -> HERE Anonymizerand verify that the second block of operators is described asParallelism: 2. - Check
TaskManagersand verify if you can see three TaskManagers.
- Check the
-
Upscaling the TaskManager deployment to three replicas (with one task slot per TaskManager) results in the parallelism increasing to three. Use the following command:
docker-compose -f ./deployments/docker/demo/docker-compose.yml \ scale taskmanager=3 -
Downscaling the TaskManager deployment to two replicas (with one task slot per TaskManager) results in the parallelism decreasing to two. Use the following command:
docker-compose -f ./deployments/docker/demo/docker-compose.yml \ scale taskmanager=2 -
Run this command to stop the example:
sh demo-stop.sh
Updated 26 days ago