Guides
Guides

Rescaling

Apply the following configuration to enable adaptive rescaling with checkpointing.

Adaptive mode

jobmanager:
  image: here-anonymizer:latest
  environment:
    # ...
    - |
      FLINK_PROPERTIES=
      jobmanager.rpc.address: jobmanager
      jobmanager.scheduler: adaptive
      execution.checkpointing.mode: EXACTLY_ONCE
      execution.checkpointing.dir: file:///tmp/hash-checkpoints
      execution.checkpointing.interval: 60s
      execution.checkpointing.timeout: 30s
      execution.checkpointing.max-concurrent: 1
      parallelism.default: 4

taskmanager:
  image: here-anonymizer:latest
  # ...
  scale: 2

Example

  1. From the application directory, build the here-anonymizer:latest image:

    docker build -t here-anonymizer:latest .
  2. Run the example:

    sh demo-start.sh
  3. Inspect the Flink console at http://localhost:8081:

    • Check the Running Jobs -> HERE Anonymizer and verify that the second block of operators is described as Parallelism: 2.
    • Check TaskManagers and verify if you can see three TaskManagers.
  4. Upscaling the TaskManager deployment to three replicas (with one task slot per TaskManager) results in the parallelism increasing to three. Use the following command:

    docker-compose -f ./deployments/docker/demo/docker-compose.yml \
    scale taskmanager=3
  5. Downscaling the TaskManager deployment to two replicas (with one task slot per TaskManager) results in the parallelism decreasing to two. Use the following command:

    docker-compose -f ./deployments/docker/demo/docker-compose.yml \
    scale taskmanager=2
  6. Run this command to stop the example:

    sh demo-stop.sh