Parallelism

To enable parallel task execution, configure multiple Flink TaskManagers to point to a single JobManager.

The basic configuration pattern for Docker Compose is:

jobmanager:
  image: here-anonymizer:latest
  environment:
    # ...
    - |
      FLINK_PROPERTIES=
      jobmanager.rpc.address: jobmanager
      parallelism.default: 3

taskmanager:
  image: here-anonymizer:latest
  # ...
  scale: 3

Example

1. From the application directory, build the here-anonymizer:latest image:

   docker build -t here-anonymizer:latest .

2. Run the example:

   Shell

   cd ./deployments/docker/parallelism/
   docker-compose up -d

3. Inspect the Flink console at http://localhost:8081:

   • Check the Running Jobs -> HERE Anonymizer and verify that the second block of operators is described as Parallelism: 3.
   • Check TaskManagers and verify if you can see three TaskManagers.

4. Run this command to stop the example:

   Shell

   docker-compose down