Double processing
When working with Docker deployments of HERE Anonymizer Self-Hosted you can create a setup in which two Anonymizer instances process the same input raw probe data stream. This allows you to compare the results of the anonymization process and select the output data that matches your criteria.
----- Anonymizer A --- Anonymized stream A
|
Raw probes stream ------|
|
------Anonymizer B --- Anonymized stream B
The implementation changes depending on the type of source connectors used.
RabbitMQ
An example for RabbitMQ is available in the deployments/docker/ab-case/docker-compose.yml file;
it contains two Flink JobManagers and two Flink TaskManagers pointed to corresponding JobManagers.
Follow these steps to run the example and see the setup in action:
-
From the application directory, build the
here-anonymizer:latestimage.docker build -t here-anonymizer:latest . -
Run the example.
cd ./deployments/docker/ab-case/ docker-compose up -d -
Access the consoles of two Anonymizer instances.
- Anonymizer 1 at http://localhost:8081
- Anonymizer 2 at http://localhost:8091
-
Increase the number of message streams consumed by RabbitMQ by declaring a special
fanoutexchange and binding multiple queues to it. Run the supplied script../configure-routing-and-publish.shRunning this script:
- Declares two queues:
input-queue-aandinput-queue-b - Declares fanout exchange
input-queue - Binds
input-queueto bothinput-queue-aandinput-queue-b - Publishes
deployments/common/here-probe-example.jsontoinput-queueexchange
- Declares two queues:
-
Check
Running Jobs -> HERE Anonymizer -> Accumulatorsfor anonymization stats of both running Anonymizer instances.As the instances use different configuration files, the number of
HERE_output_point_info_allandHERE_anonymization_point_dropped_startdiffers between the instances due to the difference in thestartCut.publishAftervalue. -
Stop the example.
docker-compose down
HERE platform stream layers
HERE platform stream layers are based on Kafka. As a result, the same approach as one used when configuring HERE platform stream URI applies:
jobmanager-a:
image: here-anonymizer:latest
### ...
environment:
- SOURCE_URI=olp+stream://platform.here.com/hrn:here:data::my-org:my-catalog/my-stream-layer?consumerGroup=anonA
## ...
jobmanager-b:
image: here-anonymizer:latest
### ...
environment:
- SOURCE_URI=olp+stream://platform.here.com/hrn:here:data::my-org:my-catalog/my-stream-layer?consumerGroup=anonB
### ...Updated 25 days ago