Guides
Guides
  1. HERE Anonymizer Preprocessor User Guide
  2. Deployment environments

Azure VM

Prerequisites

  1. Install tools:
  2. Create an Azure Storage Account

Deploy HERE Anonymizer Preprocessor playbook

You can complete the deployment scenario by running the supplied scripts from the path where you unpacked HERE Anonymizer Preprocessor.

Run the commands and export environment variables indicated in the code block.

# Azure Storage Account name (just the name, not the FQDN)
export AZURE_STORAGE_ACCOUNT={YOUR_STORAGE_ACCOUNT_NAME}
export AZURE_STORAGE_KEY={YOUR_STORAGE_ACCOUNT_KEY}
export PREPROC_DIST_DIR={PATH_TO_UNPACKED_ANONYMIZER}

# You can use login from a service principal, see an example in
# ./deployments/kubernetes/azure/azure-login.sh
azure login

# Main deployment scenario including uploading here-anonymizer-preprocessor.jar to Azure storage,
# creating Ignite VM, JobManager VM and the ScaleSet of two instances for
# Task Managers. The script produces two files:
# - ./azure-deploy-vm.env: contains env variables for connecting to the JobManager and Ignite VMs
# - ./azure-deploy-vm.jobmanager.log: contains logs of JobManager
./deployments/vm/azure/azure-deploy-vm.sh

# Export SSH_TO_JM_CMD from ./azure-deploy-vm.env
set -a ; . ./azure-deploy-vm.env ; set +a

# There the HERE Anonymizer Preprocessor is deployed
# and processing data present in the source container.

# Getting JobID and print metrics statistic
JOB_ID=$($SSH_TO_JM_CMD curl -s "http://localhost:8081/jobs" | jq -r '.jobs[0].id')
$SSH_TO_JM_CMD curl -s "http://localhost:8081/jobs/${JOB_ID}/accumulators" | jq

# Shutdown the cluster and all its resources
./deployments/vm/azure/azure-shutdown-vm.sh

Additional variables

List of optional variables for configuring azure-deploy-vm.sh and azure-shutdown-vm.sh with their default values:

export APP_NAME="here-anon-preproc-azure-vm"
export APP_VERSION="latest"
export AZURE_RESOURCE_GROUP="here-anon-preproc-azure-vm-latest"
export AZURE_LOCATION="germanywestcentral"
export AZURE_VM_IMAGE="Ubuntu2204"
export AZURE_VM_SIZE="Standard_B2s"
export TAGS="no_user_tags=true"

Deploy manually

Login and create resource group

  1. Log in to Azure using the CLI.

    # User must have the Contributor role
az login

  2. Create a resource group.

    az group create --name "$AZURE_RESOURCE_GROUP" --location "$AZURE_LOCATION"

Upload HERE Anonymizer Preprocessor files

  1. Create an Azure Storage container.

    CONTAINER="${APP_NAME}-${APP_VERSION}"
az storage container create -n "${CONTAINER}"

  2. Upload distribution files and anonymization config.

    # Upload the application jar
az storage blob upload -f "./here-anonymizer-preprocessor.jar" -c "${CONTAINER}"

curl -Lf "${FLINK_DIST_DOWNLOAD_URL}" -o "${TMP_DIR}/flink.tgz"
az storage blob upload -f "${TMP_DIR}/flink.tgz" -c "${CONTAINER}"
az storage blob upload -f "./deployments/vm/azure/flink-install-and-run.sh" -c "${CONTAINER}"

curl -Lf "${IGNITE_DIST_DOWNLOAD_URL}" -o "${TMP_DIR}/ignite.zip"
az storage blob upload -f "${TMP_DIR}/ignite.zip" -c "${CONTAINER}" --overwrite
az storage blob upload -f "${SCRIPT_DIR}/ignite-install-and-run.sh" -c "${CONTAINER}" --overwrite

Deploy Ignite virtual machine

📘

Note

The Ignite deployment is used for demo purposes only.

  1. Create a virtual machine.

    IGNITE_VM="${APP_NAME}-${APP_VERSION}-ignite"
az vm create \
 --name "${IGNITE_VM}" \
 --image "${AZURE_VM_IMAGE}" \
 --size "${AZURE_VM_SIZE}" \
 --vnet-name "${AZURE_RESOURCE_GROUP}-vnet" \
 --subnet "${AZURE_RESOURCE_GROUP}-subnet" \
 --admin-username "azureuser" \
 --generate-ssh-keys \
 --tags ${TAGS} \
 --public-ip-sku Standard

  2. Apply the CustomScript Azure VM extension.

    envsubst < "${SCRIPT_DIR}/ignite-custom-script.template.json" > "${TMP_DIR}/ignite-custom-script.json"
az vm extension set --no-wait --vm-name "${IGNITE_VM}" -n CustomScript --publisher Microsoft.Azure.Extensions \
  --protected-settings "${TMP_DIR}/ignite-custom-script.json"

Deploy JobManager virtual machine

  1. Create a virtual machine.

    VM_FLINK_JM="${APP_NAME}-${APP_VERSION}-jobmanager"
az vm create \
 --name "${VM_FLINK_JM}" \
 --image "${AZURE_VM_IMAGE}" \
 --size "${AZURE_VM_SIZE}" \
 --vnet-name "${AZURE_RESOURCE_GROUP}-vnet" \
 --subnet "${AZURE_RESOURCE_GROUP}-subnet" \
 --admin-username "azureuser" \
 --generate-ssh-keys \
 --tags ${TAGS} \
 --public-ip-sku Standard

export IP_FLINK_JM=$(az vm show --name "${VM_FLINK_JM}" --show-details -o tsv --query publicIps)

  2. Apply the CustomScript Azure VM extension.

    export TASKMANAGER_INSTANCE_COUNT=2
JM_RPC_HOST=${VM_FLINK_JM} FLINK_MODE="jobmanager" envsubst < "${SCRIPT_DIR}/flink-custom-script.template.json" > "${TMP_DIR}/flink-jm-custom-script.json"
az vm extension set --no-wait --vm-name "${VM_FLINK_JM}" \
 -n CustomScript --publisher Microsoft.Azure.Extensions \
 --protected-settings "${TMP_DIR}/flink-jm-custom-script.json"
📘

Note

Optionally, you can enable the 8081 Flink UI port and it at http://${IP_FLINK_JM}:8081/. Note that this link is not secured.

az vm open-port --port 8081 --resource-group $AZURE_RESOURCE_GROUP --name ${VM_FLINK_JM}

Deploy Flink Task Managers Scale Set

  1. Create virtual machines scale set.

    VM_FLINK_TM="${APP_NAME}-${APP_VERSION}-taskmanager"
az vmss create \
 --orchestration-mode Uniform \
 --upgrade-policy-mode Automatic \
 --name "${VM_FLINK_TM}" \
 --image "${AZURE_VM_IMAGE}" \
 --vnet-name "${AZURE_RESOURCE_GROUP}-vnet" \
 --subnet "${AZURE_RESOURCE_GROUP}-subnet" \
 --vm-sku "${AZURE_VM_SIZE}" \
 --instance-count "${TASKMANAGER_INSTANCE_COUNT}" \
 --admin-username "azureuser" \
 --generate-ssh-keys \
 --public-ip-address-allocation static --public-ip-per-vm \
 --lb-sku Standard \
 --tags ${TAGS}

  2. Apply the CustomScript Azure VM extension.

    JM_RPC_HOST=${VM_FLINK_JM} FLINK_MODE="taskmanager" envsubst < "${SCRIPT_DIR}/flink-custom-script.template.json" > "${TMP_DIR}/flink-tm-custom-script.json"
# The extension will upgrade all the VM in scale set only if --orchestration-mode Uniform and --upgrade-policy-mode Automatic
az vmss extension set --vmss-name "${VM_FLINK_TM}" -n CustomScript --publisher Microsoft.Azure.Extensions \
  --protected-settings "${TMP_DIR}/flink-tm-custom-script.json"

Run the HERE Anonymizer Preprocessor

  1. Export the SSH command.

     export SSH_TO_JM_CMD="ssh azureuser@${IP_FLINK_JM}"

  2. Run the Indexer job.

    $SSH_TO_JM_CMD "
 export SOURCE_URI=\"az+blob+index://${AZURE_STORAGE_ACCOUNT}:${AZURE_STORAGE_KEY}@${SOURCE_PATH}?cache_endpoint=ignite://${IP_IGNITE}:10800&endpoint=https://${AZURE_STORAGE_ACCOUNT}.blob.core.windows.net\"
 export SOURCE_FORMAT=HERE_PROBE
 export SINK_FORMAT=HERE_PROBE
 /opt/flink/bin/flink run -c com.here.anonymization.data.preprocessor.Indexer /opt/flink/usrlib/here-anonymizer-preprocessor.jar
"

  3. Once the Indexer job has run successfully, submit the Preprocessor job.

    $SSH_TO_JM_CMD "
 export SOURCE_URI=\"az+blob+preprocess://${AZURE_STORAGE_ACCOUNT}:${AZURE_STORAGE_KEY}@${SOURCE_PATH}?cache_endpoint=ignite://${IP_IGNITE}:10800&endpoint=https://${AZURE_STORAGE_ACCOUNT}.blob.core.windows.net\"
 export SINK_URI=\"az+blob+batch+files://${AZURE_STORAGE_ACCOUNT}:${AZURE_STORAGE_KEY}@${SINK_PATH}?endpoint=https://${AZURE_STORAGE_ACCOUNT}.blob.core.windows.net\"
 export SOURCE_FORMAT=HERE_PROBE
 export SINK_FORMAT=HERE_PROBE
 /opt/flink/bin/flink run -c com.here.anonymization.data.preprocessor.PreProcessor /opt/flink/usrlib/here-anonymizer-preprocessor.jar
"

  4. Run this command to check the indexer report:

     az storage blob download --account-name "${AZURE_STORAGE_ACCOUNT}" --container-name "${AZURE_BLOB_CONTAINER}" --name "${SOURCE_PATH}/INDEXER_REPORT.json" --no-progress | jq "."

  5. Check if the essential metrics like HERE_decoding_point_info_all and HERE_output_point_info_all have a value greater than zero. See all the anonymization metrics explained here.

  6. Run this command to access the preprocessor report:

     az storage blob download --account-name "${AZURE_STORAGE_ACCOUNT}" --container-name "${AZURE_BLOB_CONTAINER}" --name "${SINK_PATH}/PREPROCESSOR_REPORT.json" --no-progress | jq "."

  7. Additionally, the preprocessed data can be browsed at the location referred by ${SINK_PATH}.

Cleanup

  1. Remove the Azure Resource Group to delete all associated resources.

    az group delete --name "$AZURE_RESOURCE_GROUP" -y

  2. Delete the container created for the uploaded application files.

    az storage container delete \
--name "${APP_NAME}-${APP_VERSION}" \
--account-name "$AZURE_STORAGE_ACCOUNT"

Updated 25 days ago