How to move or migrate resources into projects

HERE platform projects are resource containers specific to your organization, that allow you to group and manage access to platform resources. Projects let you easily manage access control and track resource usage, both through the platform and command line interface (CLI).

HERE recommends that you use projects to manage all your platform resources. This guide walks you through the mostly CLI-based steps to migrate resources for a use case — including catalogs, schemas, pipelines, and associated pipeline templates — into a platform project, if those resources weren't originally created in a platform project. Once your catalogs for a use case are in a project, you can still share them with all or specific projects in your organization so that you can reuse them for other use cases.

Terminology

Home project: This typically means the project in which you created a resource. For this migration guide, it means the project to which you are adding or moving resources. This project now becomes the "home project" for those resources. A resource can only have one home project, but you can share catalogs with other projects in your organization to reuse them for multiple use cases. Storage for a catalog logs to the home project, and data I/O for using the catalog logs to the project that's using it.
Adding vs. moving resources: This guide refers to adding catalogs to projects. This means that you are adding the catalog to a project, but not removing any permissions associated with the catalog outside the context of the project. You can choose to remove those permissions later and purely manage the resource in the project, but "adding" rather than completely "moving" the catalog into a project helps mitigate the risk of breaking anything as you migrate your use case to a project. With pipelines, the guide refers to moving them rather than adding them, so they aren't accessible outside of project context once moved.

Step 1 - Create a project for your use case

Create a project using the platform portal Projects Manager or the olp project create CLI command.

To create a project using the Projects Manager in the platform, follow these steps:

Sign in to the HERE platform.
Select the Projects Manager from the launcher.
Click Create new project.
Enter a name for the project (project names don't have to be unique).
Enter a project ID. Project IDs must be unique within an org and can't change for the lifetime of the org. Project IDs must be 4-16 characters in length.
Enter an optional description.
Click Save.

Step 2 - Link HERE catalogs to your project

Link any HERE catalogs used in your use case to the project you created in step 1 using the platform portal, or the olp project resource link availability list and olp project resources link CLI commands.

🚧
Note
Link the "HERE Optimized Map for Visualization Plus" catalog (hrn:here:data::olp-here:here-optimized-map-for-visualization-plus-2) to your project so Data Inspector continues to work seamlessly. This allows you to visually inspect layers of catalogs in your projects (Data Inspector uses the Optimized Map for Visualization to provide the background map).

To link a HERE catalog from the HERE platform, follow these steps:

Sign in to the HERE platform.
Select the Projects Manager from the launcher.
Select the project to which you want to add the catalog and select the Resources tab.
Click Add catalog and choose to Link a catalog.
Select the catalog to link from the list displayed.

Step 3 - Add input and output catalogs and schemas to your project using the CLI

You can add input and output catalogs and schemas to your project.

🚧
Caution
Ensure at least one app has manage access to your input and output catalogs before you add the catalogs to projects so that you can continue to manage access outside of projects. Once you add your catalogs to projects, you won't be able to manage access to them outside the project using the portal, but can continue to do so using the CLI, which requires an app to authenticate.

Add your input and output catalogs and schemas to the project you created in step 1 using the resource project add CLI command. Permissions outside of the project aren't impacted, as any pipelines that currently use these catalogs continue to run. Access to schemas associated to a catalog is available in the project scope without any additional work required. However, you can also add schemas you created to a project using the olp resource project add CLI command.

How to remove catalogs and schemas outside of a project

If you create a catalog or schema outside of a project by mistake, you can remove it with the olp resource project remove CLI command. If the catalog or schema is shared to other projects before it's removed from a project, this action revokes its availability in the other projects. If the catalog or schema is shared and also linked to another project before it's removed from its home project, this action revokes those links.

Step 4 - Add members to your project and manage their access to project resources

Add members (users, groups, apps) to your project using the olp project access grant CLI command or the platform portal.

📘
Note
Add run-as ID (runtime credentials) for pipeline versions as members of the project.

Full access to all resources in the project is granted to these project members by default. You can optionally restrict access to resources by a project member by applying existing HERE or custom-created policies using the CLI. The CLI Projects Workflows explains how to set up granular access to the project resources through project policy. The Project Policy Commands allow you to create and manage custom policies.

Restrict access to catalogs by run-as ID, which are runtime credentials, if desired (optional), but verify that they have the minimum required rights of reading from input catalogs and writing to output catalogs.

To add a member to a project, follow these steps:

Log in to the HERE platform.
In the Projects Manager, click the project to which you want to give a user, group, or an app access.
Select the Access and permissions tab to view a list of users, groups, and apps with access to the project.
Click Grant access.
Select from either user, group, or app type.
Enter the name in the search field to find the desired user, group, or app by name.
Select the desired name and click Grant access. The name is now listed.

Step 5 - Move pipelines and related pipeline templates to a project

This step describes how to move a pipeline created outside of a project into a project. For batch pipelines, you may prefer to simply recreate the pipeline, cancel the versions outside the project, then activate the versions in the project, as described in steps 6-8 which follow.

📘
The pipeline job continues to run with the run-as ID (runtime credentials) using existing permissions to catalogs outside of the project from before the move. Verify the pipeline job is running as expected. The pipeline must redeploy with a new job to switch it to use the run-as ID (runtime credentials) and catalogs in project scope, and report the usage in project scope. To re-deploy, you can pause and resume the running pipeline version. You can also deactivate and activate the pipeline version if there is no processing state to preserve.

To move pipelines to a project, follow these steps:

Start by checking the pipeline versions associated with the pipeline you want to move to determine if the five most recent (the maximum supported) or fewer should move. You can do this check using the olp pipeline version list CLI command.
Next, check to see if other pipelines are using pipeline templates used by pipeline versions you need. Use the --report parameter of the olp pipeline move CLI command. If so, create separate pipeline templates, either for use in this pipeline or the other pipelines.

Now you're ready to move the pipeline, specified pipeline versions, and associated pipeline templates to the project. 3. Use the olp pipeline move CLI command.

📘
Note
The olp pipeline move command revokes permissions attached to the group that were associated to the pipeline outside of project context, without impacting the permissions inside the project. This means you can now only access the pipeline through the project.

Pipeline versions that weren't included in the move can't reactivate after the move, because their associated templates weren't moved.

What if you make a mistake? You can use the rollback option, which is the --revert-to-group parameter of the olp pipeline move CLI command to remove the pipeline, pipeline versions, and associated pipeline templates from the project. Then add them back to the group where they were previously located before the migration.

The rollback option isn't available if any of the following statements are true:

More than 5 calendar days have passed since migration.
The original group was deleted.
One or more new pipeline templates have been created and associated to new pipeline version after the move.
One or more moved pipeline templates have been deleted after the move.
One or more associated pipeline templates have been shared and/or linked with other projects.
One or more associated pipeline templates are referenced by a different pipeline's versions (within the same project).

Step 6 - Recreate batch pipelines in the project - alternative to step 5

For a batch pipeline, you can choose to run the pipeline immediately or schedule it to run when the input catalog data is updated.

You can recreate batch pipelines in the project (including pipeline template, pipeline, and first pipeline version) using the CLI or platform portal. Ensure that you specify project scope, regardless of the interface. If you use the platform portal and use the Pipelines tab instead of the Projects Manager, ensure that the project is selected from the drop-down menu.

Step 7 - Cancel the batch pipeline versions outside the project - alternative to step 5

Cancel the pipeline versions outside the project using the olp pipeline version cancel CLI command or platform portal.

Step 8 - Activate the batch pipeline versions in the project - alternative to step 5

Ensure the cancellation in step 7 is complete, then activate the pipeline versions in the project using the olp activate pipeline version CLI command or platform portal.

Step 9 - Update automated deployments to work in the project context

Update automated deployment access to resources which will be in the context of your new project by specifying the project in the --scope parameter with CLI commands, in the here.token.scope optional parameter of the credentials file, or by specifying a default project for your app using the olp app scope CLI command.

To read more about the order in which these three methods of specifying scope are resolved in the Scopes section, see Credentials setup.

Step 10 - Optional - Remove access to catalogs outside of project context

Once you have validated that your use case is working inside the project, you may choose to remove access to catalogs outside of the project context. This ensures that access to those catalogs is now managed only within the context of the project.

To remove permissions on the catalog outside of the project, use the olp catalog permission list and the olp catalog permission revoke CLI commands.