Migrating to AWS ParallelCluster v3

Alan Christie 2023-03-13

Automation IaC

Here we’re using the AWS ParallelCluster, an AWS supported open source cluster management tool that helps you to deploy and manage high performance computing clusters in the AWS Cloud. For a background refer to the AWS introduction.

In our earlier post we demonstrated the use of AWS ParallelCluster (v2) for Nextflow. Here we briefly discuss switching from ParallelCluster v2 to v3.

Our [nextflow-pcluster] repository contains further documentation and a more detailed guide for both v2 and v3 on the repository branches pcluster-v2 and pcluster-v3.

Migrating a v2 cluster configuration

If you’re starting with a v3 cluster your best initial route may be the migration tool supported by AWS. The pcluster3-config-converter tool was added in AWS ParallelCluster version 3.0.1. To use it you simply need to install the appropriate Python package: -

$ python -m venv venv
$ source venv/bin/activate
$ pip install aws-parallelcluster

And then, given a pre-existing v2 cluster configuration file (say config-v2.yaml), just run the tool: -

$ pcluster3-config-converter -c config-v2.yaml -o config-v3.yaml

You will then need to adjust the roles available to the designates AWS IAM user. The roles and permissions for v3 are much more complex than v3 but we have extracted the most useful (as an example) in our [nextflow-pcluster] repository. There you will find the following files: -

• iam/head-node.json
• iam/instance.json
• iam/privileged.json

The render-policies.sh should simplify the rendering of these files to include your REGION and ACCOUNT.

Once rendered, add these additional policy files to your chosen IAM user.