AWS ParallelCluster

Home » AWS Cheat Sheets » AWS Compute Services » AWS ParallelCluster

AWS ParallelCluster

Last updated on June 23, 2023

AWS ParallelCluster Cheat Sheet

  • An AWS-supported open source cluster management tool for deploying and managing High Performance Computing (HPC) clusters on AWS. ParallelCluster uses a simple text file to model and provision all the resources needed for your HPC applications in an automated and secure manner.
  • Tutorials dojo strip
  • AWS ParallelCluster provisions a master instance for build and control, a cluster of compute instances, a shared filesystem, and a batch scheduler. You can also extend and customize your use cases using custom pre-install and post-install bootstrap actions.

How It Works

AWS ParallelCluster

  • You have four supported schedulers to use along with ParallelCluster:
    • SGE (Son of Grid Engine)
    • Torque
    • Slurm
    • AWS Batch
  • AWS ParallelCluster supports 
    • On-Demand,
    • Reserved,
    • and Spot Instances

AWS ParallelCluster Networking

    • AWS ParallelCluster uses Amazon Virtual Private Cloud (VPC) for networking. The VPC must have DNS Resolution = yes, DNS Hostnames = yes and DHCP options with the correct domain-name for the Region. 
    • AWS ParallelCluster supports the following high-level configurations:
      • One subnet for both master and compute instances.
      • Two subnets, with the master in one public subnet, and compute instances in a private subnet. The subnets can be new or existing.
    • AWS ParallelCluster can also be deployed to use an HTTP proxy for all AWS requests.

AWS ParallelCluster Storage

    • By default, AWS ParallelCluster automatically configures an external volume of 15 GB of Elastic Block Storage (EBS) attached to the cluster’s master node and exported to the cluster’s compute nodes via Network File System (NFS).
    • AWS ParallelCluster is also compatible with Amazon Elastic File System (EFS), RAID, and Amazon FSx for Lustre file systems. 
    • You can configure AWS ParallelCluster with Amazon S3 object storage as the source of job inputs or as a destination for job output.
  • Cluster Configuration
    • By default, AWS ParallelCluster uses the file ~/.parallelcluster/config for all configuration parameters. A custom configuration file may be specified via the -c or –config command line option or the AWS_PCLUSTER_CONFIG_FILE environment variable.
    • The following sections are required: 
      • [global] section and [aws] section.
      • At least one [cluster] section and one [vpc] section.
  • Cluster Processes
    • When a cluster is running, a process called a jobwatcher monitors the configured scheduler ( SGE , Slurm , or Torque ) and each minute, it evaluates the queue in order to decide when to scale up.
    • The sqswatcher process monitors for Amazon SQS messages that are sent by Auto Scaling, to notify you of state changes within the cluster.
    • The nodewatcher process runs on each node in the compute fleet and terminates instances that have been idle for a set amount of time.

AWS ParallelCluster Pricing

    • AWS ParallelCluster is available at no additional charge. You pay only for the AWS resources needed to run your applications.

Limitations

    • AWS ParallelCluster does not support building Windows clusters.
    • AWS ParallelCluster does not currently support mixed instance types for a cluster. However, you can pick one instance type for the master node and another instance type for the compute nodes.

 

AWS ParallelCluster Cheat Sheet References:

https://aws.amazon.com/hpc/parallelcluster/
https://docs.aws.amazon.com/parallelcluster/latest/ug/what-is-aws-parallelcluster.html
https://aws.amazon.com/hpc/faqs/
https://aws.amazon.com/blogs/opensource/aws-parallelcluster/

Tutorials Dojo portal

Be Inspired and Mentored with Cloud Career Journeys!

Tutorials Dojo portal

Enroll Now – Our Azure Certification Exam Reviewers

azure reviewers tutorials dojo

Enroll Now – Our Google Cloud Certification Exam Reviewers

Tutorials Dojo Exam Study Guide eBooks

tutorials dojo study guide eBook

FREE AWS Exam Readiness Digital Courses

Subscribe to our YouTube Channel

Tutorials Dojo YouTube Channel

FREE Intro to Cloud Computing for Beginners

FREE AWS, Azure, GCP Practice Test Samplers

Recent Posts

Written by: Jon Bonso

Jon Bonso is the co-founder of Tutorials Dojo, an EdTech startup and an AWS Digital Training Partner that provides high-quality educational materials in the cloud computing space. He graduated from Mapúa Institute of Technology in 2007 with a bachelor's degree in Information Technology. Jon holds 10 AWS Certifications and is also an active AWS Community Builder since 2020.

AWS, Azure, and GCP Certifications are consistently among the top-paying IT certifications in the world, considering that most companies have now shifted to the cloud. Earn over $150,000 per year with an AWS, Azure, or GCP certification!

Follow us on LinkedIn, YouTube, Facebook, or join our Slack study group. More importantly, answer as many practice exams as you can to help increase your chances of passing your certification exams on your first try!

View Our AWS, Azure, and GCP Exam Reviewers Check out our FREE courses

Our Community

~98%
passing rate
Around 95-98% of our students pass the AWS Certification exams after training with our courses.
200k+
students
Over 200k enrollees choose Tutorials Dojo in preparing for their AWS Certification exams.
~4.8
ratings
Our courses are highly rated by our enrollees from all over the world.

What our students say about us?