In the cloud era, a compute cluster that once took months to build out can now be created and ready to use in minutes. In this blog post, we will discuss all pieces that come together to make this near-instant infrastructure a reality. From there, we will show how this infrastructure and file system fulfills the promise of performance right out of the box.
The express path to a high-performance distributed file system and compute cluster begins with the IBM Spectrum Scale catalog tile. Follow the link, and the IBM Cloud Schematics interface offers a straightforward process for filling out the parameters to configure your cloud-based storage and compute cluster. After you provide all the configuration details, your input is stored as a Schematics workspace. This workspace contains your infrastructure specification, and upon your command, the workspace connects with the Terraform and Ansible code contained in the repository to create your cloud-based infrastructure.
The IBM Cloud VPC infrastructure used by the Spectrum Scale catalog tile can employ storage nodes based on bare metal instances with NVMe devices or use virtual instances with instance storage. For this post, we will be using bare metal instances that offer the following:
The number and configuration of the compute nodes is up to the user with virtual instance profiles:
In addition to the storage and compute nodes, the automation provisions and configures a bastion node that helps to secure the cluster’s VPC in several ways:
IBM Spectrum Scale is a high-performance clustered file system that provides concurrent access to a shared file system from multiple nodes. It can be used in a wide variety of hardware and software configurations. For our purposes, it is configured as a collection of nodes built from both bare metal servers for storage and virtual instances for compute. Each bare metal instance has direct-attached NVMe storage serving as NSD volumes and a 100 Gbps network interface. We’ll have more to say about this in the performance section.
The tile automation scripts build a cluster that employs simple and effective security practices to get you started:
From there, it is expected that you employ the rich set of tools supplied by IBM Cloud and Spectrum Scale to implement the level of security that meets your needs.
As discussed earlier, before it is rendered in real hardware, the cluster exists as a specification stored in a Schematics workspace. This workspace can be thought of as a form of infrastructure that incurs no cost or energy while in storage.
Assuming the cluster is already configured, the process of bringing it to life begins with invoking the “apply” command, which executes the pre-existing and well-tested Terraform scripts from the Schematics repository to provision the cloud resources. Whenever possible, the provision steps are carried out in parallel. In the case of our largest example, a 10 storage node and 64 compute node cluster, there can be close to 100 discrete cloud operations in flight at one time. In this way, for one example, the 64 compute nodes are provisioned concurrently and complete in a little over 1 minute, and so it goes with subnets, security rules, a bastion node, storage nodes and so on. Once the hardware is in place, Ansible scripts are kicked off to install and configure the software.
The following timings were measured on varying cluster configurations in real experiments and can be used as a guideline. As always, your results may vary to some degree. Three different cluster sizes were tested, and the times needed to create them were broken down to give an idea of how long various operations take.
| Cluster Type | Schematics Time | Controller Terraform Time | Controller Ansible Time | Total Time |
| 3-storage, 3-compute | 05:20 | 16:38 | 19:35 | 41:35 |
| 6-storage 64-compute | 05:02 | 17:11 | 32:12 | 54:25 |
| 10-storage 64-compute | 05:12 | 17:17 | 34:12 | 56:41 |
“Schematics time” is the amount of time spent running Terraform scripts in a Schematics container. This time is spent provisioning a login node and a “controller” node to which we transfer the responsibility for finishing the cluster. The reason we make this transition is to allow us to move execution to a node that we own and control. We can also size to speed up the process that is executing Terraform scripts to provision resources and later Ansible scripts to install software and configure the cluster.
In the table above, this time is split into the Controller Terraform and Controller Ansible components. The “Total Time” column is the elapsed time from “apply” to the cluster being ready to get to work. It is interesting to note how the performance varies as we scale up the cluster size. Schematics time is essentially invariant because it is the same amount of work in this phase, regardless of cluster size. The controller Terraform illustrates how successfully we can parallelize the Terraform provisions. In this case, the time needed to do 74 (10 storage + 64 compute) provisions is less than 5% longer than the time needed to do 6. In contrast, the Ansible-based configurations run serially in many cases, so the time needed is proportional to the number of nodes in the cluster.
| Cluster Type | Schematics Time | Controller Terraform Time | Controller Ansible Time | Total Time |
| 3-storage, 3-compute | 05:20 | 16:38 | 19:35 | 41:35 |
| 6-storage 64-compute | 05:02 | 17:11 | 32:12 | 54:25 |
| 10-storage 64-compute | 05:12 | 17:17 | 34:12 | 56:41 |
We also tested the time needed to destroy a cluster, and the results are in Table 2 above. The total time is made up of two separate operations. There are two operations due to the split nature of the Terraform work. Some of it runs on the Schematics container, while the bulk of the work is carried out on the bootstrap instance.
These two operations run sequentially, so the total time is obtained by adding the two operations together. Regardless of cluster size, it takes approximately 10 minutes to free all the resources and return them to the cloud. Just as in resource creation, we take advantage of the ability to run Terraform operations in parallel to keep the total time down.
Out of the box, our cluster offers resiliency that allows for the loss of a storage node and the loss of a storage block.
This level of redundancy requires two settings that are applied at cluster creation time:
The above settings can be seen as providing the basic level of resiliency that befits a large, clustered file system. Beyond this, and depending on your needs, Spectrum Scale and IBM Cloud can be customized to provide resiliency and security at very high levels.
| Operation | Performance | Threads/Compute | Request Size |
| Write Sequential | 35 GiB/sec | 12 | 4 MiB |
| Read Sequential | 112 Gib/sec | 8 | 4 MiB |
| Write Random | 861,797 IOPS | 128 | 4 KiB |
| Read Random | 5,447,134 IOPS | 80 | 4 KiB |
Table 3 provides an overview of Scale file system performance for a few key metrics. The testing was performed on a system with the following characteristics:
On the compute side, we have:
Digging into the results in Table 3, it should be evident that these are very good numbers for a clustered file system. The read bandwidth of 112 GiB/sec is essentially all the bandwidth supplied by the 10 100 Gbps network adapters, which means when it comes to read bandwidth, the Scale software and IBM Cloud network infrastructure is leaving nothing on the table. Write bandwidth is also good, operating under the constraints imposed by replication. The 5.4 million read IOPs supplied are also impressive. In short, this is a very high-performance offering out of the box.
It should be noted that all the results listed above were achieved “out of the box.” As with any high-performance computing system, the cluster has benefited from testing and tuning, but it was done over the course of our development and performance testing and the tuning is now applied automatically when a cluster is built from the tile.
The IBM Cloud Spectrum Scale catalog tile has been designed and built to offer you the shortest path possible to get to a high-performance compute and storage cluster. In less than one hour, you can build a compute/storage cluster to your specification with up to a 100 TB distributed file system, as much compute capacity as you desire and tuned to extract maximum performance from the underlying hardware. We invite you to try out our offering and embrace the cloud-based future of high-performance computing today.
Get started with IBM Cloud Spectrum Scale
The post Spectrum Scale on IBM Cloud performance appeared first on IBM Blog.
Understanding what's happening behind large language models (LLMs) is essential in today's machine learning landscape.
AI accelerationists have won as a consequence of the election, potentially sidelining those advocating for…
L'Oréal's first professional hair dryer combines infrared light, wind, and heat to drastically reduce your…
TL;DR A conversation with 4o about the potential demise of companies like Anthropic. As artificial…
Whether a company begins with a proof-of-concept or live deployment, they should start small, test…
Digital tools are not always superior. Here are some WIRED-tested agendas and notebooks to keep…