HPC Cluster Blog – HPC in the Cloud

Is HPC in the cloud right for you?

Over the years, I’ve attended many birds of feather where the topic was whether or not there was a future in moving HPC into the cloud.  There were plenty of vigorous debates for and against, and they tended to go this way:

For:
The most obvious argument for moving your computing to the cloud is eliminating the need for purchasing and maintaining hardware.  Most of the users are scientists or engineers who don’t want to spend their time managing a cluster.  By moving to the cloud, they allow others to buy, power, cool, and maintain the hardware while they focus on what they’re experts in.  The actual cost of a cluster is substantial, and the costs to house it, power it, and cool it are not insignificant.  Then, of course, someone has to administer and maintain it.  But there are many professors and engineers where this responsibility falls to them.  So, now they have to hire someone to maintain it, or do it themselves.  And if that’s the case, it takes away from their ability to do science.  With the cloud, you simply upload your data to the online resource, pay for the cycles you need, and once the job is complete, download the results to your local machine and do your work.  If you do not require constant access to a cluster to run your jobs, then this could be a viable option.

Against:
The biggest obstacle to overcome, from an HPC cloud provider, is that the und-user loses control of their data.  As a good many of the jobs being run on clusters are research-based and not production-type jobs, scientists and engineers are reluctant to let someone else hold “ownership” to their data (regardless of how well-encrypted it may be).  Scientists can spend many months using a cluster to prove one thing or another that they hope to publish; they’d rather not risk the chance (however small that may be) of their work being released prematurely.  There are other concerns, including potential access problems if the cloud goes down, not having access to proper hardware, and the time it could take to retrieve their results (particularly if we’re talking gigabytes or especially terabytes of data).

Another concern that has recently been brought up is that if you use a cloud to run your jobs, then someone in your organization typically has to help in getting you started as well as managing the transfer of data and so forth.  So, in addition to running an organization’s cluster, the added responsibility of effectively managing an outside resource falls to the cluster administrator.  At that point, the better option would be to commit the funds to organization’s cluster to increase capacity and get time on that resource.

These are the most common talking points, but they are by no means the only ones.  I fully expect this conversation to continue for quite some time.  I also suspect that HPC in a cloud may be a good fit for some, but for the majority of users, the preference seems to be to want the resources located on their campus or in their datacenter.