HPC Cluster Blog – Oversubscribing Your Network?

To oversubscribe your IB network or not:

When deploying an high performance computing cluster with more than 36 nodes that has an InfiniBand network, one of the most important considerations (beyond the bandwidth – QDR vs. FDR) is whether to oversubscribe or not. There are valid reasons for both approaches. Our first option is a non-blocking network – that is, where it operates at full wire speed. The primary consideration here is if there are any jobs that will run on this cluster can use more than 36 nodes. If there are, then the decision is pretty straightforward. You either buy a switch large enough to accommodate all of the available nodes or you set up a fat tree topology that provides for full bisectional bandwidth.

But what if the jobs to be run on the cluster can’t scale beyond 12-, 24, or 36-nodes? Then the choice should be to oversubscribe the network (the subscription rate is topic for another discussion). There are several benefits to doing this:

• Cost savings – if you have a cluster of 48 compute nodes, you’d need five 36-port switches to get full bandwidth (2 switches on the top level and three on the bottom). But if no job scales beyond 24 nodes, then you’d only need three switches which can result in savings of ~$10 – 15K. This additional money can be used to buy additional nodes which can allow more jobs to run concurrently.
• Easier cable management – with fewer switches, you need fewer cables. And depending on the size of the cluster, this could potentially lead to better airflow within the cabinet.
• More jobs can run – since no job will utilize all of the nodes at once, more jobs can run. This increases cluster utilization and can lead to more discovery.

We’re not here to advocate one over the other, but our experience has shown that most applications do not scale over 24 or 36 nodes. If that is the case, then the better approach would be to set up an oversubscribed network. Ask your Advanced Clustering rep. to discuss your clustering needs at [email protected].