HPC Cluster Blog – Memory Amount Limits

Memory amount limits: What’s possible for your system?

The appropriate amount of memory for a particular node or server is usually best determined by looking at the number of memory channels available on the system’s motherboard. These channels are communication pathways that run between the memory and the memory controller, a component inside the processor chips; more channels generally translate to a faster data transfer rate. In order to take advantage of the fastest data rate possible, all channels should be filled with at least one memory stick.

With the latest processors using a four-channel memory architecture, it’s clear that all memory should be supplied in a multiplier of four: a minimum of four sticks (or DIMMs, “dual inline memory modules”) per processor. For dual-processor systems, this means that eight DIMMs, two sets of four, is the smallest amount needed to provide good performance.

The next obvious question is what size DIMMs can be used. Many of our repeat customers, familiar with the memory sizes they purchased with their dual-processor systems three or four years ago, will often ask for 16GB (eight 2GB DIMMs) or 32GB (eight 4GB DIMMs) in their new nodes and servers. However, as is the case with all components that comprise HPC systems, memory availability changes rapidly; what was appropriate in 2011 is now obsolete. At present, 2GB DIMMs for HPC applications are no longer available, and while 4GB DIMMs are available, memory manufacturers are moving production away from these and toward larger sizes, meaning that we’re starting to experience long lead times for orders using 4GB DIMMs.

As a result, the memory configuration we’re now recommending for our dual-processor systems is a minimum of 64GB: eight 8GB DIMMs. This 8GB size is still readily available and is generally a good amount for most HPC applications. Undoubtedly, these 8GB DIMMs will be phased out at some point, so be aware that the recommended amount now may double in the near future as 16GB DIMMs become the new standard. Eight 16GB DIMMs equals 128GB memory per dual-processor system, and as pricing on the 16GB DIMMs falls, many of our customers are already opting for this larger amount.

While we aren’t quoting many 32GB DIMMs yet, they do exist, and we expect to see more sales of this larger size as prices continue to fall. Only two years ago, 32GB DIMMs were prohibitively expensive at $1,000 or more per DIMM. Now the price is a few hundred per DIMM. While still expensive, it may not be too far in the future that we’ll begin to see 256GB systems (using eight 32GB DIMMs) becoming more common.

Keep in mind that should you need more than 64GB or 128GB of memory now, all of our server motherboards hold at least 16 DIMMs, two for each channel. So you can easily reach 256GB of memory using 16 of the 16GB DIMMs.

As memory sizes and prices continue to fluctuate, our sales and technical staff are here to walk you through all your available options. Please contact us for recommendations on the best memory configuration for your HPC systems.