Working with a recent customer, we had the experience of
designing a solution involving a number of very large (average 12-16 vCPU) machines. In order to maxime the performance of these VMs, we needed to fully understand the intricacies of server resource management technologies NUMA and vNUMA. Failing
to understand how they worked could have cost the customer performance gains
that these technologies offer.
So what are NUMA and vNUMA, exactly? And how does the proper
understanding of them benefit an administrator’s virtual environment?
So, what is NUMA?
“NUMA,” which is short for “Non-Uniform Memory Access,”
describes a system with more than one system bus. CPU resources and memory
resources are grouped together into a “NUMA node.”
The memory in a NUMA
node is thus much more easily accessed by an associated CPU. A CPU that needs
to access memory in a different node (“Remote Access”) will experience much
higher latency, and thus reduced application performance.
NUMA is, in short, an alternative approach to server
architecture that links several small, high-performing nodes together inside a
single server case.
So, why NUMA?
So long as the memory and CPU being used falls within the
bounds of the NUMA node, local communication within a NUMA node allows a CPU
much faster access to memory than in an ordinary system layout. This is
especially important in the multi-GHz era of today, when CPUs operate
significantly faster than the memory they are using. NUMA helps keep CPUs from
entering a stalled state, waiting for data, by allowing the fastest possible
access to memory.
How do I determine
the size of my NUMA nodes?
According to Microsoft
, “In most cases you can determine your NUMA node boundaries by
dividing the amount of physical RAM by the number of logical processors
This can be considered a very loose guideline. Further information on
determining the specific setup for your NUMA nodes can be found here:
ESX has been NUMA-aware since at least 2002, with VMware ESX
Server 1.5 introducing memory management features to improve locality on NUMA
hardware. This has worked well for placement of VMs and memory locality for
resources being used by that virtual machine, particularly for virtual machines
that are smaller than the NUMA node. Large VMs, however, could benefit from
extra help when it comes to scheduling their applications.
When enabled, vNUMA exposes a VM operating system to the
physical NUMA topology. This allows for performance improvements within the VM
by allowing the operating system and applications to take advantage of NUMA
optimizations. This allows VMs to benefit from NUMA, even if the VM itself is
larger than the physical size of the NUMA nodes.
A few quick points:
An administrator can adjust, enable, or disable
vNUMA on VMs using advanced vNUMA controls.
If a VM has more than eight vCPUs, vNUMA is
If you enable CPU HotAdd, vNUMA is disabled.
What happens regarding
vNUMA during a vMotion between hosts?
A VM’s vNUMA topology will mimic the topology of the host on
which it is initally placed; this topology does not adjust if a VM moves to a different
host unless the VM is restarted. This is another excellent argument for keeping
your hardware consistent within an ESXi cluster, as moving the VM to an ESXi host with a different NUMA topology could result in lost CPU/Memory locality and reduced performance.
For more information, consult the "Guest Operating Systems" section of the VMware Performance Best Practices
Regardless of your virtualization platform, NUMA plays an
important part in understanding performance within a virtual environment.
VMware, in ESXi versions 5.0 and beyond, has extended the capabilities of large
VMs by intelligent NUMA scheduling and improving VM-level optimization with
vNUMA. It is important to understand both your NUMA and vNUMA topologies when
sizing your virtual machines.
Labels: ESXi, NUMA, Vmware, vNUMA