Transparent Page Sharing (TPS) disabled in ESXi future release

By default TPS has been enabled on ESXi hosts. TPS allows virtual machines to share identical memory pages and reduce the overall host memory footprint. This is achieved by allowing the hypervisor to scan at 60 minute intervals (default) for identical memory pages, reclaiming redundant copies and keeping a single memory page in physical RAM mapped to the virtual machines.

ESXi_TPS_ImageA recent article from VMware – Security Considerings and disallowing inter-Virtual Machine Transparent Page Sharing (KB2080735) acknowledges recent academic research that, under certain controlled circumstances, makes it possible to measure memory timings to try and determine a AES encryption key in use on another virtual machine running the same physical CPU. Whilst the probability of this occurring in the real world is low, VMware have taken the decision to disable TPS by default from future ESXi update releases as a security measure. However, administrators can turn it back on for the whole host or even individual virtual machines as needed manually.

New installations;

TPS no longer enabled by default from the following shipped releases.

  • ESXi 5.5 Update release – Q1 2015
  • ESXi 5.1 Update release – Q4 2014
  • ESXi 5.0 Update release – Q1 2015
  • The next major version of ESXi

Click more to read on

Existing environments following the patch;

ESXi patches will be introduced as follows. These will not change existing settings for TPS, but will introduce additional TPS management capabilities added to releases which allow granular control of TPS within ESXi hosts. Planned patch releases for existing environments are below.

New Settings for TPS;

As described in KB2091682 new Advanced Configuration options are introduced by the patch, including a new concept of salting which can control the virtual machine participating in TPS on the host.

Firstly we must enable TPS at a host level;

Mem.ShareForceSalting: This is a host-level configuration option. This is what disables/enables TPS on an ESXi host. If this is set to “0″, it means that TPS is STILL enabled on the host. If set to “1″, it means that TPS has been disabled on the Host, and salting is required in order for TPS to work on any VM located on that host.

  1. Log in to ESXi or vCenter with the VI-Client.
  2. Select the ESXi relevant host.
  3. In the Configuration tab, click Advanced Settings (link) under the software section.
  4. In the Advanced Settings window, click Mem.
  5. Look for Mem.ShareForceSalting and set the value to 1 (enable salting) / 0 (disable salting)
  6. Click OK.
  7. Follow one of the options shown here to get immediate effect on page sharing
    • Migrate all the VMs to some other host in cluster and back to original host or,
    • Gracefully shutdown and power-on the VMs

Secondly and optional, at a VM level salting can be configured to allow granular per VM allow/disallow participation in TPS;

sched.mem.pshare.salt: This value enables customers to selectively enable page sharing between/among specific VMs. WhenShareForceSalting is set to “1″ on an ESXi host, the only way for two or more VMs to share a page is for both their salt and the content of the page to be same. The salt is the value specified by customers for this per-VM Advanced Configuration option. This value must be identical on all the VMs that you intend to enable page sharing for.

  1. Log in to ESXi or vCenter with the VI-Client.
  2. Select the ESXi relevant host.
  3. In the Configuration tab, click Advanced Settings (link) under the software section.
  4. In the Advanced Settings window, click Mem.
  5. Look for ‘Mem.ShareForceSalting’ and set the value to 1.
  6. Click OK.
  7. Power off the VM, which you want to set salt value.
  8. Right click on VM, click on Edit settings.
  9. Select options menu, click on General under Advanced section
  10. Click on Configuration Parameters…
  11. Click on Add Row, new row will be added.
  12. On left hand side add text ‘sched.mem.pshare.salt’ and on the right hand side specify the unique string.
  13. Power on the VM to take effect of salting.
  14. Repeat steps 7 to 13 to set the salt value for individual VMs.
  15. Same salting values can be specified to achieve the page sharing across VMs.

IF ShareForceSalting is set to “1″ and the ‘sched.mem.pshare.salt’ is not set on a VM, the VM’s ‘vc.uuid’ will be substituted for the salt value instead. Because the ‘vc.uuid’ is unique to a VM, that VM will only be able to share page with itself – effectively, no sharing for this VM.

Will it make a difference to my environment?

ESXi manages memory by actively employing TPS to remove duplicate pages of RAM, if there is a memory scarcity (default 6% remaining), then the balloon driver engages followed by memory compression and finally swap out to disk.

I take this opportunity to note that TPS only actively scans for duplicate memory pages in virtual machines using 4KB memory pages, as opposed to not scanning the 2MB large pages ESXi which supports in virtual machines – because the cost of scanning is much higher in terms of CPU. TPS is only disabled for active scanning in large page virtual machines until there is a host memory scarcity (6% remaining), if there is a scarcity large pages are broken down into 4KB chunks and TPS sharing can kick in again. Is this a bad thing? No not in my opinion, as large pages mean less TLB misses and thus less host CPU cycles, but as a result of large pages you will see higher memory consumption on the host as nothing is being shared until there is a memory scarcity to enforce the sharing of pages. A nice read is this paper on Large Page performance.

In summary – having TPS off may affect the amount of memory you plan in your host design, and effect existing environments that are overcommitted actively relying on TPS. I would suggest an analysis on your environment to see how much TPS savings are currently being realised to aid your planning.

Check how much memory sharing is occurring on your host(s);

  1. Run esxtop on host via vCLI, switch to memory mode by pressing m
  2. free’ from ‘PMEM /MB’ row gives free memory available on the host
  3. curr’ from ‘MEMCTL/MB’ row gives total ballooned memory
  4. curr’ from ‘SWAP/MB’ row gives total swapped memory
  5. The one you are looking for in particular is ‘PSHARE’, where ‘saving’ value is the total memory saved because of TPS.