For a side-job I've got, there's a bunch of VMs running via VMWare Fusion. In trying to start another VM, Fusion kept reporting that it didn't have enough memory to start the new VM.
The inner workings of Fusion are a bit of a mystery, but it appears that all of the VMs run as vmware-vmx processes running as root, with the GUI frontend running as whichever user lauched the VM, the two halves communicating via a pipe. There are various hints online for decypering a WSSCAN line that appears in a particular VM's log, but nothing definitive and no indication of what limits, if any, apply to Fusion, how to determine what those limits are, and what knobs may apply to change the limits.
Activity Monitor on the host was able to tell me how much memory was wired (top will also tell you the same, along with other things). I could verify that there was memory left to be wired, and, with a small python script I was able, as root, to malloc and mlock 1.5GB of memory (and could see the memory actually being used and being wired). So, it wasn't a system-wide or per-user limit.
In the end, it appeared that a reboot fixed the issue. The amount of wired memory used dropped signifcantly, even with all the previous VMs plus the new one running, although I have no idea how Fusion decides to wire memory. So, no idea if over time this amount will grow and then at some future date we'll be unable to create a new VM even with memory free.
There's also some indication that you can put:
prefvmx.allVMMemoryLimit = "20480"
in your /Library/Preferences/VMware Fusion/config configuration file to change the amount of memory that Fusion thinks it can use (the amount in the quotes is memory in megabytes). But, again, no definitive guide to that that I can find; just a bunch of cargo-cult thinking that it works.