Isolating guest VMs from each other (not just the host)

Question

Hello Everyone,
I've been playing around with visualization for a long time, but largely for testing and educational purposes.
For the first time I'm thinking of actually running some production systems in virtual machines instead of on physical hardware which which raises some new concerns for me that weren't really relevant previously, namely, security.  In particular, I've been thinking about how to isolate the guest VM processes from each other.
It's decades-old best practice to isolate different server services to different machines to help prevent, say, security problems with your web server also comprising your mysql server.  When you start virtualizing your infrastructure, you lose this benefit to a degree, if someone manages to compromise one of your VM servers and manages to exploit a vulnerability in your hypervisor to gain access to the host, it's no big leap to assume that they may be able to get access to your other virtual machines running on that host.
It seems to me, that if you want to improve your resilience to this kind-of exploit that you want to increase the isolation of your VM processes from each other.  You could, say, run them as different users.  If you're careful about permissions, then you gain back some of the security of running your servers on seperate physical hardware.  An attacker could still exploit a local privilege escalation bug, but it at least provides another line of defense.
This sort of thing seems to be possible with qemu virtual machines, though it does require some work to get anything besides user mode networking working. 
On the other hand, this sort of thing seems to be largely impossible with libvirt.  It's possible to run as qemu vms as different users using 'qemu://session' but the documentation seems to suggest that this limits you to using the qemu user mode networking, which isn't practical for running publicly accessible servers.  Since, as far as I can tell, all of the visualization management products around are based off of libvirt, this surprises me.  It seems to me that someone would have wanted to try and do something like this before, but nobody has (as far as I can tell).
Indeed, there seems to be very little documentation out there about using qemu with tun/tap networking when running as non-root user.  There is some documentation out there about using VDE (mostly via the deprecated vdeq wrapper), but alot of qemu write-ups skip over it.  There's enough out there to suggest it's possible, it just doesn't seem to be written up anywhere (yet, maybe I'll write it up myself someday).
So, this makes me wonder, 'why not?'.  Are there other things people are doing to isolate guest VMs from each other?
What are people running production services in VMs doing?

Answer 1

I suppose this is also something to think about, but I don't think I'd be that worried about it for my use-case.  The majority of systems I plan to run could run completely headless and the physical security for the host computers should be pretty good.  I'm more concerned remote code exploits in one particular VM allowing an attacker to execute on the host or in other guest VMs.