<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <body>
    <h1>Bhyve driver</h1>

    <ul id="toc"></ul>

<p>
Bhyve is a FreeBSD hypervisor. It first appeared in FreeBSD 10.0. However, it's
recommended to keep tracking FreeBSD 10-STABLE to make sure all new features
of bhyve are supported.

In order to enable bhyve on your FreeBSD host, you'll need to load the <code>vmm</code>
kernel module. Additionally, <code>if_tap</code> and <code>if_bridge</code> modules
should be loaded for networking support.
</p>

<p>
Additional information on bhyve could be obtained on <a href="http://bhyve.org/">bhyve.org</a>.
</p>

<h2><a name="uri">Connections to the Bhyve driver</a></h2>
<p>
The libvirt bhyve driver is a single-instance privileged driver. Some sample
connection URIs are:
</p>

<pre>
bhyve:///system                     (local access)
bhyve+unix:///system                (local access)
bhyve+ssh://root@example.com/system (remote access, SSH tunnelled)
</pre>

<h2><a name="exconfig">Example guest domain XML configurations</a></h2>

<h3>Example config</h3>
<p>
The bhyve driver in libvirt is in its early stage and under active development. So it supports
only limited number of features bhyve provides.
</p>

<p>
Note: in older libvirt versions, only a single network device and a single
disk device were supported per-domain. However,
<span class="since">since 1.2.6</span> the libvirt bhyve driver supports
up to 31 PCI devices.
</p>

<p>
Note: the Bhyve driver in libvirt will boot whichever device is first. If you
want to install from CD, put the CD device first. If not, put the root HDD
first.
</p>

<p>
Note: Only the SATA bus is supported. Only <code>cdrom</code>- and
<code>disk</code>-type disks are supported.
</p>

<pre>
&lt;domain type='bhyve'&gt;
    &lt;name&gt;bhyve&lt;/name&gt;
    &lt;uuid&gt;df3be7e7-a104-11e3-aeb0-50e5492bd3dc&lt;/uuid&gt;
    &lt;memory&gt;219136&lt;/memory&gt;
    &lt;currentMemory&gt;219136&lt;/currentMemory&gt;
    &lt;vcpu&gt;1&lt;/vcpu&gt;
    &lt;os&gt;
       &lt;type&gt;hvm&lt;/type&gt;
    &lt;/os&gt;
    &lt;features&gt;
      &lt;apic/&gt;
      &lt;acpi/&gt;
    &lt;/features&gt;
    &lt;clock offset='utc'/&gt;
    &lt;on_poweroff&gt;destroy&lt;/on_poweroff&gt;
    &lt;on_reboot&gt;restart&lt;/on_reboot&gt;
    &lt;on_crash&gt;destroy&lt;/on_crash&gt;
    &lt;devices&gt;
      &lt;disk type='file'&gt;
        &lt;driver name='file' type='raw'/&gt;
        &lt;source file='/path/to/bhyve_freebsd.img'/&gt;
        &lt;target dev='hda' bus='sata'/&gt;
      &lt;/disk&gt;
      &lt;disk type='file' device='cdrom'&gt;
        &lt;driver name='file' type='raw'/&gt;
        &lt;source file='/path/to/cdrom.iso'/&gt;
        &lt;target dev='hdc' bus='sata'/&gt;
        &lt;readonly/&gt;
      &lt;/disk&gt;
      &lt;interface type='bridge'&gt;
        &lt;model type='virtio'/&gt;
        &lt;source bridge="virbr0"/&gt;
      &lt;/interface&gt;
    &lt;/devices&gt;
&lt;/domain&gt;
</pre>

<p>(The &lt;disk&gt; sections may be swapped in order to install from
<em>cdrom.iso</em>.)</p>

<h3>Example config (Linux guest)</h3>

<p>
Note the addition of &lt;bootloader&gt;.
</p>

<pre>
&lt;domain type='bhyve'&gt;
    &lt;name&gt;linux_guest&lt;/name&gt;
    &lt;uuid&gt;df3be7e7-a104-11e3-aeb0-50e5492bd3dc&lt;/uuid&gt;
    &lt;memory&gt;131072&lt;/memory&gt;
    &lt;currentMemory&gt;131072&lt;/currentMemory&gt;
    &lt;vcpu&gt;1&lt;/vcpu&gt;
    &lt;bootloader&gt;/usr/local/sbin/grub-bhyve&lt;/bootloader&gt;
    &lt;os&gt;
       &lt;type&gt;hvm&lt;/type&gt;
    &lt;/os&gt;
    &lt;features&gt;
      &lt;apic/&gt;
      &lt;acpi/&gt;
    &lt;/features&gt;
    &lt;clock offset='utc'/&gt;
    &lt;on_poweroff&gt;destroy&lt;/on_poweroff&gt;
    &lt;on_reboot&gt;restart&lt;/on_reboot&gt;
    &lt;on_crash&gt;destroy&lt;/on_crash&gt;
    &lt;devices&gt;
      &lt;disk type='file' device='disk'&gt;
        &lt;driver name='file' type='raw'/&gt;
        &lt;source file='/path/to/guest_hdd.img'/&gt;
        &lt;target dev='hda' bus='sata'/&gt;
      &lt;/disk&gt;
      &lt;disk type='file' device='cdrom'&gt;
        &lt;driver name='file' type='raw'/&gt;
        &lt;source file='/path/to/cdrom.iso'/&gt;
        &lt;target dev='hdc' bus='sata'/&gt;
        &lt;readonly/&gt;
      &lt;/disk&gt;
      &lt;interface type='bridge'&gt;
        &lt;model type='virtio'/&gt;
        &lt;source bridge="virbr0"/&gt;
      &lt;/interface&gt;
    &lt;/devices&gt;
&lt;/domain&gt;
</pre>

<h2><a name="usage">Guest usage / management</a></h2>

<h3><a name="console">Connecting to a guest console</a></h3>

<p>
Guest console connection is supported through the <code>nmdm</code> device. It could be enabled by adding
the following to the domain XML (<span class="since">Since 1.2.4</span>):
</p>

<pre>
  ...
  &lt;devices&gt;
    &lt;serial type="nmdm"&gt;
      &lt;source master="/dev/nmdm0A" slave="/dev/nmdm0B"/&gt;
    &lt;/serial&gt;
  &lt;/devices&gt;
  ...</pre>


<p>Make sure to load the <code>nmdm</code> kernel module if you plan to use that.</p>

<p>
Then <code>virsh console</code> command can be used to connect to the text console
of a guest.</p>

<p><b>NB:</b> Some versions of bhyve have a bug that prevents guests from booting
until the console is opened by a client. This bug was fixed in FreeBSD
<a href="http://svnweb.freebsd.org/changeset/base/262884">r262884</a>. If
an older version is used, one either has to open a console manually with <code>virsh console</code>
to let a guest boot or start a guest using:</p>

<pre>start --console domname</pre>

<p><b>NB:</b> An bootloader configured to require user interaction will prevent
the domain from starting (and thus <code>virsh console</code> or <code>start
--console</code> from functioning) until the user interacts with it manually on
the VM host. Because users typically do not have access to the VM host,
interactive bootloaders are unsupported by libvirt. <em>However,</em> if you happen to
run into this scenario and also happen to have access to the Bhyve host
machine, you may select a boot option and allow the domain to finish starting
by using an alternative terminal client on the VM host to connect to the
domain-configured null modem device. One example (assuming
<code>/dev/nmdm0B</code> is configured as the slave end of the domain serial
device) is:</p>

<pre>cu -l /dev/nmdm0B</pre>

<h3><a name="xmltonative">Converting from domain XML to Bhyve args</a></h3>

<p>
The <code>virsh domxml-to-native</code> command can preview the actual
<code>bhyve</code> commands that will be executed for a given domain.
It outputs two lines, the first line is a <code>bhyveload</code> command and
the second is a <code>bhyve</code> command.
</p>

<p>Please note that the <code>virsh domxml-to-native</code> doesn't do any
real actions other than printing the command, for example, it doesn't try to
find a proper TAP interface and create it, like what is done when starting
a domain; and always returns <code>tap0</code> for the network interface. So
if you're going to run these commands manually, most likely you might want to
tweak them.</p>

<pre>
# virsh -c "bhyve:///system"  domxml-to-native --format bhyve-argv --xml /path/to/bhyve.xml
/usr/sbin/bhyveload -m 214 -d /home/user/vm1.img vm1
/usr/sbin/bhyve -c 2 -m 214 -A -I -H -P -s 0:0,hostbridge -s 3:0,virtio-net,tap0,mac=52:54:00:5d:74:e3 -s 2:0,virtio-blk,/home/user/vm1.img -s 1,lpc -l com1,/dev/nmdm0A vm1
</pre>

<h3><a name="zfsvolume">Using ZFS volumes</a></h3>

<p>It's possible to use ZFS volumes as disk devices <span class="since">since 1.2.8</span>.
An example of domain XML device entry for that will look like:</p>

<pre>
  ...
  &lt;disk type='volume' device='disk'&gt;
    &lt;source pool='zfspool' volume='vol1'/&gt;
    &lt;target dev='vdb' bus='virtio'/&gt;
  &lt;/disk&gt;
  ...</pre>

<p>Please refer to the <a href="storage.html">Storage documentation</a> for more details on storage
management.</p>

<h3><a name="grubbhyve">Using grub2-bhyve or Alternative Bootloaders</a></h3>

<p>It's possible to boot non-FreeBSD guests by specifying an explicit
bootloader, e.g. <code>grub-bhyve(1)</code>. Arguments to the bootloader may be
specified as well. If the bootloader is <code>grub-bhyve</code> and arguments
are omitted, libvirt will try and infer boot ordering from user-supplied
&lt;boot order='N'&gt; configuration in the domain. Failing that, it will boot
the first disk in the domain (either <code>cdrom</code>- or
<code>disk</code>-type devices). If the disk type is <code>disk</code>, it will
attempt to boot from the first partition in the disk image.</p>

<pre>
  ...
    &lt;bootloader&gt;/usr/local/sbin/grub-bhyve&lt;/bootloader&gt;
    &lt;bootloader_args&gt;...&lt;/bootloader_args&gt;
  ...
</pre>

<p>Caveat: <code>bootloader_args</code> does not support any quoting.
Filenames, etc, must not have spaces or they will be tokenized incorrectly.</p>

  </body>
</html>
