# Remote support

Libvirt allows you to access hypervisors running on remote machines through authenticated and encrypted connections.

- Basic usage
- Transports
- Remote URIs
  - Extra parameters
- Generating TLS certificates
  - Public Key Infrastructure set up
  - Background to TLS certificates
  - Setting up a Certificate Authority (CA)
  - Issuing server certificates
  - Issuing client certificates
  - Troubleshooting TLS certificate problems
- libvirtd configuration file
- IPv6 support
- Limitations
- Implementation notes

### Basic usage

On the remote machine, `libvirtd` should be running. See the section on configuring libvirtd for more information.

To tell libvirt that you want to access a remote resource, you should supply a hostname in the normal URI that is passed to `virConnectOpen` (or `virsh -c ...`). For example, if you normally use `qemu:///system` to access the system-wide QEMU daemon, then to access the system-wide QEMU daemon on a remote machine called `oirase` you would use `qemu://oirase/system`.

The section on remote URIs describes in more detail these remote URIs.

From an API point of view, apart from the change in URI, the API should behave the same. For example, ordinary calls are routed over the remote connection transparently, and values or errors from the remote side are returned to you as if they happened locally. Some differences you may notice:

- Additional errors can be generated, specifically ones relating to failures in the remote transport itself.
- Remote calls are handled synchronously, so they will be much slower than, say, direct hypervisor calls.

### Transports

Remote libvirt supports a range of transports:

tls
  TLS 1.0 (SSL 3.1) authenticated and encrypted TCP/IP socket, usually listening on a public port number. To use this you will need to generate client and server certificates. The standard port is 16514.
unix
  Unix domain socket. Since this is only accessible on the local machine, it is not encrypted, and uses Unix permissions or SELinux for authentication. The standard socket names are `/var/run/libvirt/libvirt-sock` and `/var/run/libvirt/libvirt-sock-ro` (the latter for read-only connections).
ssh
  Transported over an ordinary ssh (secure shell) connection. Requires Netcat (nc) installed and libvirtd should be running on the remote machine. You should use some sort of ssh key management (eg. ssh-agent) otherwise programs which use this transport will stop to ask for a password.
ext
  Any external program which can make a connection to the remote machine by means outside the scope of libvirt.
tcp
  Unencrypted TCP/IP socket. Not recommended for production use, this is normally disabled, but an administrator can enable it for testing or use over a trusted network. The standard port is 16509.

The default transport, if no other is specified, is `tls`.

### Remote URIs

See also: documentation on ordinary ("local") URIs.

Remote URIs have the general form ("[...]" meaning an optional part):

`driver`[`+transport`]`://`[`username@`][`hostname`][`:port`]`/`[`path`][`?extraparameters`]

Either the transport or the hostname must be given in order to distinguish this from a local URI.

Some examples:

- `xen+ssh://rjones@towada/`
  — Connect to a remote Xen hypervisor on host `towada` using ssh transport and ssh username `rjones`.
- `xen://towada/`
  — Connect to a remote Xen hypervisor on host `towada` using TLS.
- `xen://towada/?no_verify=1`
  — Connect to a remote Xen hypervisor on host `towada` using TLS. Do not verify the server's certificate.
- `qemu+unix:///system?socket=/opt/libvirt/run/libvirt/libvirt-sock`
  — Connect to the local qemu instances over a non-standard Unix socket (the full path to the Unix socket is supplied explicitly in this case).
- `test+tcp://localhost:5000/default`
  — Connect to a libvirtd daemon offering unencrypted TCP/IP connections on localhost port 5000 and use the test driver with default settings.

#### Extra parameters

Extra parameters can be added to remote URIs as part of the query string (the part following `?`). Remote URIs understand the extra parameters shown below. Any others are passed unmodified through to the back end. Note that parameter values must be URI-escaped.

Name | Transports | Meaning
--- | --- | ---
`name` | any transport | The name passed to the remote virConnectOpen function. The name is normally formed by removing transport, hostname, port number, username and extra parameters from the remote URI, but in certain very complex cases it may be better to supply the name explicitly.
| Example: `name=qemu:///system`
`command` | ssh, ext | The external command. For ext transport this is required. For ssh the default is `ssh`. The PATH is searched for the command.
| Example: `command=/opt/openssh/bin/ssh`
`socket` | unix, ssh | The path to the Unix domain socket, which overrides the compiled-in default. For ssh transport, this is passed to the remote netcat command (see next).
| Example: `socket=/opt/libvirt/run/libvirt/libvirt-sock`
`netcat` | ssh | The name of the netcat command on the remote machine. The default is `nc`. For ssh transport, libvirt constructs an ssh command which looks like:
    command -p port [-l username] hostname netcat -U socket
where port, username, hostname can be specified as part of the remote URI, and command, netcat and socket come from extra parameters (or sensible defaults).
| Example: `netcat=/opt/netcat/bin/nc`
`no_verify` | tls | If set to a non-zero value, this disables client checks of the server's certificate. Note that to disable server checks of the client's certificate or IP address you must change the libvirtd configuration.
| Example: `no_verify=1`
`no_tty` | ssh | If set to a non-zero value, this stops ssh from asking for a password if it cannot log in to the remote machine automatically (eg. using ssh-agent etc.). Use this when you don't have access to a terminal - for example in graphical programs which use libvirt.
| Example: `no_tty=1`

### Generating TLS certificates

#### Public Key Infrastructure set up

If you are unsure how to create TLS certificates, skip to the next section.

Location | Machine | Description | Required fields
--- | --- | --- | ---
`/etc/pki/CA/cacert.pem` | Installed on all clients and servers | CA's certificate (more info) | n/a
`/etc/pki/libvirt/ private/serverkey.pem` | Installed on the server | Server's private key (more info) | n/a
`/etc/pki/libvirt/ servercert.pem` | Installed on the server | Server's certificate signed by the CA. (more info) | CommonName (CN) must be the hostname of the server as it is seen by clients.
`/etc/pki/libvirt/ private/clientkey.pem` | Installed on the client | Client's private key. (more info) | n/a
`/etc/pki/libvirt/ clientcert.pem` | Installed on the client | Client's certificate signed by the CA (more info) | Distinguished Name (DN) can be checked against an access control list (`tls_allowed_dn_list`).

#### Background to TLS certificates

Libvirt supports TLS certificates for verifying the identity of the server and clients. There are two distinct checks involved:

- The client should know that it is connecting to the right server. Checking done by client by matching the certificate that the server sends to the server's hostname. May be disabled by adding `?no_verify=1` to the remote URI.
- The server should know that only permitted clients are connecting. This can be done based on client's IP address, or on client's IP address and client's certificate. Checking done by the server. May be enabled and disabled in the libvirtd.conf file.

For full certificate checking you will need to have certificates issued by a recognised Certificate Authority (CA) for your server(s) and all clients. To avoid the expense of getting certificates from a commercial CA, you can set up your own CA and tell your server(s) and clients to trust certificates issues by your own CA. Follow the instructions in the next section.

Be aware that the default configuration for libvirtd allows any client to connect provided they have a valid certificate issued by the CA for their own IP address. You may want to change this to make it less (or more) permissive, depending on your needs.

#### Setting up a Certificate Authority (CA)

You will need the GnuTLS certtool program documented here. In Fedora, it is in the `gnutls-utils` package.

Create a private key for your CA:

    certtool --generate-privkey > cakey.pem

and self-sign it by creating a file with the signature details called `ca.info` containing:

    cn = Name of your organization
    ca
    cert_signing_key

    certtool --generate-self-signed --load-privkey cakey.pem \
      --template ca.info --outfile cacert.pem

(You can delete `ca.info` file now if you want).

Now you have two files which matter:

- `cakey.pem` - Your CA's private key (keep this very secret!)
- `cacert.pem` - Your CA's certificate (this is public).

`cacert.pem` has to be installed on clients and server(s) to let them know that they can trust certificates issued by your CA.

The normal installation directory for `cacert.pem` is `/etc/pki/CA/cacert.pem` on all clients and servers.

To see the contents of this file, do:

    certtool -i --infile cacert.pem

    X.509 certificate info:

    Version: 3
    Serial Number (hex): 00
    Subject: CN=Red Hat Emerging Technologies
    Issuer: CN=Red Hat Emerging Technologies
    Signature Algorithm: RSA-SHA
    Validity:
            Not Before: Mon Jun 18 16:22:18 2007
            Not After: Tue Jun 17 16:22:18 2008
    [etc]

This is all that is required to set up your CA. Keep the CA's private key carefully as you will need it when you come to issue certificates for your clients and servers.

#### Issuing server certificates

For each server (libvirtd) you need to issue a certificate with the X.509 CommonName (CN) field set to the hostname of the server. The CN must match the hostname which clients will be using to connect to the server.

In the example below, clients will be connecting to the server using a URI of `xen://oirase/`, so the CN must be "`oirase`".

Make a private key for the server:

    certtool --generate-privkey > serverkey.pem

and sign that key with the CA's private key by first creating a template file called `server.info` (only the CN field matters, which as explained above must be the server's hostname):

    organization = Name of your organization
    cn = oirase
    tls_www_server
    encryption_key
    signing_key

and sign:

    certtool --generate-certificate --load-privkey serverkey.pem \
      --load-ca-certificate cacert.pem --load-ca-privkey cakey.pem \
      --template server.info --outfile servercert.pem

This gives two files:

- `serverkey.pem` - The server's private key.
- `servercert.pem` - The server's public key.

We can examine this certificate and its signature:

    certtool -i --infile servercert.pem
    X.509 certificate info:

    Version: 3
    Serial Number (hex): 00
    Subject: O=Red Hat Emerging Technologies,CN=oirase
    Issuer: CN=Red Hat Emerging Technologies
    Signature Algorithm: RSA-SHA
    Validity:
            Not Before: Mon Jun 18 16:34:49 2007
            Not After: Tue Jun 17 16:34:49 2008

Note the "Issuer" CN is "Red Hat Emerging Technologies" (the CA) and the "Subject" CN is "oirase" (the server).

Finally we have two files to install:

- `serverkey.pem` is the server's private key which should be copied to the server only as `/etc/pki/libvirt/private/serverkey.pem`.
- `servercert.pem` is the server's certificate which can be installed on the server as `/etc/pki/libvirt/servercert.pem`.

#### Issuing client certificates

For each client (ie. any program linked with libvirt, such as virt-manager) you need to issue a certificate with the X.509 Distinguished Name (DN) set to a suitable name. You can decide this on a company / organisation policy. For example, I use:

    C=GB,ST=London,L=London,O=Red Hat,CN=name_of_client

The process is the same as for setting up the server certificate so here we just briefly cover the steps.

1. Make a private key:

       certtool --generate-privkey > clientkey.pem

2. Act as CA and sign the certificate. Create client.info containing:

       country = GB
       state = London
       locality = London
       organization = Red Hat
       cn = client1
       tls_www_client
       encryption_key
       signing_key

   and sign by doing:

       certtool --generate-certificate --load-privkey clientkey.pem \
         --load-ca-certificate cacert.pem --load-ca-privkey cakey.pem \
         --template client.info --outfile clientcert.pem

3. Install the certificates on the client machine:

       cp clientkey.pem /etc/pki/libvirt/private/clientkey.pem
       cp clientcert.pem /etc/pki/libvirt/clientcert.pem

#### Troubleshooting TLS certificate problems

failed to verify client's certificate

  On the server side, run the libvirtd server with the '--listen' and '--verbose' options while the client is connecting. The verbose log messages should tell you enough to diagnose the problem.

You can use the pki_check.sh shell script to analyze the setup on the client or server machines, preferably as root. It will try to point out the possible problems and provide solutions to fix the set up up to a point where you have secure remote access.

### libvirtd configuration file

Libvirtd (the remote daemon) is configured from a file called `/etc/libvirt/libvirtd.conf`, or specified on the command line using `-f filename` or `--config filename`.

This file should contain lines of the form below. Blank lines and comments beginning with `#` are ignored.

    setting = value

The following settings, values and default are:

Line | Default | Meaning
--- | --- | ---
listen_tls [0|1] | 1 (on) | Listen for secure TLS connections on the public TCP/IP port. Note: it is also necessary to start the server in listening mode by running it with --listen or editing /etc/sysconfig/libvirtd by uncommenting the LIBVIRTD_ARGS="--listen" line to cause the server to come up in listening mode whenever it is started.
listen_tcp [0|1] | 0 (off) | Listen for unencrypted TCP connections on the public TCP/IP port. Note: it is also necessary to start the server in listening mode.
tls_port "service" | "16514" | The port number or service name to listen on for secure TLS connections.
tcp_port "service" | "16509" | The port number or service name to listen on for unencrypted TCP connections.
mdns_adv [0|1] | 1 (advertise with mDNS) | If set to 1 then the virtualization service will be advertised over mDNS to hosts on the local LAN segment.
mdns_name "name" | "Virtualization Host HOSTNAME" | The name to advertise for this host with Avahi mDNS. The default includes the machine's short hostname. This must be unique to the local LAN segment.
unix_sock_group "groupname" | "root" | The UNIX group to own the UNIX domain socket. If the socket permissions allow group access, then applications running under matching group can access the socket. Only valid if running as root
unix_sock_ro_perms "octal-perms" | "0777" | The permissions for the UNIX domain socket for read-only client connections. The default allows any user to monitor domains.
unix_sock_rw_perms "octal-perms" | "0700" | The permissions for the UNIX domain socket for read-write client connections. The default allows only root to manage domains.
tls_no_verify_certificate [0|1] | 0 (certificates are verified) | If set to 1 then if a client certificate check fails, it is not an error.
tls_no_verify_address [0|1] | 0 (addresses are verified) | If set to 1 then if a client IP address check fails, it is not an error.
key_file "filename" | "/etc/pki/libvirt/ private/serverkey.pem" | Change the path used to find the server's private key. If you set this to an empty string, then no private key is loaded.
cert_file "filename" | "/etc/pki/libvirt/ servercert.pem" | Change the path used to find the server's certificate. If you set this to an empty string, then no certificate is loaded.
ca_file "filename" | "/etc/pki/CA/cacert.pem" | Change the path used to find the trusted CA certificate. If you set this to an empty string, then no trusted CA certificate is loaded.
crl_file "filename" | (no CRL file is used) | Change the path used to find the CA certificate revocation list (CRL) file. If you set this to an empty string, then no CRL is loaded.
tls_allowed_dn_list ["DN1", "DN2"] | (none - DNs are not checked) | Enable an access control list of client certificate Distinguished Names (DNs) which can connect to the TLS port on this server. The default is that DNs are not checked. This list may contain wildcards such as `"C=GB,ST=London,L=London,O=Red Hat,CN=*"` See the POSIX `fnmatch` function for the format of the wildcards. Note that if this is an empty list, no client can connect. Note also that GnuTLS returns DNs without spaces after commas between the fields (and this is what we check against), but the `openssl x509` tool shows spaces.

### IPv6 support

The libvirtd service and libvirt remote client driver both use the `getaddrinfo()` functions for name resolution and are thus fully IPv6 enabled. ie, if a server has IPv6 address configured the daemon will listen for incoming connections on both IPv4 and IPv6 protocols. If a client has an IPv6 address configured and the DNS address resolved for a service is reachable over IPv6, then an IPv6 connection will be made, otherwise IPv4 will be used. In summary it should just 'do the right thing(tm)'.

### Limitations

- Remote storage: To be fully useful, particularly for creating new domains, it should be possible to enumerate and provision storage on the remote machine. This is currently in the design phase.
- Migration: We expect libvirt will support migration, and obviously remote support is what makes migration worthwhile. This is also in the design phase. Issues to discuss include which path the migration data should follow (eg. client to client direct, or client to server to client) and security.
- Fine-grained authentication: libvirt in general, but in particular the remote case should support more fine-grained authentication for operations, rather than just read-write/read-only as at present.

Please come and discuss these issues and more on the mailing list.

### Implementation notes

The current implementation uses XDR-encoded packets with a simple remote procedure call implementation which also supports asynchronous messaging and asynchronous and out-of-order replies, although these latter features are not used at the moment.

The implementation should be considered strictly internal to libvirt and subject to change at any time without notice. If you wish to talk to libvirtd, link to libvirt. If there is a problem that means you think you need to use the protocol directly, please first discuss this on the mailing list.

The messaging protocol is described in `qemud/remote_protocol.x`.

Authentication and encryption (for TLS) is done using GnuTLS and the RPC protocol is unaware of this layer.

Protocol messages are sent using a simple 32 bit length word (encoded XDR int) followed by the message header (XDR `remote_message_header`) followed by the message body. The length count includes the length word itself, and is measured in bytes. Maximum message size is `REMOTE_MESSAGE_MAX` and to avoid denial of services attacks on the XDR decoders strings are individually limited to `REMOTE_STRING_MAX` bytes. In the TLS case, messages may be split over TLS records, but a TLS record cannot contain parts of more than one message. In the common RPC case a single `REMOTE_CALL` message is sent from client to server, and the server then replies synchronously with a single `REMOTE_REPLY` message, but other forms of messaging are also possible.

The protocol contains support for multiple program types and protocol versioning, modelled after SunRPC.