Commit Graph

317 Commits

Author SHA1 Message Date
Alex Bennée 21d71c8c41 tests/tcg: enable building for m68k
As before, using Debian SID compilers.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-06-20 20:22:34 +01:00
Alex Bennée 3ae8a1000b tests/tcg: enable building for HPPA
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-06-20 20:22:34 +01:00
Alex Bennée 409c1c9cbd tests/tcg: enable building for Alpha
We can't use our normal Debian based compilers as Alpha isn't an
officially supported architecture. However it is available as a port
and fortunately cross compilers for all these targets are included in
Debian Sid, the perpetual rolling/unstable/testing version of Debian.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-06-20 20:22:34 +01:00
Alex Bennée 6bf77518dd tests/docker/Makefile.include: fix mipsel-cross dependancy
This got broken in commit 4319db7 but generally only shows up when you
try and do massive parallel builds on fresh machines.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-06-20 20:22:34 +01:00
Alex Bennée e27cae268f docker: Makefile.include introduce DOCKER_SCRIPT
Define this in one place to make it easy to re-use.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-06-20 20:22:34 +01:00
Alex Bennée 50b7273854 docker: allow "cc" command to run in user context
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-06-20 20:22:34 +01:00
Alex Bennée 99cfdb8662 docker: extend "cc" command to accept compiler
When calling our cross-compilation images we want to call something
other than the default cc.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Fam Zheng <famz@redhat.com>
2018-06-20 20:22:34 +01:00
Alex Bennée 5e03c2d816 docker: Add "cc" subcommand
Signed-off-by: Fam Zheng <famz@redhat.com>
[AJB: add if args.paths check]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-06-20 20:22:34 +01:00
Alex Bennée 7a5d936b6f docker: docker.py wrap StringIO import for python3
Although the docker.py is nominally python2 we actually invoke it with
the configured python from the configure script.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2018-06-20 16:57:34 +01:00
Eduardo Habkost f03868bd56 python: futurize -f libfuturize.fixes.fix_print_with_import
Change all Python code to use print as a function.

This is necessary for Python 3 compatibility.

Done using:

  $ py=$( (g grep -l -E '^#!.*python';find -name '*.py' -printf '%P\n';) | \
    sort -u | grep -v README.sh4)
  $ futurize -w -f libfuturize.fixes.fix_print_with_import $py

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Acked-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Message-Id: <20180608122952.2009-2-ehabkost@redhat.com>
[ehabkost: fixup tests/docker/docker.py]
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
2018-06-08 14:39:24 -03:00
Philippe Mathieu-Daudé 36dc5fedab docker: add debian/tricore image
TriCore binutils is built from Bastian Koppelmann repository.

Note: There is no TriCore compiler in this image (only assembler/linker).

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
[AJB: base of Debian9, add to Makefile.include]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2018-06-05 16:25:43 +01:00
Paolo Bonzini cabd358407 docker: test-mingw: use SDL2 and GTK+3
Do not test the deprecated API versions.  debian-win32-cross and debian-win64-cross
are already using SDL2 (they do not cover GTK+ at all).

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
[AJB: fix merge conflicts]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2018-06-05 16:25:42 +01:00
Fam Zheng 8bce19d3a6 docker: Add fedora-i386-cross image
It has some basic *-devel.i686 packages to be used with "gcc -m32" as a
32 bit cross build environment.

Signed-off-by: Fam Zheng <famz@redhat.com>
[AJB: add glibc-static]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-06-05 16:25:42 +01:00
Alex Bennée 15df9d3783 docker: add "probe" command for configure
This is a helper function for the configure script. It replies yes,
sudo or no to inform the user if non-interactive docker support is
available. We trap the Exception to fail gracefully.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Fam Zheng <famz@redhat.com>
2018-06-05 16:25:42 +01:00
Fam Zheng 63dba573fa docker: Update fedora image to 28
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20180601023557.9770-1-famz@redhat.com>
2018-06-04 14:39:18 +08:00
Alex Bennée 5609c5127d tests/docker/Makefile.include: handle empty TARGET_LIST
If the user doesn't specify a TARGET_LIST they get the current
configuration but with spaces and hilarity ensues. This adds some make
magic to turn the TARGET_LIST back into a comma separated list.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

Message-Id: <20180521103504.26432-1-alex.bennee@linaro.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2018-06-04 14:39:18 +08:00
Alex Bennée 67659ab1eb docker: update Travis docker image
This is still poorly documented by Travis but according to:

  https://docs.travis-ci.com/user/common-build-problems/#Running-a-Container-Based-Docker-Image-Locally

their reference images are now hosted on Docker Hub. So we update the
FROM line to refer to the new default image. We also need a few
additional tweaks:

  - re-enable deb-src lines for our build-dep install
  - add explicit PATH definition for tools
  - force the build USER to be Travis
  - add clang to FEATURES for our test-clang machinery

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-06-01 15:00:54 +01:00
Philippe Mathieu-Daudé bcaf457786 docker: do not display deprecated images in 'make docker' help
the 'debian' base image is deprecated since 3e11974988

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2018-06-01 14:55:03 +01:00
Philippe Mathieu-Daudé 1a06f5c910 docker: sort images list displayed by 'make docker'
we can now directly see different version sort consecutively.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2018-06-01 14:55:03 +01:00
Philippe Mathieu-Daudé 9d5e546af0 docker: Fix trivial typo
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20180508144358.13530-1-f4bug@amsat.org>
Reviewed-by: Laurent Vivier <laurent@vivier.eu>
Signed-off-by: Fam Zheng <famz@redhat.com>
2018-05-18 16:52:53 +08:00
Paolo Bonzini 07fed47417 docker: fedora: test more components
Install optional dependencies of QEMU to get better coverage.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <1520942752-19449-1-git-send-email-pbonzini@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2018-04-09 15:13:02 +08:00
Fam Zheng 185809224a docker: Inline "prep_fail" in run script
We don't source common.rc where prep_fail is defined, so spell out the
commands and do what was intended.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20180326090350.30014-1-famz@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2018-04-09 15:13:02 +08:00
Philippe Mathieu-Daudé 0c153b4c6f docker: dump 'config.log' if ./configure fails
Suggested-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20180315142713.30960-1-f4bug@amsat.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2018-04-09 15:13:02 +08:00
Marc-André Lureau 02f769b7ee tests: make docker-test-debug@fedora run sanitizers
Since --enable-debug no longer enable sanitizers, we need explicit
--enable-sanitizers.

llvm package is required for llvm-symbolizer, to get symbols in
backtraces.

Add make V=1 to get details about failing tests.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180312120849.20073-1-marcandre.lureau@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2018-03-13 01:19:56 +08:00
Paolo Bonzini 5b9b49d7bd docker: change Fedora base image to fedora:27
Using "fedora:latest" makes behavior different depending on when you
actually pulled the image from the docker repository.  In my case,
the supposedly "latest" image was a Fedora 25 download from 8 months
ago, and the new "test-debug" test was failing.

Use "27" to improve reproducibility and make it clear when the image
is obsolete.

Cc: Fam Zheng <famz@redhat.com>
Cc: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <1515755504-21341-1-git-send-email-pbonzini@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2018-02-08 09:21:34 +08:00
Daniel P. Berrange 356dc290f0 docker: change Fedora images to run with python3
Fedora has switched to Python 3 by default, so it makes sense to use that
for testing QEMU builds, so we get testing of Python 3 compatibility.

Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Message-Id: <20180116134217.8725-15-berrange@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
2018-02-05 19:53:55 -02:00
Marc-André Lureau 11545663d1 tests/docker: add test-debug
Add a new test with --enable-debug using clang/asan/ubsan, remove
--enable-debug from test-clang & test-mingw.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180104160523.22995-7-marcandre.lureau@redhat.com>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2018-01-12 13:22:02 +01:00
Marc-André Lureau c08d08b27c tests/docker: add some sanitizers to fedora dockerfile
Build fedora image with ASAN/UBSan support.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180104160523.22995-6-marcandre.lureau@redhat.com>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2018-01-12 13:22:02 +01:00
Fam Zheng dbd73b569b Revert "docker: Enable features explicitly in test-full"
This reverts commit 5e8a7fe673.

It's hard to get all images to have all these packages, the usual
"FEATURES" and "require" mechanism doesn't scale with so many features.
With that change, the test basically only works in ubuntu.

Until a better way comes up, leave the feature enabling to ./configure
detection.

But don't remove the "-e" removal.

Reported-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20171018082002.9406-1-famz@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-12-20 22:01:23 +08:00
Philippe Mathieu-Daudé 6423795efc docker: correctly escape $BACKEND in the help output
In Makefiles the $ must be escaped as $$ in shell uses.

Since 8a2390a4f47:

 $ make docker
     [...]
     NETWORK=1            Enable virtual network interface with default backend.
     NETWORK=ACKEND     Enable virtual network interface with ACKEND.

Once escaped:

 $ make docker
     [...]
     NETWORK=1            Enable virtual network interface with default backend.
     NETWORK=$BACKEND     Enable virtual network interface with $BACKEND.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Fam Zheng <famz@redhat.com>
Message-Id: <20171108024719.8389-1-f4bug@amsat.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-11-08 10:59:42 +08:00
Fam Zheng c1958e9d54 docker: Improved image checksum
When a base image locally defined by QEMU, such as in the debian images,
is updated, the dockerfile checksum mechanism in docker.py still skips
updating the derived image, because it only looks at the literal content
of the dockerfile, without considering changes to the base image.

For example we have a recent fix e58c1f9b35 that fixed
debian-win64-cross by updating its base image, debian8-mxe, but due to
above "feature" of docker.py the image in question is automatically NOT
rebuilt unless you add NOCACHE=1. It is noticed on Shippable:

https://app.shippable.com/github/qemu/qemu/runs/541/2/console

because after the fix is merged, the error still occurs, and the log
shows the container image is, as explained above, not updated.

This is because at the time docker.py was written, there wasn't any
dependencies between QEMU's docker images.

Now improve this to preprocess any "FROM qemu:*" directives in the
dockerfiles while doing checksum, and inline the base image's dockerfile
content, recursively. This ensures any changes on the depended _QEMU_
images are taken into account.

This means for external images that we expect to retrieve from docker
registries, we still do it as before. It is not perfect, because
registry images can get updated too. Technically we could substitute the
image name with its hex ID as obtained with $(docker images $IMAGE
--format="{{.Id}}"), but --format is not supported by RHEL 7, so leave
it for now.

Reported-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20171103131229.4737-1-famz@redhat.com>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-11-08 10:59:21 +08:00
Philippe Mathieu-Daudé e58c1f9b35 docker: add python stdlib dependency (required by keycodemapdb)
Since 927128222b QEMU depends of keycodemapdb, which uses the python 'csv'
module from stdlib to parse keymaps.csv.

Without this package the build fails:

    GEN     ui/input-keymap-linux-to-qcode.c
  Traceback (most recent call last):
    File "ui/keycodemapdb/tools/keymap-gen", line 15, in <module>
      import csv
  ImportError: No module named csv
    GEN     ui/input-keymap-qcode-to-qnum.c
  Traceback (most recent call last):
    File "ui/keycodemapdb/tools/keymap-gen", line 15, in <module>
      import csv
  ImportError: No module named csv
  [...]
    CC      ui/input-keymap.o
  ui/input-keymap.c:8:44: fatal error: ui/input-keymap-linux-to-qcode.c: No such file or directory
  make: *** [ui/input-keymap.o] Error 1

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-11-02 11:46:41 +00:00
Fam Zheng 6a2e11974f docker: Fix PATH for ccache
Before bcd7f06f57 we source /etc/profile
so the PATH included the right paths to ccache binaries. Now we need to
update $PATH explicitly from run script.

Keep the old /usr/lib around just so that in the future, ccache from 32
bit images will just work.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20171018073841.30062-1-famz@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-10-20 14:28:50 +08:00
Greg Kurz 1dc1700a60 docker: fix out-of-tree 'make docker-test-build@debian-powerpc-cross'
Without this patch, it fails with:

IOError: [Errno 2] No such file or directory:
 'tests/docker/dockerfiles/debian-apt-fake.sh'

Signed-off-by: Greg Kurz <groug@kaod.org>
Message-Id: <150841324224.31292.4012464539983692364.stgit@bahia.lan>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-10-20 14:28:50 +08:00
Paolo Bonzini bc739b8e7d docker: allow running from srcdir != builddir build
The new script uses "git submodule", which is picky about being invoked
from the top of the git checkout.  Invoke the script from $(SRC_PATH)
to avoid git's wrath.

Fixes: b7f404201e
Cc: Fam Zheng <famz@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Message-Id: <1508331989-142364-1-git-send-email-pbonzini@redhat.com>
Reviewed-by: Daniel P. Berrange <berrange@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-10-20 14:28:50 +08:00
Peter Xu 08c3f6d57f docker: cleanup temp directory after test
There are temp directories named "docker-src.*" after doing docker
tests.  I don't see much point in keeping that (it only contains the
qemu.tar which is exactly current tree, and the copied "run" file).
Let's remove it after test finished.

CC: "Alex Bennée" <alex.bennee@linaro.org>
CC: Fam Zheng <famz@redhat.com>
CC: "Philippe Mathieu-Daudé" <f4bug@amsat.org>
Signed-off-by: Peter Xu <peterx@redhat.com>
Message-Id: <20171017071247.32355-1-peterx@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-10-20 14:28:50 +08:00
Fam Zheng 4b2c6bc563 docker: Don't allocate tty unless DEBUG=1
The existence of tty in the container seems to urge gcc into colorizing
the errors, but the escape chars will clutter the report once turned
into email replies on patchew. Move -t to debug mode.

Reported-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20171013011954.9975-1-famz@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-10-20 14:28:50 +08:00
Fam Zheng 36ac78e65a docker: Don't mount ccache db if NOUSER=1
With NOUSER=1 the container runs code as root, which may create
privileged files that will not be be accssible next time. Skip ccache
dir mount in this case.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170925075458.18047-1-famz@redhat.com>
Acked-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
2017-09-29 11:14:15 +08:00
Fam Zheng 12b25a7d96 docker: test-block: Don't continue if build fails
Report error and exit upon compiling error, otherwise the iotests output
will be pure noise.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170926110134.2786-1-famz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-09-29 11:14:15 +08:00
Alex Bennée bcd7f06f57 tests/docker/run: don't source /etc/profile
The usual behaviour of /etc/profile is to set the default PATH for
users. This runs into problems when we have updated PATH in our
dockerfile e.g. to access a cross-compiler in a non-standard
location. It shouldn't be needed anyway as we inherit the env from the
image when it was setup.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
CC: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20170926133622.14991-1-alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-09-29 11:14:15 +08:00
Fam Zheng 299d296ea9 docker: Fix test-mingw
Feature "dtc" is explicitly required by test-mingw, but is not detected
by the run script since we switched to archive-source.sh in b7f404201e.
Since it isn't available in the Fedora image which runs this test on
patchew, the way we get dtc is still from submodule.

archive-source.sh takes care of bundling the submodule files already, so
what we need to do is just checking if files are there. Makefile is
chosen because it is one that is unlikely to get renamed in the future.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170925082913.22089-1-famz@redhat.com>
Reviewed-by: Alistair Francis <alistair.francis@xilinx.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-09-29 11:14:14 +08:00
Paolo Bonzini 6283847857 docker: add installation to build tests
Basic test that "make install" works; this requires msgfmt so add
gettext to the packages.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <1506095371-23160-1-git-send-email-pbonzini@redhat.com>
[Rebase to master. - Fam]
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-09-29 11:14:14 +08:00
Fam Zheng 4f6afe41f2 docker: Drop 'set -e' from run script
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-09-22 14:51:43 +08:00
Fam Zheng b7f404201e docker: Use archive-source.py
Signed-off-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
2017-09-22 14:51:43 +08:00
Fam Zheng f300ca63c7 docker: Add test-block
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170905025614.579-6-famz@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Based-on: 20170905021201.25684-1-famz@redhat.com
2017-09-22 10:20:34 +08:00
Fam Zheng 18d4e35f93 docker: Add nettle-devel to fedora image
The LUKS cases in qemu-iotests requires this.

Reviewed-by: Kashyap Chamarthy <kchamart@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170905025614.579-5-famz@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Based-on: 20170905021201.25684-1-famz@redhat.com
2017-09-22 10:20:34 +08:00
Fam Zheng 4470749186 docker: Use unconfined security profile
Some by default blocked syscalls are required to run tests for example
userfaultfd.

Reviewed-by: Kashyap Chamarthy <kchamart@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170905025614.579-4-famz@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Based-on: 20170905021201.25684-1-famz@redhat.com
2017-09-22 10:20:34 +08:00
Fam Zheng 82659e844a docker: Add test_fail and prep_fail
They both print a message and exit, but with different status code so
distinguish real test errors from env preparation failures.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170905025614.579-3-famz@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Based-on: 20170905021201.25684-1-famz@redhat.com
2017-09-22 10:20:34 +08:00
Fam Zheng d8a2f5116d docker: Fix return code of build_qemu()
Without "set -e", the "&&" makes sure that the return code reflects the
result status, and that make only runs if configure succeeds.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170905025614.579-2-famz@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Based-on: 20170905021201.25684-1-famz@redhat.com
2017-09-22 10:20:34 +08:00
Fam Zheng 05790dafef tests/docker: Clean up paths
The 'run' script already creats src, build and install directories under
$TEST_DIR, use it in common.rc.

Also the tests always run from $QEMU_SRC/tests/docker, so use a relative
$CMD string.

Message-Id: <20170817035721.11064-1-famz@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-09-22 10:20:34 +08:00
Fam Zheng 5e8a7fe673 docker: Enable features explicitly in test-full
Also avoid "set -e".

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170907141245.31946-3-famz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2017-09-22 10:20:34 +08:00
Fam Zheng 7fc581c295 docker: Update ubuntu image
Base on the newer ubuntu-lts (16.06) and include more packages for
better build coverage.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170907141245.31946-2-famz@redhat.com>
2017-09-22 10:20:34 +08:00
Alex Bennée 3f2ff267af docker: reduce noise when building travis.docker
Set the DEBIAN_FRONTEND and locale env vars to stop apt complaining so
much as we build the image.

Suggested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20170725133425.436-7-alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-09-22 10:20:34 +08:00
Alex Bennée 9b4154a570 docker: don't install device-tree-compiler build-deps in travis.docker
Installing the device-tree-compiler build-deps is a little extreme. We
only actually need the binary so include it with the other packages.

Suggested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20170725133425.436-6-alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-09-22 10:20:34 +08:00
Alex Bennée 6fe3ae3f19 docker: docker.py make --no-cache skip checksum test
If you invoke with NOCACHE=1 we pass --no-cache in the argv to
docker.py but may still not force a rebuild if the dockerfile checksum
hasn't changed. By testing for its presence we can force builds
without having to manually remove the docker image.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20170725133425.436-5-alex.bennee@linaro.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-09-22 10:20:34 +08:00
Alex Bennée 1fddbf7c5e docker: ensure NOUSER for travis images
While adding the current user is a useful default behaviour for
creating new images it is not appropriate for Travis which already has
a default user.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20170725133425.436-2-alex.bennee@linaro.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-09-22 10:20:34 +08:00
Paolo Bonzini 7437866bfc docker: fix creation of archives
The pixman submodule does not exist anymore, and its removal broke
docker-based tests.  Fix it.

Cc: Fam Zheng <famz@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2017-09-19 16:20:49 +02:00
Stefan Hajnoczi c2d3189667 docker.py: Python 2.6 argparse compatibility
Add the scripts/ directory to sys.path so Python 2.6 will be able to
import argparse.

Cc: Fam Zheng <famz@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Acked-by: John Snow <jsnow@redhat.com>
Acked-by: Fam Zheng <famz@redhat.com>
Message-id: 20170825155732.15665-3-stefanha@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
2017-08-30 12:02:11 +01:00
Philippe Mathieu-Daudé a8132a2f28 docker: add centos7 image
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20170728233316.13352-5-f4bug@amsat.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-08-15 15:16:45 +08:00
Philippe Mathieu-Daudé 06f3c7b852 docker: install more packages on CentOS to extend code coverage
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20170728233316.13352-4-f4bug@amsat.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-08-15 15:16:45 +08:00
Philippe Mathieu-Daudé 18a2b7affc docker: add Xen libs to centos6 image
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20170728233316.13352-3-f4bug@amsat.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-08-15 15:16:45 +08:00
Philippe Mathieu-Daudé 6aef2ad70b docker: use one package per line in CentOS config
This ease rebase/cherry-pick, also it is faster to visually find if a package
is here.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20170728233316.13352-2-f4bug@amsat.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-08-15 15:16:45 +08:00
Philippe Mathieu-Daudé 118d1becc0 docker: install clang since Shippable setup_ve() verify it is available
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-07-18 10:58:36 +01:00
Philippe Mathieu-Daudé 3e11974988 docker: warn users to use newer debian8/debian9 base image
to stay backward incompatible.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-07-18 10:58:36 +01:00
Philippe Mathieu-Daudé f847325106 docker: add debian Ports base image
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-07-18 10:58:36 +01:00
Philippe Mathieu-Daudé 982e758af7 docker: add MXE (M cross environment) base image for MinGW-w64
see http://mxe.cc/

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-07-18 10:58:36 +01:00
Philippe Mathieu-Daudé fd7b284d79 docker: add debian/mips64el image
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-07-18 10:58:36 +01:00
Philippe Mathieu-Daudé 4319db7b13 docker: add debian/mips[eb] images
change image mips little -> big endian

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-07-18 10:58:35 +01:00
Alex Bennée 16dcece4d5 docker: add debian/powerpc based on Jessie
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
[AJB: split from merged patch]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-07-18 10:58:35 +01:00
Philippe Mathieu-Daudé bec45514e5 docker: add 'apt-fake' script which generate fake debian packages
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-07-18 10:58:35 +01:00
Philippe Mathieu-Daudé a3cf6b1961 docker: add qemu:debian-jessie based on outdated jessie release
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-07-18 10:58:35 +01:00
Philippe Mathieu-Daudé c705b8617b docker: enable nettle to extend code coverage on arm64
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-07-18 10:54:30 +01:00
Philippe Mathieu-Daudé 6f8bbb374b docker: enable gcrypt to extend code coverage on amd64
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-07-18 10:54:30 +01:00
Philippe Mathieu-Daudé 0cf01f5b4a docker: enable netmap to extend code coverage on amd64
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-07-18 10:54:30 +01:00
Philippe Mathieu-Daudé b2e7717425 docker: enable virgl to extend code coverage on amd64
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
[AJB: mv, comments in dockerfile]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-07-18 10:54:30 +01:00
Philippe Mathieu-Daudé 15288046e0 docker: add debian/amd64 based on Stretch
By itself this doesn't add much to our coverage. However later patches
will extend this image to include more bleeding edge libraries which
are not yet widely available in distros.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
[AJB: extend commit msg]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-07-18 10:54:30 +01:00
Philippe Mathieu-Daudé 2b1c10bdcf docker: add debian/ppc64el based on Stretch
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-07-18 10:54:30 +01:00
Philippe Mathieu-Daudé 31e205d1bf docker: add debian/armel based on Stretch
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-07-18 10:54:30 +01:00
Philippe Mathieu-Daudé aaa6ccc9ff docker: optimize debian9 base image
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-07-18 10:54:30 +01:00
Philippe Mathieu-Daudé 2a444f55eb docker: remove packages now dependent of qemu in Stretch
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-07-18 10:54:30 +01:00
Philippe Mathieu-Daudé 7d2bcf2363 docker: debian/s390x no more in unstable, now available in Stretch
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-07-18 10:54:30 +01:00
Philippe Mathieu-Daudé 94afe49eab docker: add common packages to debian base
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-07-18 10:54:30 +01:00
Philippe Mathieu-Daudé 24044d4825 docker: rename debian stable -> 9 (Stretch)
We'll also want to support some older Debian combinations for
architectures that didn't make the Debian 9 cut.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
[AJB: extend commit msg]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-07-18 10:54:30 +01:00
Alex Bennée ed7023b05b docker: add debug tools to travis.docker
When a test fails/hangs you don't want the hassle of getting the debug
tools installed. Lets install them on our image by default so we can
debug when we need to.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-07-18 10:54:23 +01:00
Alex Bennée dca34e8c4e docker: include python-yaml in travis.docker
Although the upstream Travis images don't need this library our
"travis-lite" scripts are written in python. This allows us to do:

  make docker-travis@travis J=10

and approximate a travis run on their default image.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-07-18 10:54:13 +01:00
Paolo Bonzini 6ca9f7fedb docker: allow customizing Travis global_env variables
This is useful so that we can do builds at higher than -j3 when running
travis.py locally.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-07-18 09:39:19 +01:00
Fam Zheng 0b95ff72cb docker.py: Improve subprocess exit code handling
A few error handlings are missing because we ignore the subprocess exit
code, for example "docker build" errors are currently ignored.

Introduce _do_check() aside the existing _do() method and use it in a
few places.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170712075528.22770-3-famz@redhat.com>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-07-17 11:34:20 +08:00
Fam Zheng 58bf7b6d8c docker.py: Drop infile parameter
The **kwargs can do this just well.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170712075528.22770-2-famz@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-07-17 11:34:20 +08:00
Daniel P. Berrange 8a2390a4f4 docker: Don't enable networking as a side-effect of DEBUG=1
When trying to debug problems with tests it is natural to set
DEBUG=1 when starting the docker environment. Unfortunately
this has a side-effect of enabling an eth0 network interface
in the container, which changes the operating environment of
the test suite. IOW tests with fail may suddenly start
working again if DEBUG=1 is set, due to changed network setup.

Add a separate NETWORK variable to allow enablement of
networking separately from DEBUG=1. This can be used in two
ways. To enable the default docker network backend

  make docker-test-build@fedora NETWORK=1

while to enable a specific network backend, eg join the network
associated with the container 'wibble':

  make docker-test-build@fedora NETWORK=container:wibble

Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Message-Id: <20170713144352.2212-1-berrange@redhat.com>
[Drop the superfluous second $(subst ...). - Fam]
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-07-17 11:34:20 +08:00
Philippe Mathieu-Daudé 2e1d6bdcce docker: add mipsel build target
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
[AJB: remove apt-fake kludge]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-06-21 15:02:43 +01:00
Philippe Mathieu-Daudé c9c06eb832 docker: add extra libs to s390x target to extend codebase coverage
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-06-21 15:01:48 +01:00
Philippe Mathieu-Daudé 905bf0ee8a docker: add extra libs to arm64 target to extend codebase coverage
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-06-21 15:01:48 +01:00
Philippe Mathieu-Daudé 32809e7f7b docker: add extra libs to armhf target to extend codebase coverage
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-06-21 15:01:48 +01:00
Philippe Mathieu-Daudé 8a98bfc6e3 docker: use eatmydata in debian arm64 image
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-06-21 15:01:48 +01:00
Philippe Mathieu-Daudé 8a48be0e87 docker: use eatmydata in debian armhf image
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-06-21 15:01:48 +01:00
Philippe Mathieu-Daudé 96e659d006 docker: use eatmydata, install common build packages in base image
The common build packages are: build-essential clang git bison flex

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
[AJB: fixups following stretch update]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-06-21 15:01:48 +01:00
Philippe Mathieu-Daudé cf80eb8d09 docker: use better regex to generate deb-src entries
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
[AJB: fixed up following dropping emdebian]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-06-21 15:01:48 +01:00
Philippe Mathieu-Daudé 2c1c31ed55 docker: install ca-certificates package in base image
Resolve SSL verification issue at shippable container's git_sync stage:

shippable logs:
--------------
git_sync
- ssh-agent bash -c 'ssh-add /tmp/ssh/01_deploy; git clone https://github.com/philmd/qemu.git /root/src/github.com/philmd/qemu'
Identity added: /tmp/ssh/01_deploy (rsa w/o comment)
Cloning into '/root/src/github.com/philmd/qemu'...
fatal: unable to access 'https://github.com/philmd/qemu.git/': Problem with the SSL CA cert (path? access rights?)
retrying 1 of 3 times...

Suggested-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
[AJB: fixed re-base conflict following stretch updates]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-06-21 15:01:48 +01:00
Philippe Mathieu-Daudé 438d116872 docker: rebuild image if 'extra files' checksum does not match
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-06-21 15:01:48 +01:00
Philippe Mathieu-Daudé 4c84f662c2 docker: add --include-files argument to 'build' command
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-06-21 15:01:48 +01:00
Philippe Mathieu-Daudé 2499ee9fad docker: let _copy_with_mkdir() sub_path argument be optional
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-06-21 15:01:48 +01:00
Alex Bennée 7af25f9f6a docker: update qemu:debian base following stretch release
Debian has now released Stretch as its new stable. As we track
debian:stable-slim this has a few consequences. For one thing we can
now drop the emdebian hacks as cross compilers are part of the
official repositories now. However we do loose the ability to build
against powerpc (not ppc64) since that is no longer a release
architecture.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-06-21 15:01:48 +01:00
Fam Zheng 79f24568e5 docker: Add flex and bison to centos6 image
Currently there are warnings about flex and bison being missing when
building in the centos6 image:

    make[1]: flex: Command not found
             BISON dtc-parser.tab.c
    make[1]: bison: Command not found

Add them.

Reported-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170524005206.31916-1-famz@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-06-16 07:55:00 +08:00
Fam Zheng 80c58a5b1b docker: Add libaio to fedora image
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170505032340.26467-5-famz@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-06-16 07:55:00 +08:00
Fam Zheng 73a27bbb69 docker: Add bzip2 and hostname to fedora image
It is used by qemu-iotests.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170505032340.26467-3-famz@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-06-16 07:55:00 +08:00
Fam Zheng f9f65a4af0 docker: Run tests with current user
We've used --add-current-user to create a user in the image, use it to
run tests, because root has too much priviledge, and can surprise test
cases.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170505032340.26467-2-famz@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-06-16 07:55:00 +08:00
Alex Bennée 8ba1e5f72b docker/dockerfiles/debian-s390-cross: include clang
It's a silly little limitation on Shippable that is looks for clang
in the container even though we won't use it. The arm/aarch64 cross
builds inherit this from debian.docker but as we needed to use
debian-testing for this we add it here. We also collapse the update
step into one RUN line to remove and intermediate layer of the docker
build.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20170306112848.659-1-alex.bennee@linaro.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-03-10 15:05:22 +08:00
Philippe Mathieu-Daudé 06cc355171 tests/docker: support proxy / corporate firewall
if ftp_proxy/http_proxy/https_proxy standard environment variables available,
pass them to the docker daemon to build images.
this is required when building behind corporate proxy/firewall, but also help
when using local cache server (ie: apt/yum).

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20170306205520.32311-1-f4bug@amsat.org>
Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-03-07 18:20:40 +08:00
Alex Bennée 267004d991 new: dockerfiles/debian-s390-cross
This adds an s390 cross build target to our library of docker setups.
There is an issue with the xfslibs-dev:s390x package having a clash so
we do a || apt-get -f install to fixup the rest of the dependencies.

This doesn't build on the debian.docker file as we are using the
multilib compiler which is only available in stretch (the current
testing repo).

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
CC: Christian Borntraeger <borntraeger@de.ibm.com>
Message-Id: <20170227143028.16428-2-alex.bennee@linaro.org>
Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-02-28 20:31:01 +08:00
Fam Zheng a8f159d45b docker: Install python2 explicitly in docker image
Python is no longer installed implicitly, but the QEMU build system
requires it. List it in PACKAGES.

Reported-by: Auger Eric <eric.auger@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170222021801.28658-1-famz@redhat.com>
Tested-by: Eric Auger <eric.auger@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-02-24 14:18:11 +08:00
Alex Bennée 24e0131f37 new: debian docker targets for cross-compiling
This provides a basic Debian install with access to the emdebian cross
compilers. The debian-armhf-cross and debian-arm64-cross targets build
on the basic Debian image to allow cross compiling to those targets.

A new environment variable (QEMU_CONFIGURE_OPTS) is set as part of the
docker container and passed to the build to specify the
--cross-prefix. The user still calls the build in the usual way, for
example:

  make docker-test-build@debian-arm64-cross \
    TARGET_LIST="aarch64-softmmu,aarch64-linux-user"

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20170220105139.21581-3-alex.bennee@linaro.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-02-24 14:18:11 +08:00
Alex Bennée 414a8ce57e tests/docker: add basic user mapping support
Currently all docker builds are done by exporting a tarball to the
docker container and running the build as the containers root user.
Other use cases are possible however and it is possible to map a part
of users file-system to the container. This is useful for example for
doing cross-builds of arbitrary source trees. For this to work
smoothly the container needs to have a user created that maps cleanly
to the host system.

This adds a -u option to the docker script so that:

  DEB_ARCH=armhf DEB_TYPE=stable ./tests/docker/docker.py build \
    -u --include-executable=arm-linux-user/qemu-arm \
    debian:armhf ./tests/docker/dockerfiles/debian-bootstrap.docker

Will build a container that can then be run like:

  docker run --rm -it -v /home/alex/lsrc/qemu/risu.git/:/src \
    --user=alex:alex -w /src/ debian:armhf \
    sh -c "make clean && ./configure -s && make"

All docker containers built will add the current user unless
explicitly disabled by specifying NOUSER when invoking the Makefile:

  make docker-image-debian-armhf-cross NOUSER=1

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Fam Zheng <famz@redhat.com>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20170220105139.21581-2-alex.bennee@linaro.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-02-24 14:18:11 +08:00
Alex Bennée e86c9a64f4 tests/docker/Makefile.include: add a generic docker-run target
This re-factors the docker makefile to include a docker-run target which
can be controlled entirely from environment variables specified on the
make command line. This allows us to run against any given docker image
we may have in our repository, for example:

    make docker-run TEST="test-quick" IMAGE="debian:arm64" \
         EXECUTABLE=./aarch64-linux-user/qemu-aarch64

The existing docker-foo@bar targets still work but the inline
verification has been dropped because we already don't hit that due to
other pattern rules in rules.mak.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

Message-Id: <20161011161625.9070-5-alex.bennee@linaro.org>
Message-Id: <20161011161625.9070-6-alex.bennee@linaro.org>
[Squash in the verification removal patch. - Fam]
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-10-17 10:05:48 +08:00
Alex Bennée 86a17cb3f4 tests/docker: make test-mingw honour TARGET_LIST
The other builders honour this variable, so should the mingw build.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20161011161625.9070-4-alex.bennee@linaro.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-10-17 10:05:48 +08:00
Alex Bennée bdecba6e97 tests/docker: test-build script
Much like test-quick but only builds. This is useful for some of the
build targets like ThreadSanitizer that don't yet pass "make check".

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

Message-Id: <20161011161625.9070-3-alex.bennee@linaro.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-10-17 10:05:48 +08:00
Alex Bennée 8b9b3177a2 tests/docker: add travis dockerfile
This target grabs the latest Travis containers from their repository at
quay.io and then installs QEMU's build dependencies. With this it is
possible to run on broadly the same setup as they have on travis-ci.org.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20161011161625.9070-2-alex.bennee@linaro.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-10-17 10:05:48 +08:00
Peter Maydell 0bdb12c7c5 rules.mak: quiet-command: Split command name and args to print
The quiet-command make rule currently takes two arguments:
the command and arguments to run, and a string to print if
the V flag is not set (ie we are not being verbose).
By convention, the string printed is of the form
"  NAME   some args". Unfortunately to get nicely lined up
output all the strings have to agree about what column the
arguments should start in, which means that if we add a
new quiet-command usage which wants a slightly longer CMD
name then we either put up with misalignment or change
every quiet-command string.

Split the quiet-mode string into two, the "NAME" and
the "same args" part, and use printf(1) to format the
string automatically. This means we only need to change
one place if we want to support a longer maximum name.

In particular, we can now print 7-character names lined
up properly (they are needed for the OSX "SETTOOL" invocation).

Change all the uses of quiet-command to the new syntax.
(Any which are missed or inadvertently reintroduced
via later merges will result in slightly misformatted
quiet output rather than disaster.)

A few places in the pc-bios/ makefiles are updated to use
"BUILD", "SIGN" and "STRIP" rather than "Building",
"Signing" and "Stripping" for consistency and to keep them
below 7 characters. Module .mo links now print "LD" rather
than the nonstandard "LD -r".

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 1475598441-27908-1-git-send-email-peter.maydell@linaro.org
2016-10-06 12:11:51 +01:00
Fam Zheng c16fe84f07 docker: Build in a clean directory
Currently we configure and build under "$QEMU_SRC/tests/docker" which is
dubious. Create a fixed directory (to be friendly to ccache) and change
to there before calling build_qemu.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <1475047892-11955-1-git-send-email-famz@redhat.com>
2016-10-02 09:47:23 +08:00
Fam Zheng 9b77336d83 docker: exec $CMD
This is the last command to run (unless DEBUG), make it 'exec' to
simplify the process tree.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <1474429768-25027-10-git-send-email-famz@redhat.com>
Reviewed-by: Daniel P. Berrange <berrange@redhat.com>
2016-09-23 11:48:32 +08:00
Fam Zheng 97cba1a1d1 docker: Terminate instances at SIGTERM and SIGHUP
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <1474429768-25027-9-git-send-email-famz@redhat.com>
Reviewed-by: Daniel P. Berrange <berrange@redhat.com>
2016-09-23 11:48:32 +08:00
Fam Zheng ec960b9a73 docker: Support showing environment information
Add a make variable SHOW_ENV. When it's set to non empty, print the
package information and environment variables.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <1474429768-25027-8-git-send-email-famz@redhat.com>
Reviewed-by: Daniel P. Berrange <berrange@redhat.com>
2016-09-23 11:48:32 +08:00
Fam Zheng 9445c28ec3 docker: Print used options before doing configure
This makes the configure command more obvious which usually has useful
information.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <1474429768-25027-7-git-send-email-famz@redhat.com>
Reviewed-by: Daniel P. Berrange <berrange@redhat.com>
2016-09-23 11:48:32 +08:00
Fam Zheng 3d3a6eb60d docker: Flatten default target list in test-quick
Previously it is expanded to a whitespace separated list which is not
the most appropriate format. Since it's only two items, flatten it.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <1474429768-25027-6-git-send-email-famz@redhat.com>
Reviewed-by: Daniel P. Berrange <berrange@redhat.com>
2016-09-23 11:48:32 +08:00
Fam Zheng a2bd7252fe docker: Update fedora image to latest
Now that 23 is becoming an "old" release with 24 available. Fedora has a
quick release cycle, so use latest to follow more closely.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <1474429768-25027-5-git-send-email-famz@redhat.com>
Reviewed-by: Daniel P. Berrange <berrange@redhat.com>
2016-09-23 11:48:32 +08:00
Fam Zheng 5008fc787b docker: Generate /packages.txt in ubuntu image
Put the list of package names in an environment, and output their
package names to the target file in the end.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <1474429768-25027-4-git-send-email-famz@redhat.com>
Reviewed-by: Daniel. P. Berrange <berrange@redhat.com>
2016-09-23 11:48:32 +08:00
Fam Zheng 28a0cccd54 docker: Generate /packages.txt in fedora image
Put the list of package names in an environment, and output their
package names to the target file in the end.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <1474429768-25027-3-git-send-email-famz@redhat.com>
Reviewed-by: Daniel P. Berrange <berrange@redhat.com>
2016-09-23 11:48:32 +08:00
Fam Zheng a34ac8d472 docker: Generate /packages.txt in centos6 image
Put the list of package names in an environment, and output their
package names to the target file in the end.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <1474429768-25027-2-git-send-email-famz@redhat.com>
Reviewed-by: Daniel P. Berrange <berrange@redhat.com>
2016-09-23 11:48:32 +08:00
Eduardo Habkost 0679f98b40 docker: Handle exceptions when looking for docker command
When trying to run docker tests on a host without the docker
command,  we get the following Python backtrace:

  $ make docker-test-quick@centos6 V=1
  .../qemu/tests/docker/docker.py build qemu:centos6 .../qemu/tests/docker/dockerfiles/centos6.docker
  Traceback (most recent call last):
    File ".../qemu/tests/docker/docker.py", line 339, in <module>
      sys.exit(main())
    File ".../qemu/tests/docker/docker.py", line 336, in main
      return args.cmdobj.run(args, argv)
    File ".../qemu/tests/docker/docker.py", line 231, in run
      dkr = Docker()
    File ".../qemu/tests/docker/docker.py", line 98, in __init__
      self._command = _guess_docker_command()
    File ".../qemu/tests/docker/docker.py", line 41, in _guess_docker_command
      stdout=DEVNULL, stderr=DEVNULL) == 0:
    File "/usr/lib64/python2.7/subprocess.py", line 523, in call
      return Popen(*popenargs, **kwargs).wait()
    File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
      errread, errwrite)
    File "/usr/lib64/python2.7/subprocess.py", line 1343, in _execute_child
      raise child_exception
  OSError: [Errno 2] No such file or directory
  .../qemu/tests/docker/Makefile.include:47: recipe for target 'docker-image-centos6' failed
  make: *** [docker-image-centos6] Error 1

Change _guess_docker_command() to handle OSError exceptions
raised by subprocess.call(), so we will keep looking for other
commands and print a better error message.

New output will be:

  $ make docker-test-quick@centos6 V=1
  .../qemu/tests/docker/docker.py build qemu:centos6 .../qemu/tests/docker/dockerfiles/centos6.docker
  Traceback (most recent call last):
    File ".../qemu/tests/docker/docker.py", line 343, in <module>
      sys.exit(main())
    File ".../qemu/tests/docker/docker.py", line 340, in main
      return args.cmdobj.run(args, argv)
    File ".../qemu/tests/docker/docker.py", line 235, in run
      dkr = Docker()
    File ".../qemu/tests/docker/docker.py", line 102, in __init__
      self._command = _guess_docker_command()
    File ".../qemu/tests/docker/docker.py", line 49, in _guess_docker_command
      commands_txt)
  Exception: Cannot find working docker command. Tried:
    docker
    sudo -n docker
  .../qemu/tests/docker/Makefile.include:47: recipe for target 'docker-image-centos6' failed
  make: *** [docker-image-centos6] Error 1

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Message-Id: <1474369559-16903-1-git-send-email-ehabkost@redhat.com>
[exceptions.OSError -> OSError and drop the import. - Fam]
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-09-23 11:38:57 +08:00
Sascha Silbe f8042deafa docker: silence debootstrap when --quiet is given
If we silence docker when --quiet is given, we should also silence the
.pre script (i.e. debootstrap).

Only discards stdout, so some diagnostics (e.g. from git clone) are
still printed. Most of the verbose output is gone however and this way
we still have a chance to see error messages.

Signed-off-by: Sascha Silbe <silbe@linux.vnet.ibm.com>
Message-Id: <1473192351-601-9-git-send-email-silbe@linux.vnet.ibm.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-09-08 19:56:34 +08:00
Sascha Silbe ae2f659ca5 docker: build debootstrap after cloning
When using the git version of debootstrap (because no usable version
of debootstrap was installed on the host), we need to run 'make' so
that devices.tar.gz gets built. Otherwise the first debootstrap stage
will fail without printing any error message.

Signed-off-by: Sascha Silbe <silbe@linux.vnet.ibm.com>
Message-Id: <1473192351-601-8-git-send-email-silbe@linux.vnet.ibm.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-09-08 19:56:34 +08:00
Sascha Silbe 00263139f8 docker: make sure debootstrap is at least 1.0.67
debootstrap prior to 1.0.67 generated an empty sources.list during
foreign bootstraps (Debian#732255 [1]). Fall back to the git checkout
if the installed debootstrap version is too old.

[1] https://bugs.debian.org/732255

Signed-off-by: Sascha Silbe <silbe@linux.vnet.ibm.com>
Message-Id: <1473192351-601-7-git-send-email-silbe@linux.vnet.ibm.com>
[Update 'sort -C' to 'sorc -c &>/dev/null' - Fam]
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-09-08 19:56:34 +08:00
Sascha Silbe a351b4b06e docker: print warning if EXECUTABLE is not set when building debootstrap image
Building the debian-debootstrap image will usually fail if EXECUTABLE
isn't set (when using the Makefile). Warn the user in this case so
they know why it's failing.

Signed-off-by: Sascha Silbe <silbe@linux.vnet.ibm.com>
Message-Id: <1473192351-601-6-git-send-email-silbe@linux.vnet.ibm.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-09-08 19:56:34 +08:00
Sascha Silbe 341edc0c47 docker: debian-bootstrap.pre: print helpful message if DEB_ARCH/DEB_TYPE unset
The debian-bootstrap image doesn't choose a default architecture and
distribution version, instead the user has to set both DEB_ARCH and
DEB_TYPE in the environment. Print a reasonably helpful message if
either of them isn't set instead of complaining about "qemu-" being
missing or erroring out because we cannot cd to the mirror URL.

Signed-off-by: Sascha Silbe <silbe@linux.vnet.ibm.com>
Message-Id: <1473192351-601-5-git-send-email-silbe@linux.vnet.ibm.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-09-08 19:56:34 +08:00
Sascha Silbe b5dc88ce24 docker: debian-bootstrap.pre: print error messages to stderr
Send error messages where they belong so they're seen even if stdout
is redirected to /dev/null.

Signed-off-by: Sascha Silbe <silbe@linux.vnet.ibm.com>
Message-Id: <1473192351-601-4-git-send-email-silbe@linux.vnet.ibm.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-09-08 19:56:34 +08:00
Sascha Silbe 08f4e8d23d docker: avoid dependency on 'realpath' package
The 'realpath' executable is shipped in a separate package that isn't
installed by default on some distros.

We already use 'readlink -e' (provided by GNU coreutils) in some other
part of the code, so let's settle for that instead.

Signed-off-by: Sascha Silbe <silbe@linux.vnet.ibm.com>
Message-Id: <1473192351-601-3-git-send-email-silbe@linux.vnet.ibm.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-09-08 19:56:34 +08:00
Sascha Silbe c977257045 docker.py: don't hang on large docker output
Unlike Popen.communicate(), subprocess.call() doesn't read from the
stdout file descriptor. If the child process produces more output than
fits into the pipe buffer, it will block indefinitely.

If we don't intend to consume the output, just send it straight to
/dev/null to avoid this issue.

Signed-off-by: Sascha Silbe <silbe@linux.vnet.ibm.com>
Reviewed-by: Janosch Frank <frankja@linux.vnet.ibm.com>
Message-Id: <1473192351-601-2-git-send-email-silbe@linux.vnet.ibm.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-09-08 19:56:34 +08:00
Fam Zheng 9af4c174a3 docker: Add a glib2-2.22 image
It's a variation of our existing centos6, plus two more lines to
downgrade glib2 to version 2.22 which we download from vault.centos.org.

Suggested-by: Paolo Bonzini <pbonzoni@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <1470708908-12885-1-git-send-email-famz@redhat.com>
2016-09-08 19:56:34 +08:00
Fam Zheng 4a93f78ed0 docker: Add "--enable-werror" to configure command line
We don't have .git in the docker checkout, add this to enable -Werror
explicitly.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-id: 1469453510-658-1-git-send-email-famz@redhat.com
2016-08-05 16:34:55 +08:00
Fam Zheng 95d203cd1e docker: Be compatible with older docker
By not using "--format" with docker images command.

The option is not available on RHEL 7 docker command. Use an awk
matching command instead.

Reported-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <1470202928-3392-1-git-send-email-famz@redhat.com>
2016-08-05 16:34:52 +08:00
Alex Bennée b7c851b2b8 docker: pass EXECUTABLE to build script
To build a docker image with which needs qemu linux-user emulation we
need to pass --include-executable to the build script. Using the same
mechanism as for other container controls we enable the option is
EXECUTABLE is set on the make command line e.g:

    make docker-image-debian-bootstrap V=1 J=9 DEB_ARCH=armhf \
        DEB_TYPE=stable EXECUTABLE=./arm-linux-user/qemu-arm

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 1468934445-32183-11-git-send-email-famz@redhat.com
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-07-20 19:19:43 +08:00
Fam Zheng ff31e2256d docker: Don't start a container that doesn't exist
Image building targets are dependencies of test running targets, so when
a docker image doesn't exist, it means it's skipped (due to dependency
checks in pre script). Therefore, skip the test too.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-id: 1468934445-32183-10-git-send-email-famz@redhat.com
2016-07-20 19:19:43 +08:00
Fam Zheng 4b08af6019 docker: Add "images" subcommand to docker.py
This is a wrapper for the 'docker images' command.

Signed-off-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 1468934445-32183-9-git-send-email-famz@redhat.com
2016-07-20 19:19:43 +08:00
Fam Zheng 1ad76b8af8 docker: Fix exit code if $CMD failed
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-id: 1468934445-32183-8-git-send-email-famz@redhat.com
2016-07-20 19:19:43 +08:00
Fam Zheng c81585130e docker: More sensible run script
It is very easy to figure out current directory and bash option from the
execution, so do less in the Makefile invocation command line, and
figure both options in the script.

This makes the next patch easier.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-id: 1468934445-32183-7-git-send-email-famz@redhat.com
2016-07-20 19:19:43 +08:00
Alex Bennée 6e733da676 tests/docker/docker.py: add update operation
This adds a new operation to the docker script to allow updating of
binaries in an existing container. This is because it would be
inefficient to re-build the whole container just for an update to the
QEMU binary.

To update the executable run:

    ./tests/docker/docker.py update \
        debian:armhf ./arm-linux-user/qemu-arm

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 1468934445-32183-6-git-send-email-famz@redhat.com
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-07-20 19:19:43 +08:00
Alex Bennée 95c975013a tests/docker/dockerfiles: new debian-bootstrap.docker
Together with the debian-bootstrap.pre script can now build an arbitrary
architecture of Debian using debootstrap. This allows debootstrap to set
up its first stage before the container is built.

To build a container you need a command line like:

  DEB_ARCH=armhf DEB_TYPE=testing \
    ./tests/docker/docker.py build \
    --include-executable=arm-linux-user/qemu-arm debian:armhf \
    ./tests/docker/dockerfiles/debian-bootstrap.docker

Although a number of non-debian systems package the debootstrap script
it is fairly portable in itself. Assuming we have some sort of fakeroot
implementation we can just clone the upstream repository and use the
script from there.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 1468934445-32183-5-git-send-email-famz@redhat.com
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-07-20 19:19:43 +08:00
Alex Bennée 920776ea5e tests/docker/docker.py: check and run .pre script
The docker script will now search for an associated $dockerfile.pre
script which gets run in the same build context as the dockerfile will
be. This is to support pre-seeding the build context before running the
docker build.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 1468934445-32183-4-git-send-email-famz@redhat.com
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-07-20 19:19:43 +08:00
Alex Bennée 504ca3c208 tests/docker/docker.py: support --include-executable
When passed the path to a binary we copy it and any linked libraries (if
it is dynamically linked) into the docker build context. These can then
be included by a dockerfile with the line:

  # Copy all of context into container
  ADD . /

This is mainly intended for setting up foreign architecture docker
images which use qemu-$arch to do cross-architecture linux-user
execution. It also relies on the host and guest file-system following
reasonable multi-arch layouts so the copied libraries don't clash with
the guest ones.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 1468934445-32183-3-git-send-email-famz@redhat.com
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-07-20 19:19:43 +08:00
Alex Bennée a9f8d03891 tests/docker/docker.py: docker_dir outside build
Instead of letting the build_image create the temporary working dir we
move the creation to the build command. This is preparation for the
later patches where additional files can be added to the build context
before the build step is run.

We also ensure we remove the build context after we are done (mkdtemp
doesn't do this automatically for you).

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Fam Zheng <famz@redhat.com>
Message-id: 1468934445-32183-2-git-send-email-famz@redhat.com
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-07-20 19:19:43 +08:00
Paolo Bonzini d3a49cbed5 tests/docker: build all targets in test-clang
Warnings specific to clang may affect devices that are not build by
x86_64-softmmu and aarch64-softmmu.  Build all targets since that
is also what Peter does.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-id: 1465224417-141321-7-git-send-email-pbonzini@redhat.com
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-06-08 15:19:30 +08:00
Paolo Bonzini 78465d74c2 tests/docker: support travis test with fedora image
Install sparse and PyYAML.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-id: 1465224417-141321-6-git-send-email-pbonzini@redhat.com
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-06-08 15:19:30 +08:00
Paolo Bonzini 8080214dc8 tests/docker: remove unused feature "ccache"
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-id: 1465224417-141321-5-git-send-email-pbonzini@redhat.com
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-06-08 15:19:30 +08:00
Paolo Bonzini 2346b12fc5 tests/docker: fix test-mingw
Add flex and bison for use in test-mingw, because test-mingw
uses the in-tree libdtc.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-id: 1465224417-141321-4-git-send-email-pbonzini@redhat.com
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-06-08 15:19:30 +08:00
Paolo Bonzini 53735f0b82 tests/docker: make test-full build all targets, not none
Fix common.rc to avoid passing an empty --target-list= option to configure.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-id: 1465224417-141321-3-git-send-email-pbonzini@redhat.com
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-06-08 15:19:30 +08:00
Paolo Bonzini 34c98c54c3 tests/docker: fix make-archive-maybe
make-archive-maybe expects an archive path relative
to $1, but receives a path relative to the current directory.  Redirect
the output outside the subshell to bypass the "cd $1".

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-id: 1465224417-141321-2-git-send-email-pbonzini@redhat.com
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-06-08 15:19:30 +08:00
Eduardo Habkost d5aebef884 docker: Don't use eval trick on Makefile
The eval trick for defining DOCKER_SRC_COPY doesn't do anything
useful, as DOCKER_SRC_COPY is immediately expanded just after it
is defined, and CUR_TIME is already defined using ":=". Simply
define it using ":=" so it is evaluated only once.

The eval trick was also triggering an weird error on Travis builds:
  qemu/tests/docker/Makefile.include:34: *** unterminated variable reference.  Stop.

The issue is not easily reproducible (maybe it's a bug in some
versions of Make), but it is avoided if removing the eval trick.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2016-06-07 15:00:02 +01:00
Fam Zheng 35e0f959b5 docker: Add EXTRA_CONFIGURE_OPTS
Whatever passed in this variable will be appended to all
configure commands.

Signed-off-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 1464755128-32490-15-git-send-email-famz@redhat.com
2016-06-01 17:27:35 +08:00
Fam Zheng d5bd789198 docker: Add travis tool
The script is not prefixed with test- so it won't run with "make docker-test",
because it can take too long.

Run it with "make docker-travis@ubuntu".

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-id: 1464755128-32490-13-git-send-email-famz@redhat.com
2016-06-01 17:27:35 +08:00
Fam Zheng c4f0eed1f3 docker: Add mingw test
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-id: 1464755128-32490-12-git-send-email-famz@redhat.com
2016-06-01 17:27:35 +08:00
Fam Zheng c8908570dc docker: Add clang test
The (currently partially commented out) configure options are suggested
by John Snow <jsnow@redhat.com>.

Signed-off-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 1464755128-32490-11-git-send-email-famz@redhat.com
2016-06-01 17:27:35 +08:00
Fam Zheng d710ac871c docker: Add full test
This builds all available targets.

Signed-off-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 1464755128-32490-10-git-send-email-famz@redhat.com
2016-06-01 17:27:35 +08:00
Fam Zheng b7899d63c8 docker: Add quick test
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-id: 1464755128-32490-9-git-send-email-famz@redhat.com
2016-06-01 17:27:35 +08:00
Fam Zheng 3568f98ca5 docker: Add common.rc
"requires" checks the "FEATURE" environment for specified prerequisits,
and skip the execution of test if not found.

"build_qemu" is the central routine to compile QEMU for tests to call.

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-id: 1464755128-32490-8-git-send-email-famz@redhat.com
2016-06-01 17:27:35 +08:00
Fam Zheng b344aa9132 docker: Add test runner
It's better to have a launcher for all tests, to make it easier to
initialize and manage the environment.

If "DEBUG=1"  a shell prompt will show up before the test runs.

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-id: 1464755128-32490-7-git-send-email-famz@redhat.com
2016-06-01 17:27:35 +08:00
Fam Zheng ca853f0c76 docker: Add images
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-id: 1464755128-32490-6-git-send-email-famz@redhat.com
2016-06-01 17:27:35 +08:00
Fam Zheng 324027c24c Makefile: Rules for docker testing
This adds a group of make targets to run docker tests, all are available
in source tree without running ./configure.

The usage is shown with "make docker".

Besides the fixed ones, dynamic targets for building each image and
running each test in each image are generated automatically by make,
scanning $(SRC_PATH)/tests/docker/ files with specific patterns.

Alternative to manually list particular targets (docker-TEST@IMAGE)
set, you can control which tests/images to run by filtering variables,
TESTS= and IMAGES=, which are expressed in Makefile pattern syntax,
"foo% %bar ...". For example:

    $ make docker-test IMAGES="ubuntu fedora"

Unfortunately, it's impossible to propagate "-j $JOBS" into make in
containers, however since each combination is made a first class target
in the top Makefile, "make -j$N docker-test" still parallels the tests
coarsely.

Still, $J is made a magic variable to let all make invocations in
containers to use -j$J.

Instead of providing a live version of the source tree to the docker
container we snapshot it with git-archive. This ensures the tree is in a
pristine state for whatever operations the container is going to run on
them.

Uncommitted changes known to files known by the git index will be
included in the snapshot if there are any.

Signed-off-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 1464755128-32490-5-git-send-email-famz@redhat.com
2016-06-01 17:27:34 +08:00
Fam Zheng 4485b04be9 tests: Add utilities for docker testing
docker.py is added with a number of useful subcommands to manager docker
images and instances for QEMU docker testing. Subcommands are:

run: A wrapper of "docker run" (or "sudo -n docker run" if necessary),
which takes care of killing and removing the running container at
SIGINT.

clean: Tear down all the containers including inactive ones that are
started by docker_run.

build: Compare an image from given dockerfile and rebuild it if they're
different.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-id: 1464755128-32490-2-git-send-email-famz@redhat.com
2016-06-01 17:25:50 +08:00