Commit Graph

1073 Commits

Author SHA1 Message Date
Markus Armbruster 9af2398977 Include less of the generated modular QAPI headers
In my "build everything" tree, a change to the types in
qapi-schema.json triggers a recompile of about 4800 out of 5100
objects.

The previous commit split up qmp-commands.h, qmp-event.h, qmp-visit.h,
qapi-types.h.  Each of these headers still includes all its shards.
Reduce compile time by including just the shards we actually need.

To illustrate the benefits: adding a type to qapi/migration.json now
recompiles some 2300 instead of 4800 objects.  The next commit will
improve it further.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180211093607.27351-24-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
[eblake: rebase to master]
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-02 13:45:50 -06:00
Markus Armbruster 252dc3105f qapi: Generate separate .h, .c for each module
Our qapi-schema.json is composed of modules connected by include
directives, but the generated code is monolithic all the same: one
qapi-types.h with all the types, one qapi-visit.h with all the
visitors, and so forth.  These monolithic headers get included all
over the place.  In my "build everything" tree, adding a QAPI type
recompiles about 4800 out of 5100 objects.

We wouldn't write such monolithic headers by hand.  It stands to
reason that we shouldn't generate them, either.

Split up generated qapi-types.h to mirror the schema's modular
structure: one header per module.  Name the main module's header
qapi-types.h, and sub-module D/B.json's header D/qapi-types-B.h.

Mirror the schema's includes in the headers, so that qapi-types.h gets
you everything exactly as before.  If you need less, you can include
one or more of the sub-module headers.  To be exploited shortly.

Split up qapi-types.c, qapi-visit.h, qapi-visit.c, qmp-commands.h,
qmp-commands.c, qapi-event.h, qapi-event.c the same way.
qmp-introspect.h, qmp-introspect.c and qapi.texi remain monolithic.

The split of qmp-commands.c duplicates static helper function
qmp_marshal_output_str() in qapi-commands-char.c and
qapi-commands-misc.c.  This happens when commands returning the same
type occur in multiple modules.  Not worth avoiding.

Since I'm going to rename qapi-event.[ch] to qapi-events.[ch], and
qmp-commands.[ch] to qapi-commands.[ch], name the shards that way
already, to reduce churn.  This requires temporary hacks in
commands.py and events.py.  Similarly, c_name() must temporarily
be taught to munge '/' in common.py.  They'll go away with the rename.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180211093607.27351-23-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
[eblake: declare a dummy variable in each .c file, to shut up OSX
toolchain warnings about empty .o files, including hacking c_name()]
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-02 13:44:24 -06:00
Markus Armbruster f9c146399d qapi/common: Fix guardname() for funny filenames
guardname() fails to return a valid C identifier for arguments
containing anything but [A-Za-z0-9_.-'].  Fix that.  Don't bother
protecting ticklish identifiers; header guards are all-caps, and no
ticklish identifiers are.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180211093607.27351-22-armbru@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-02 13:14:10 -06:00
Markus Armbruster cdb6610ae4 qapi/types qapi/visit: Generate built-in stuff into separate files
Linking code from multiple separate QAPI schemata into the same
program is possible, but involves some weirdness around built-in
types:

* We generate code for built-in types into .c only with option
  --builtins.  The user is responsible for generating code for exactly
  one QAPI schema per program with --builtins.

* We generate code for built-in types into .h regardless of
  --builtins, but guarded by #ifndef QAPI_VISIT_BUILTIN.  Because all
  copies of this code are exactly the same, including any combination
  of these headers works.

Replace this contraption by something more conventional: generate code
for built-in types into their very own files: qapi-builtin-types.c,
qapi-builtin-visit.c, qapi-builtin-types.h, qapi-builtin-visit.h, but
only with --builtins.  Obey --output-dir, but ignore --prefix for
them.

Make qapi-types.h include qapi-builtin-types.h.  With multiple
schemata you now have multiple qapi-types.[ch], but only one
qapi-builtin-types.[ch].  Same for qapi-visit.[ch] and
qapi-builtin-visit.[ch].

Bonus: if all you need is built-in stuff, you can include a much
smaller header.  To be exploited shortly.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180211093607.27351-21-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
[eblake: fix octal constant for python 3]
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-02 13:14:10 -06:00
Markus Armbruster 71b3f0459c qapi: Make code-generating visitors use QAPIGen more
The use of QAPIGen is rather shallow so far: most of the output
accumulation is not converted.  Take the next step: convert output
accumulation in the code-generating visitor classes.  Helper functions
outside these classes are not converted.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180211093607.27351-20-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
[eblake: rebase to earlier guardstart cleanup]
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-02 13:14:10 -06:00
Markus Armbruster 834a3f3498 qapi: Rename generated qmp-marshal.c to qmp-commands.c
All generated .c are named like their .h, except for qmp-marshal.c and
qmp-commands.h.  To add to the confusion, tests-qmp-commands.c falsely
matches generated test-qmp-commands.h.

Get rid of this unnecessary complication.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180211093607.27351-19-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-02 13:14:10 -06:00
Markus Armbruster cf40a0a5c2 qapi: Record 'include' directives in intermediate representation
The include directive permits modular QAPI schemata, but the generated
code is monolithic all the same.  To permit generating modular code,
the front end needs to pass more information on inclusions to the back
ends.  The commit before last added the necessary information to the
parse tree.  This commit adds it to the intermediate representation
and its QAPISchemaVisitor.  A later commit will use this to to
generate modular code.

New entity QAPISchemaInclude represents inclusions.  Call new visitor
method visit_include() for it, so visitors can see the sub-modules a
module includes.

Note that unlike other entities, QAPISchemaInclude has no name, and is
therefore not added to entity_dict.

New QAPISchemaEntity attribute @module names the entity's source file.
Call new visitor method visit_module() when it changes during a visit,
so visitors can keep track of the module being visited.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180211093607.27351-18-armbru@redhat.com>
[eblake: avoid accidental deletion of self._predefining]
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-02 13:14:10 -06:00
Markus Armbruster 8a84767cc4 qapi: Generate in source order
The generators' conversion to visitors (merge commit 9e72681d16)
changed the processing order of entities from source order to
alphabetical order.  The next commit needs source order, so change it
back.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180211093607.27351-17-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-02 13:14:10 -06:00
Markus Armbruster 97f0249474 qapi: Record 'include' directives in parse tree
The parse tree is a list of expressions.  Except include expressions
currently get replaced by the included file's parse tree.

Instead of throwing away the include expression, keep it with the file
name expanded so you don't have to track the including file's
directory to make sense of it.

A future commit will put this include expression to use.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180211093607.27351-16-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
[eblake: fix check of expr after assignment]
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-02 13:14:09 -06:00
Markus Armbruster 4257053083 qapi: Concentrate QAPISchemaParser.exprs updates in .__init__()
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180211093607.27351-15-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-02 13:14:09 -06:00
Markus Armbruster 181feaf355 qapi: Lift error reporting from QAPISchema.__init__() to callers
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180211093607.27351-14-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-02 13:14:09 -06:00
Markus Armbruster 71a7510baf qapi/common: Eliminate QAPISchema.exprs
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180211093607.27351-13-armbru@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-02 13:14:09 -06:00
Markus Armbruster af97502ce9 qapi: Improve include file name reporting in error messages
Error messages print absolute file names of included files even if the
user gave a relative one on the command line:

    $ PYTHONPATH=scripts python -B tests/qapi-schema/test-qapi.py tests/qapi-schema/include-cycle.json
    In file included from tests/qapi-schema/include-cycle.json:1:
    In file included from /work/armbru/qemu/tests/qapi-schema/include-cycle-b.json:1:
    /work/armbru/qemu/tests/qapi-schema/include-cycle-c.json:1: Inclusion loop for include-cycle.json

Improve this to

    In file included from tests/qapi-schema/include-cycle.json:1:
    In file included from tests/qapi-schema/include-cycle-b.json:1:
    tests/qapi-schema/include-cycle-c.json:1: Inclusion loop for include-cycle.json

The error message when an include file can't be opened prints the
include directive's file name, which is relative to the including
file.  Change this to print the file name relative to the working
directory.  Visible in tests/qapi-schema/include-no-file.err.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180211093607.27351-12-armbru@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-02 13:14:09 -06:00
Markus Armbruster 907b846653 qapi: Touch generated files only when they change
A massive number of objects depends on QAPI-generated headers.  In my
"build everything" tree, it's roughly 4800 out of 5100.  This is
particularly annoying when only some of the generated files change,
say for a doc fix.

Improve qapi-gen.py to touch its output files only if they actually
change.  Rebuild time for a QAPI doc fix drops from many minutes to a
few seconds.  Rebuilds get faster for certain code changes, too.  For
instance, adding a simple QMP event now recompiles less than 200
instead of 4800 objects.  But adding a QAPI type is as bad as ever;
we've clearly got more work to do.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180211093607.27351-11-armbru@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
[eblake: fix octal constant for python3]
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-02 13:14:09 -06:00
Markus Armbruster 3b446a1817 qapi-gen: Convert from getopt to argparse
argparse is nicer to use than getopt, and gives us --help almost for
free.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180211093607.27351-10-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
[eblake: Fix --output-dir editing accident]
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-02 13:14:09 -06:00
Markus Armbruster fb0bc835e5 qapi-gen: New common driver for code and doc generators
Whenever qapi-schema.json changes, we run six programs eleven times to
update eleven files.  Similar for qga/qapi-schema.json.  This is
silly.  Replace the six programs by a single program that spits out
all eleven files.

The programs become modules in new Python package qapi, along with the
helper library.  This requires moving them to scripts/qapi/.  While
moving them, consistently drop executable mode bits.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180211093607.27351-9-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
[eblake: move change to one-line 'blurb' earlier in series, mention mode
bit change as intentional, update qapi-code-gen.txt to match actual
generated events.c file]
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-02 13:14:09 -06:00
Markus Armbruster 26df4e7fab qapi: Turn generators into modules
The next commit will introduce a common driver program for all
generators.  The generators need to be modules for that.  qapi2texi.py
already is.  Make the other generators follow suit.

The changes are actually trivial.  Obvious in the diffs once you view
them with whitespace changes ignored.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180211093607.27351-8-armbru@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
[eblake: minor tweak to keep 'blurb' one line]
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-02 13:14:09 -06:00
Markus Armbruster 93b564c444 qapi: Reduce use of global variables in generators some
In preparation of the next commit, which will turn the generators into
modules.  These global variables will become local to main() then.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180211093607.27351-7-armbru@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-02 13:14:09 -06:00
Markus Armbruster 47a6ea9aab qapi: New classes QAPIGenC, QAPIGenH, QAPIGenDoc
These classes encapsulate accumulating and writing output.

Convert C code generation to QAPIGenC and QAPIGenH.  The conversion is
rather shallow: most of the output accumulation is not converted.
Left for later.

The indentation machinery uses a single global variable indent_level,
even though we generally interleave creation of a .c and its .h.  It
should become instance variable of QAPIGenC.  Also left for later.

Documentation generation isn't converted, and QAPIGenDoc isn't used.
This will change shortly.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180211093607.27351-6-armbru@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
[eblake: fix nits spotted by Michael]
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-02 13:14:09 -06:00
Markus Armbruster d46eec4260 qapi: Rename variable holding the QAPISchemaGenFOOVisitor
Rename the variable holding the QAPISchemaGenFOOVisitor from gen to
vis, to avoid confusion in the next commit.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180211093607.27351-5-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-02 13:14:09 -06:00
Markus Armbruster 5ddeec83eb qapi: Generate up-to-date copyright notice
Each generator carries a copyright notice for the generator itself,
and another one for the files it generates.  Only the former have been
updated along the way, the latter have not, and are all out of date.

Fix by copying the generator's copyright notice to the generated files
instead.  Note that the fix doesn't copy the "Authors:" part; the
generated files' outdated Authors list goes away without replacement.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180211093607.27351-4-armbru@redhat.com>
Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
[eblake: Flatten each 'blurb' to one line]
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-02 13:14:09 -06:00
Markus Armbruster c263de3f41 qapi: Streamline boilerplate comment generation
Every generator has separate boilerplate for .h and .c, and their
differences are boring.  All of them repeat the license note.

Reduce the repetition as follows.  Move common text like the license
note to common open_output(), next to the existing common text there.
For each generator, replace the two separate descriptions by a single
one.

While there, emit an "automatically generated" note into generated
documentation, too.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180211093607.27351-3-armbru@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-02 13:14:08 -06:00
Markus Armbruster 0dd13589b0 Include qapi/qmp/qerror.h exactly where needed
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180211093607.27351-2-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2018-03-02 13:14:08 -06:00
Richard Henderson 768055980b decodetree: Propagate return value from translate subroutines
Allow the translate subroutines to return false for invalid insns.

At present we can of course invoke an invalid insn exception from within
the translate subroutine, but in the short term this consolidates code.
In the long term it would allow the decodetree language to support
overlapping patterns for ISA extensions.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Message-id: 20180227232618.2908-1-richard.henderson@linaro.org
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2018-03-02 11:03:45 +00:00
Richard Henderson 568ae7efae scripts: Add decodetree.py
To be used to decode ARM SVE, but could be used for any fixed-width ISA.

Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
2018-02-22 15:44:07 -08:00
Peter Maydell afd3397a81 -----BEGIN PGP SIGNATURE-----
iQEcBAABAgAGBQJaivkiAAoJEJykq7OBq3PIyFYIAKxOSqa0gQp2YyHDqj3Bg6lS
 qdxiejuoJet8FAb+a2RryZERlaN7ovTFc+1O247FEwy9nQGqFoRHRqF8qsVqOzex
 4zvDQjYDJch9BrIdApzuNHVOpmKXgPHfGKPl+viU3Qh08xLCZ9fWB88mYzYlq3OC
 N6GhpT0ojpjOmAa7k+Ur38zjn4jhHL6MY7RdhwsoNFJGvW62Y00jN58OqBYg0ax7
 YUUN76zwFhyDZPS1ayehUWgTvDP804B5zJWZwv7V4AfXLf29820EU381DgSk9D/n
 1RN1RmJ3nq/K323RNvVabTLXHI8nqrJOluK47BA319X+6jJp5wAhXBl2aL8BHtg=
 =soKg
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/stefanha/tags/tracing-pull-request' into staging

# gpg: Signature made Mon 19 Feb 2018 16:19:46 GMT
# gpg:                using RSA key 9CA4ABB381AB73C8
# gpg: Good signature from "Stefan Hajnoczi <stefanha@redhat.com>"
# gpg:                 aka "Stefan Hajnoczi <stefanha@gmail.com>"
# Primary key fingerprint: 8695 A8BF D3F9 7CDA AC35  775A 9CA4 ABB3 81AB 73C8

* remotes/stefanha/tags/tracing-pull-request:
  trace: avoid SystemTap "char const" warnings
  tracetool: For ust trace bool type as ctf_integer
  tracetool: Update argument format regex to non-greedy star

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2018-02-19 16:44:12 +00:00
Jon Emil Jahren 61b01bbc6c tracetool: For ust trace bool type as ctf_integer
Previously functions having arguments of type bool was not traced
properly. The bool arguments were missing from the trace.

Signed-off-by: Jon Emil Jahren <jonemilj@gmail.com>
Message-id: 20180129041648.30884-3-jonemilj@gmail.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
2018-02-19 13:09:44 +00:00
Jon Emil Jahren fb1a66bc01 tracetool: Update argument format regex to non-greedy star
Using the greedy star matching, arguments like "...%"PRIx64 caused issues
for functions with multiple PRI formats.

The issue was only seen with the ust backend, as it is the only one
using the format regex.

The result for many functions was that the arguments coming after the
greedy star end was left out of the tracepoint, and in some cases some
of the arguments that was traced had the wrong format.

Signed-off-by: Jon Emil Jahren <jonemilj@gmail.com>
Message-id: 20180129041648.30884-2-jonemilj@gmail.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
2018-02-19 13:09:44 +00:00
Marcel Apfelbaum e1c5f1f04a scripts/update-linux-headers: import pvrdma headers
Modify the script to import the headers used by the pvrdma device.
Part of them are interfaces between the guest driver and the device,
import them under include/standart-headers/drivers/infiniband/... .

Remove the unused functions from pvrdma_verbs.h avoiding the
unnecessary import of several infiniband/networking/other headers.

Reviewed-by: Gal Hammer <ghammer@redhat.com>
Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
Signed-off-by: Yuval Shaia <yuval.shaia@oracle.com>
2018-02-19 13:03:24 +02:00
Laurent Vivier bff8a0bbe9 build: fix typo in error message
Signed-off-by: Laurent Vivier <lvivier@redhat.com>
Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Fixes: f62bbee55d
2018-02-10 10:17:06 +03:00
Cole Robinson d0081e8f26 scripts/make-release: Don't archive .git files
As was last done in 379e21c25, we don't want .git files for
submodules here, which we aren't presently doing for capstone and
keycodemapdb.

Rather than delete the offending files before archiving, ask tar
to --exclude=.git

Signed-off-by: Cole Robinson <crobinso@redhat.com>
Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>
Reviewed-by: Thomas Huth <thuth@redhat.com>
2018-02-10 10:11:20 +03:00
Michael Tokarev 8c2486a512 scripts/argparse.py: spelling (independant)
Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>
Reviewed-by: Eric Blake <eblake@redhat.com>
2018-02-10 10:03:39 +03:00
Markus Armbruster 452fcdbc49 Include qapi/qmp/qdict.h exactly where needed
This cleanup makes the number of objects depending on qapi/qmp/qdict.h
drop from 4550 (out of 4743) to 368 in my "build everything" tree.
For qapi/qmp/qobject.h, the number drops from 4552 to 390.

While there, separate #include from file comment with a blank line.

Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180201111846.21846-13-armbru@redhat.com>
2018-02-09 13:52:15 +01:00
Markus Armbruster 6b67395762 Eliminate qapi/qmp/types.h
qapi/qmp/types.h is a convenience header to include a number of
qapi/qmp/ headers.  Since we rarely need all of the headers
qapi/qmp/types.h includes, we bypass it most of the time.  Most of the
places that use it don't need all the headers, either.

Include the necessary headers directly, and drop qapi/qmp/types.h.

Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180201111846.21846-9-armbru@redhat.com>
2018-02-09 13:52:15 +01:00
Markus Armbruster e688df6bc4 Include qapi/error.h exactly where needed
This cleanup makes the number of objects depending on qapi/error.h
drop from 1910 (out of 4743) to 1612 in my "build everything" tree.

While there, separate #include from file comment with a blank line,
and drop a useless comment on why qemu/osdep.h is included first.

Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180201111846.21846-5-armbru@redhat.com>
[Semantic conflict with commit 34e304e975 resolved, OSX breakage fixed]
2018-02-09 13:50:17 +01:00
Peter Maydell 7b213bb475 * socket option parsing fix (Daniel)
* SCSI fixes (Fam)
 * Readline double-free fix (Greg)
 * More HVF attribution fixes (Izik)
 * WHPX (Windows Hypervisor Platform Extensions) support (Justin)
 * POLLHUP handler (Klim)
 * ivshmem fixes (Ladi)
 * memfd memory backend (Marc-André)
 * improved error message (Marcelo)
 * Memory fixes (Peter Xu, Zhecheng)
 * Remove obsolete code and comments (Peter M.)
 * qdev API improvements (Philippe)
 * Add CONFIG_I2C switch (Thomas)
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.22 (GNU/Linux)
 
 iQEcBAABAgAGBQJaexoYAAoJEL/70l94x66DVL0IAJC//aZCwwgyN9CRNDcOo10/
 UPtzprfezERkur77r1KvEYVNIfslRF6iTBou2+suOWkzoNL2LJ0XZ+wi+2u2sFIF
 ikvbQVk4dOWqJJQj7e1cmv5A2EZy2dcxjAoD1IG6CRy76+HzYqwjHVw+HkYY5CUS
 qwnUWjQddP6WtH9MsUHpX7p7atWo7T1tzkx4v8H+CIHBO3uUJQSZLkGYflvcstpj
 Fo04bZzSkDj2rnlqqBo/6UgJQXD8++Rs64vmiX2xwcK47TWO31Vbuwu+r8V9osWm
 LHFmRpL8ZkZfL0yqf0bpjmd688dirjVpHIJ5KE043Lo6AdI+K5xBfoBjXxtPiKE=
 =o90D
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/bonzini/tags/for-upstream' into staging

* socket option parsing fix (Daniel)
* SCSI fixes (Fam)
* Readline double-free fix (Greg)
* More HVF attribution fixes (Izik)
* WHPX (Windows Hypervisor Platform Extensions) support (Justin)
* POLLHUP handler (Klim)
* ivshmem fixes (Ladi)
* memfd memory backend (Marc-André)
* improved error message (Marcelo)
* Memory fixes (Peter Xu, Zhecheng)
* Remove obsolete code and comments (Peter M.)
* qdev API improvements (Philippe)
* Add CONFIG_I2C switch (Thomas)

# gpg: Signature made Wed 07 Feb 2018 15:24:08 GMT
# gpg:                using RSA key BFFBD25F78C7AE83
# gpg: Good signature from "Paolo Bonzini <bonzini@gnu.org>"
# gpg:                 aka "Paolo Bonzini <pbonzini@redhat.com>"
# Primary key fingerprint: 46F5 9FBD 57D6 12E7 BFD4  E2F7 7E15 100C CD36 69B1
#      Subkey fingerprint: F133 3857 4B66 2389 866C  7682 BFFB D25F 78C7 AE83

* remotes/bonzini/tags/for-upstream: (47 commits)
  Add the WHPX acceleration enlightenments
  Introduce the WHPX impl
  Add the WHPX vcpu API
  Add the Windows Hypervisor Platform accelerator.
  tests/test-filter-redirector: move close()
  tests: use memfd in vhost-user-test
  vhost-user-test: make read-guest-mem setup its own qemu
  tests: keep compiling failing vhost-user tests
  Add memfd based hostmem
  memfd: add hugetlbsize argument
  memfd: add hugetlb support
  memfd: add error argument, instead of perror()
  cpus: join thread when removing a vCPU
  cpus: hvf: unregister thread with RCU
  cpus: tcg: unregister thread with RCU, fix exiting of loop on unplug
  cpus: dummy: unregister thread with RCU, exit loop on unplug
  cpus: kvm: unregister thread with RCU
  cpus: hax: register/unregister thread with RCU, exit loop on unplug
  ivshmem: Disable irqfd on device reset
  ivshmem: Improve MSI irqfd error handling
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

# Conflicts:
#	cpus.c
2018-02-07 20:40:36 +00:00
Amador Pahim 156dc7b174 qemu.py: don't launch again before shutdown()
If a VM is launched, files are created and a cleanup is required before
a new launch. This cleanup is executed by shutdown(), so shutdown() must
be called even if the VM is manually terminated (i.e. using kill).

This patch creates a control to make sure launch() will not be executed
again if shutdown() is not called after the previous launch().

Signed-off-by: Amador Pahim <apahim@redhat.com>
Message-Id: <20180122205033.24893-7-apahim@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
2018-02-05 19:53:55 -02:00
Amador Pahim c58b535f83 qemu.py: cleanup redundant calls in launch()
Now that shutdown() is guaranteed to always execute self._load_io_log()
and self._post_shutdown(), their calls in 'except' became redundant and
we can safely replace it by a call to shutdown().

Reviewed-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Signed-off-by: Amador Pahim <apahim@redhat.com>
Message-Id: <20180122205033.24893-6-apahim@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
2018-02-05 19:53:55 -02:00
Amador Pahim 17589cae90 qemu.py: use poll() instead of 'returncode'
The 'returncode' Popen attribute is not guaranteed to be updated. It
actually depends on a call to either poll(), wait() or communicate().

On the other hand, poll() will: "Check if child process has terminated.
Set and return returncode attribute."

Let's use the poll() to check whether the process is running and to get
the updated process exit code, when the process is finished.

Reviewed-by: Fam Zheng <famz@redhat.com>
eviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Signed-off-by: Amador Pahim <apahim@redhat.com>
Message-Id: <20180122205033.24893-5-apahim@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
2018-02-05 19:53:55 -02:00
Amador Pahim 04a963b495 qemu.py: always cleanup on shutdown()
Currently we only cleanup on shutdown() if the VM is running.

To make sure we will always cleanup, this patch makes the
self._load_io_log() and the self._post_shutdown() to
always be called on shutdown(), regardless the VM running state.

Reviewed-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Signed-off-by: Amador Pahim <apahim@redhat.com>
Message-Id: <20180122205033.24893-4-apahim@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
2018-02-05 19:53:55 -02:00
Amador Pahim d301bccf73 qemu.py: refactor launch()
This is just a refactor to separate the exception handler from the
actual launch procedure, improving the readability and making future
maintenances in this piece of code easier.

Reviewed-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Signed-off-by: Amador Pahim <apahim@redhat.com>
Message-Id: <20180122205033.24893-3-apahim@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
2018-02-05 19:53:55 -02:00
Amador Pahim af99fa9fe2 qemu.py: better control of created files
To launch a VM, we need to create basically two files: the monitor
socket (if it's a UNIX socket) and the qemu log file.

For the qemu log file, we currently just open the path, which will
create the file if it does not exist or overwrite the file if it does
exist.

For the monitor socket, if it already exists, we are currently removing
it, even if it's not created by us.

This patch moves to _pre_launch() the responsibility to create a
temporary directory to host the files so we can remove the whole
directory on _post_shutdown().

Signed-off-by: Amador Pahim <apahim@redhat.com>
Message-Id: <20180122205033.24893-2-apahim@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
2018-02-05 19:53:55 -02:00
Amador Pahim 74af2e59d2 qemu.py: remove unused import
Removing 'import sys' as it's not used anywhere.

Signed-off-by: Amador Pahim <apahim@redhat.com>
Message-Id: <20171114102246.22221-2-apahim@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
2018-02-05 19:53:55 -02:00
Daniel P. Berrange 31d8f92e64 scripts: ensure signrom treats data as bytes
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Message-Id: <20180116134217.8725-10-berrange@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
2018-02-05 19:53:54 -02:00
Daniel P. Berrange f7a5376d4b qapi: ensure stable sort ordering when checking QAPI entities
Some early python 3.x versions will have different default
ordering when calling the 'values()' method on a dict, compared
to python 2.x and later 3.x versions. Explicitly sort the items
to get a stable ordering.

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Message-Id: <20180116134217.8725-8-berrange@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
2018-02-05 19:53:54 -02:00
Daniel P. Berrange 52c4272c6c qapi: Adapt to moved location of 'maketrans' function in py3
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Message-Id: <20180116134217.8725-6-berrange@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
2018-02-05 19:53:54 -02:00
Daniel P. Berrange 5f90af8e6b qapi: adapt to moved location of StringIO module in py3
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Message-Id: <20180116134217.8725-5-berrange@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
2018-02-05 19:53:54 -02:00
Daniel P. Berrange 38710a8994 qapi: Use OrderedDict from standard library if available
The OrderedDict class appeared in the 'collections' module
from python 2.7 onwards, so use that in preference to our
local backport if available.

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Message-Id: <20180116134217.8725-4-berrange@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
2018-02-05 19:53:54 -02:00
Daniel P. Berrange 2f84804470 qapi: use items()/values() intead of iteritems()/itervalues()
The iteritems()/itervalues() methods are gone in py3, but the
items()/values() methods are still around. The latter are less
efficient than the former in py2, but this has unmeasurably
small impact on QEMU build time, so taking portability over
efficiency is a net win.

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Message-Id: <20180116134217.8725-3-berrange@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
2018-02-05 19:53:54 -02:00
Daniel P. Berrange ef9d910891 qapi: convert to use python print function instead of statement
Python 3 no longer supports the bare "print" statement, it must be
called as a normal function with round brackets. It is possible to
opt-in to this new syntax with Python 2.6 onwards by importing the
"print_function" from the "__future__" module, making it easy to
support Python 2 and 3 in parallel.

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Message-Id: <20180116134217.8725-2-berrange@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
2018-02-05 19:53:54 -02:00