Currently when qemu receives a vnc connect, it creates a 'VncState' to
represent this connection. In 'vnc_worker_thread_loop' it creates a
local 'VncState'. The connection 'VcnState' and local 'VncState' exchange
data in 'vnc_async_encoding_start' and 'vnc_async_encoding_end'.
In 'zrle_compress_data' it calls 'deflateInit2' to allocate the libz library
opaque data. The 'VncState' used in 'zrle_compress_data' is the local
'VncState'. In 'vnc_zrle_clear' it calls 'deflateEnd' to free the libz
library opaque data. The 'VncState' used in 'vnc_zrle_clear' is the connection
'VncState'. In currently implementation there will be a memory leak when the
vnc disconnect. Following is the asan output backtrack:
Direct leak of 29760 byte(s) in 5 object(s) allocated from:
0 0xffffa67ef3c3 in __interceptor_calloc (/lib64/libasan.so.4+0xd33c3)
1 0xffffa65071cb in g_malloc0 (/lib64/libglib-2.0.so.0+0x571cb)
2 0xffffa5e968f7 in deflateInit2_ (/lib64/libz.so.1+0x78f7)
3 0xaaaacec58613 in zrle_compress_data ui/vnc-enc-zrle.c:87
4 0xaaaacec58613 in zrle_send_framebuffer_update ui/vnc-enc-zrle.c:344
5 0xaaaacec34e77 in vnc_send_framebuffer_update ui/vnc.c:919
6 0xaaaacec5e023 in vnc_worker_thread_loop ui/vnc-jobs.c:271
7 0xaaaacec5e5e7 in vnc_worker_thread ui/vnc-jobs.c:340
8 0xaaaacee4d3c3 in qemu_thread_start util/qemu-thread-posix.c:502
9 0xffffa544e8bb in start_thread (/lib64/libpthread.so.0+0x78bb)
10 0xffffa53965cb in thread_start (/lib64/libc.so.6+0xd55cb)
This is because the opaque allocated in 'deflateInit2' is not freed in
'deflateEnd'. The reason is that the 'deflateEnd' calls 'deflateStateCheck'
and in the latter will check whether 's->strm != strm'(libz's data structure).
This check will be true so in 'deflateEnd' it just return 'Z_STREAM_ERROR' and
not free the data allocated in 'deflateInit2'.
The reason this happens is that the 'VncState' contains the whole 'VncZrle',
so when calling 'deflateInit2', the 's->strm' will be the local address.
So 's->strm != strm' will be true.
To fix this issue, we need to make 'zrle' of 'VncState' to be a pointer.
Then the connection 'VncState' and local 'VncState' exchange mechanism will
work as expection. The 'tight' of 'VncState' has the same issue, let's also turn
it to a pointer.
Reported-by: Ying Fang <fangying1@huawei.com>
Signed-off-by: Li Qiang <liq3ea@163.com>
Message-id: 20190831153922.121308-1-liq3ea@163.com
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
No header includes qemu-common.h after this commit, as prescribed by
qemu-common.h's file comment.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190523143508.25387-5-armbru@redhat.com>
[Rebased with conflicts resolved automatically, except for
include/hw/arm/xlnx-zynqmp.h hw/arm/nrf51_soc.c hw/arm/msf2-soc.c
block/qcow2-refcount.c block/qcow2-cluster.c block/qcow2-cache.c
target/arm/cpu.h target/lm32/cpu.h target/m68k/cpu.h target/mips/cpu.h
target/moxie/cpu.h target/nios2/cpu.h target/openrisc/cpu.h
target/riscv/cpu.h target/tilegx/cpu.h target/tricore/cpu.h
target/unicore32/cpu.h target/xtensa/cpu.h; bsd-user/main.c and
net/tap-bsd.c fixed up]
When adding '-fsanitize=undefined' in compiling configuration
and connect VM with vnc, it reports following error:
ui/vnc-enc-tight.c:910:13: runtime error: load of
misaligned address 0x621000466513 for type 'uint32_t',
which requires 4 byte alignment
This patch fix this issue.
Signed-off-by: Li Qiang <liq3ea@163.com>
Message-id: 20190318010442.14897-1-liq3ea@163.com
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
I used the clang-tidy qemu-round check to generate the fix:
https://github.com/elmarco/clang-tools-extra
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Richard Henderson <rth@twiddle.net>
We would like to use a same QObject type to represent numbers, whether
they are int, uint, or floats. Getters will allow some compatibility
between the various types if the number fits other representations.
Add a few more tests while at it.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20170607163635.17635-7-marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[parse_stats_intervals() simplified a bit, comment in
test_visitor_in_int_overflow() tidied up, suppress bogus warnings]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
When enabling the sanitizer build it will complain about control
reaching a non-void function. Normally the compiler should detect that
there is only one possible exit given a static VNC_SERVER_FB_BYTES.
As we always expect a static VNC_SERVER_FB_BYTES I've added a compile
time assert and just called the sub-function directly.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
commit 095497ff added thread local storage for the color counting
palette. Unfortunately, a VncPalette is about 7kB on a x86_64 system.
This memory is reserved from the stack of every thread and it
exhausted the stack space of a libxenstore thread.
Fix this by allocating memory only for the VNC encoding thread.
Fixes: 095497ffc6
Reported-by: Juergen Gross <jgross@suse.com>
Tested-by: Juergen Gross <jgross@suse.com>
Signed-off-by: Peter Lieven <pl@kamp.de>
Message-id: 1468575911-20656-1-git-send-email-pl@kamp.de
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
In tight_encode_indexed_rect32, buf(or src)’s size is count. In for loop,
the logic is supposed to be that i is an index into src, i should be
incremented when incrementing src.
This is broken when src is incremented but i is not before while loop,
resulting in off-by-one bug in while loop.
Signed-off-by: He Rongguang <herongguang.he@huawei.com>
Message-id: 5784B8EB.7010008@huawei.com
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
currently the color counting palette is allocated from heap, used and destroyed
for each single subrect. Use a static palette per thread for this purpose and
avoid the malloc and free for each update.
Signed-off-by: Peter Lieven <pl@kamp.de>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-id: 1467280846-9674-1-git-send-email-pl@kamp.de
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Clean up includes so that osdep.h is included first and headers
which it implies are not included manually.
This commit was created with scripts/clean-includes.
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Message-id: 1454089805-5470-2-git-send-email-peter.maydell@linaro.org
Arguments in wrong order (SWAPPED_ARGUMENTS)
The positions of arguments in the call to
tight_fill_palette do not match the ordering of the parameters:
&fg is passed to bg
&bg is passed to fg
Cc: Gerd Hoffmann <kraxel@redhat.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
If we need to, we should use the pixman formats instead but for
now this is unused except in commented out code so take it out
to avoid further confusion about surface endianness.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
VncTight member uint8_t quality is either (uint8_t)-1 for lossless or
less than 10 for lossy.
tight_detect_smooth_image() first promotes it to int, then compares
with -1. Always unequal, so we always execute the lossy code. Reads
beyond tight_conf[] and returns crap when quality is actually
lossless.
Compare to (uint8_t)-1 instead, like we do elsewhere.
Spotted by Coverity.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
This patch adds an x argument to qemu_pixman_linebuf_fill so it can
also be used to convert a partial scanline. Then fix tight + png/jpeg
encoding by passing in the x+y offset, so the data is read from the
correct screen location instead of the upper left corner.
Cc: 1087974@bugs.launchpad.net
Cc: qemu-stable@nongnu.org
Reported-by: Tim Hardeneck <thardeck@suse.de>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Signed-off-by: Blue Swirl <blauwirbel@gmail.com>
Switching the vnc server framebuffer to use 32bpp unconditionally
turns the code bits which handle 8 and 16 bpp into dead code.
Remove them.
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
The vnc code uses *three* DisplaySurfaces:
First is the surface of the actual QemuConsole, usually the guest
screen, but could also be a text console (monitor/serial reachable via
Ctrl-Alt-<nr> keys). This is left as-is.
Second is the current server's view of the screen content. The vnc code
uses this to figure which parts of the guest screen did _really_ change
to reduce the amount of updates sent to the vnc clients. It is also
used as data source when sending out the updates to the clients. This
surface gets replaced by a pixman image. The format changes too,
instead of using the guest screen format we'll use fixed 32bit rgb
framebuffer and convert the pixels on the fly when comparing and
updating the server framebuffer.
Third surface carries the format expected by the vnc client. That isn't
used to store image data. This surface is switched to PixelFormat and a
boolean for bigendian byte order.
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Commit f26e428da5 fixed compilation
with --enable-vnc-png, but broke it with --enable-vnc-png.
The breakage is caused by pngconfig.h which checks whether
setjmp.h was already included and fails because qemu-common.h
includes setjmp.h.
The check is disabled by defining PNG_SKIP_SETJMP_CHECK.
Cc: Blue Swirl <blauwirbel@gmail.com>
Signed-off-by: Stefan Weil <weil@mail.berlios.de>
Reviewed-by: Stefan Hajnoczi <stefanha@linux.vnet.ibm.com>
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
Fix conflicting types for 'INT32' in basetsd.h and jmorecfg.h by
including qemu-common.h first.
Signed-off-by: Roy Tam <roytam@gmail.com>
Acked-by: Stefan Weil <weil@mail.berlios.de>
Signed-off-by: Blue Swirl <blauwirbel@gmail.com>
fix 2Gb integer overflow in in VNC tight and zlib encodings
As found by Roland Dreier <roland@purestorage.com> (excellent
catch!), when amount of VNC compressed data produced by zlib
and sent to client exceeds 2Gb, integer overflow occurs because
currently, we calculate amount of data produced at each step by
comparing saved total_out with new total_out, and total_out is
something which grows without bounds. Compare it with previous
avail_out instead of total_out, and leave total_out alone.
The same code is used in vnc-enc-tight.c and vnc-enc-zlib.c,
so fix both cases.
There, there's no actual need to save previous_out value, since
capacity-offset (which is how that value is calculated) stays
the same so it can be recalculated again after call to deflate(),
but whole thing becomes less readable this way.
Reported-by: Roland Dreier <roland@purestorage.com>
Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>
Signed-off-by: Corentin Chary <corentin.chary@gmail.com>
Signed-off-by: Aurelien Jarno <aurelien@aurel32.net>
Add some missing #ifdefs to fix compilation failures in the !CONFIG_VNC_JPEG
case introduced by commit ce702e93.
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
This option allow to disable adaptive behaviors in some encodings.
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
Use the new update frequency infrastructure to use jpeg for regions with
high update frequency.
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
bswap_NN() variants are not always available in CONFIG_MACHINE_BSWAP_H case
and bswapNN() are public APIs in "bswap.h".
Signed-off-by: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
Signed-off-by: Blue Swirl <blauwirbel@gmail.com>
Commit d167f9bc06 missed this one:
/src/qemu/ui/vnc-enc-tight.c:1483: warning: 'ret' may be used uninitialized in this function
Signed-off-by: Blue Swirl <blauwirbel@gmail.com>
Commit 7bccf57383 missed this one:
/src/qemu/ui/vnc-enc-tight.c: In function 'send_sub_rect':
/src/qemu/ui/vnc-enc-tight.c:1527: warning: comparison is always true due to limited range of data type
Signed-off-by: Blue Swirl <blauwirbel@gmail.com>
Fix mingw32 build errors like
/src/qemu/ui/vnc-enc-tight.c: In function 'tight_detect_smooth_image24':
/src/qemu/ui/vnc-enc-tight.c:119: error: 'uint' undeclared (first use in this function)
Replace 'uint' with proper 'unsigned int'.
Signed-off-by: Blue Swirl <blauwirbel@gmail.com>
Fix a warning with some GCCs:
/src/qemu/ui/vnc-enc-tight.c: In function `send_sub_rect_nojpeg':
/src/qemu/ui/vnc-enc-tight.c:1458: warning: `ret' might be used uninitialized in this function
Signed-off-by: Blue Swirl <blauwirbel@gmail.com>
Fix the following warnings:
/src/qemu/hw/ide/core.c: In function `ide_drive_pio_post_load':
/src/qemu/hw/ide/core.c:2767: warning: comparison is always false due to limited range of data type
/src/qemu/ui/vnc-enc-tight.c: In function `tight_detect_smooth_image':
/src/qemu/ui/vnc-enc-tight.c:284: warning: comparison is always true due to limited range of data type
/src/qemu/ui/vnc-enc-tight.c:297: warning: comparison is always true due to limited range of data type
/src/qemu/ui/vnc-enc-tight.c: In function `tight_encode_indexed_rect16':
/src/qemu/ui/vnc-enc-tight.c:456: warning: comparison is always false due to limited range of data type
/src/qemu/ui/vnc-enc-tight.c: In function `tight_encode_indexed_rect32':
/src/qemu/ui/vnc-enc-tight.c:457: warning: comparison is always false due to limited range of data type
Signed-off-by: Blue Swirl <blauwirbel@gmail.com>
Split send_sub_rect in send_sub_rect_jpeg and send_sub_rect_nojpeg to
remove all these #ifdef CONFIG_JPEG.
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
rgb_prepare_row bpp depends on the server display surface, not
the client.
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
This will allow to implement the threaded VNC server in a
more cleaner way.
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
Profiling with callgrind seems to show that a lot of time is spent
in the palette code (mostly due to memory allocation and qdict to int
conversion).
This patch adds a VncPalette implementation. The palette is stored
in a hash table, like qdict, but which does way less memory allocations,
and doesn't suffer from the QObject overhead.
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
Disable png filters for lower compression levels. This should lower
the CPU consumption and reduce encoding time.
This isn't in tight_conf because:
* tight_conf structure must not change, because it's shared with other
tight implementations (libvncserver, etc..).
* it'd exceed the 80 col limit.
* PNG_ macros are only defined if CONFIG_VNC_PNG is defined
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
Introduce a new encoding: VNC_ENCODING_TIGHT_PNG [1] (-269) with a new
tight filter VNC_TIGHT_PNG (0x0A). When the client tells it supports the Tight PNG
encoding, the server will use tight, but will always send encoding pixels using
PNG instead of zlib. If the client also told it support JPEG, then the server can
send JPEG, because PNG will only be used in the cases zlib was used in normal tight.
This encoding was introduced to speed up HTML5 based VNC clients like noVNC [2], but
can also be used on devices like iPhone where PNG can be rendered in hardware.
[1] http://wiki.qemu.org/VNC_Tight_PNG
[2] http://github.com/kanaka/noVNC/
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
While using indexed colors, the last color was never added to the palette.
Triggered with ubuntu livecd.
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
For the same reason that we don't use vnc-authentication-sasl.c but
vnc-auth-sals.c. Because it's tooooo long.
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
Li Qiang