rpc: Fix client crash when server drops connection

Despite the comment stating virNetClientIncomingEvent handler should
never be called with either client->haveTheBuck or client->wantClose
set, there is a sequence of events that may lead to both booleans being
true when virNetClientIncomingEvent is called. However, when that
happens, we must not immediately close the socket as there are other
threads waiting for the buck and they would cause SIGSEGV once they are
woken up after the socket was closed. Another thing is we should clear
all remaining calls in the queue after closing the socket.

The situation that can lead to the crash involves three threads, one of
them running event loop and the other two calling libvirt APIs. The
event loop thread detects an event on client->sock and calls
virNetClientIncomingEvent handler. But before the handler gets a chance
to lock client, the other two threads (T1 and T2) start calling some
APIs. T1 gets the buck and detects EOF on client->sock while processing
its RPC call. Since T2 is waiting for its own call, T1 passes the buck
on to it and unlocks client. But before T2 gets the signal, the event
loop thread wakes up, does its job and closes client->sock. The crash
happens when T2 actually wakes up and tries to do its job using a closed
client->sock.

src/rpc/virnetclient.c

@@ -1825,7 +1825,6 @@ void virNetClientIncomingEvent(virNetSocketPtr sock,
     if (!client->sock)
         goto done;
 
-    /* This should be impossible, but it doesn't hurt to check */
     if (client->haveTheBuck || client->wantClose)
         goto done;
 
@@ -1858,8 +1857,12 @@ void virNetClientIncomingEvent(virNetSocketPtr sock,
     virNetClientIOUpdateCallback(client, true);
 
 done:
-    if (client->wantClose)
+    if (client->wantClose && !client->haveTheBuck) {
         virNetClientCloseLocked(client);
+        virNetClientCallRemovePredicate(&client->waitDispatch,
+                                        virNetClientIOEventLoopRemoveAll,
+                                        NULL);
+    }
     virObjectUnlock(client);
 }