This bug seems similar to #358865.

I am using Tightvnc (tried both 1.29 and 1.39) under WinXP to connect to an Xvnc instance started with:

/usr/bin/Xvnc :42 -geometry 1024x768 -query localhost

When I connect, Xvnc immediately terminates with these messages:

_XSERVTransSocketOpenCOTSServer: Unable to open socket for inet6
_XSERVTransOpen: transport open failed for inet6/blob:42
_XSERVTransMakeAllCOTSServerListeners: failed to open listener for inet6
XDMCP warning: INET6 UDP socket creation failed
20/06/2008 18:47:09 Xvnc version X.org/xf4vnc custom version
20/06/2008 18:47:09 Copyright (C) 2001-2004 Alan Hourihane.
20/06/2008 18:47:09 Copyright (C) 2000-2004 Constantin Kaplinsky
20/06/2008 18:47:09 Copyright (C) 1999 AT&T Laboratories Cambridge
20/06/2008 18:47:09 All Rights Reserved.
20/06/2008 18:47:09 See http://www.tightvnc.com/ for information on TightVNC
20/06/2008 18:47:09 See http://xf4vnc.sf.net for xf4vnc-specific information
20/06/2008 18:47:09 Desktop name 'x11' (blob:42)
20/06/2008 18:47:09 Protocol versions supported: 3.7, 3.3
20/06/2008 18:47:09 Listening for VNC connections on TCP port 5942
Could not init font path element /usr/share/fonts/TTF/, removing from list!
Could not init font path element /usr/share/fonts/OTF, removing from list!
20/06/2008 18:47:48 
20/06/2008 18:47:48 Got VNC connection from client 192.168.10.97
20/06/2008 18:47:48 Using protocol version 3.7
20/06/2008 18:47:48 Enabling TightVNC protocol extensions
20/06/2008 18:47:48 Pixel format for client 192.168.10.97:
20/06/2008 18:47:48   8 bpp, depth 8
20/06/2008 18:47:48   true colour: max r 7 g 7 b 3, shift r 0 g 3 b 6
20/06/2008 18:47:48   no translation needed
20/06/2008 18:47:48 Using hextile encoding for client 192.168.10.97
20/06/2008 18:47:48 rfbProcessClientNormalMessage: ignoring unknown encoding 8
20/06/2008 18:47:48 Enabling X-style cursor updates for client 192.168.10.97
20/06/2008 18:47:48 Enabling cursor position updates for client 192.168.10.97
20/06/2008 18:47:48 Using image quality level 6 for client 192.168.10.97
20/06/2008 18:47:48 Enabling LastRect protocol extension for client 192.168.10.97
20/06/2008 18:47:48 rfbProcessClientNormalMessage: ignoring unknown encoding -223
*** glibc detected *** /usr/bin/Xvnc: free(): invalid next size (fast): 0x084a6690 ***
======= Backtrace: =========
/lib/libc.so.6[0xb7c5ffc4]
/lib/libc.so.6(cfree+0x9c)[0xb7c6195c]
/usr/bin/Xvnc[0x8089658]
/usr/bin/Xvnc(rfbSendRectEncodingHextile+0x13f)[0x808af6f]
/usr/bin/Xvnc(rfbSendFramebufferUpdate+0x796)[0x808fd76]
/usr/bin/Xvnc(rfbProcessClientMessage+0x592)[0x8091892]
/usr/bin/Xvnc(rfbCheckFds+0x150)[0x8098c50]
/usr/bin/Xvnc[0x808cf50]
/usr/bin/Xvnc(WakeupHandler+0x98)[0x80b6338]
/usr/bin/Xvnc(WaitForSomething+0x1ab)[0x8389f5b]
/usr/bin/Xvnc(Dispatch+0x8e)[0x80b207e]
/usr/bin/Xvnc(main+0x4b5)[0x80c6005]
/lib/libc.so.6(__libc_start_main+0xe5)[0xb7c0a5f5]
/usr/bin/Xvnc[0x807fea1]

Disabling GLIBC checking via MALLOC_CHECK_=0 does not solve the problem. When i set that, there is a segmentation fault instead of the glibc message.

This worked with openSUSE 10.3, where Xvnc seems to be another version:

_XSERVTransSocketOpenCOTSServer: Unable to open socket for inet6
_XSERVTransOpen: transport open failed for inet6/blob:42
_XSERVTransMakeAllCOTSServerListeners: failed to open listener for inet6
XDMCP warning: INET6 UDP socket creation failed

Xvnc Free Edition 4.1.2
Copyright (C) 2002-2005 RealVNC Ltd.
See http://www.realvnc.com for information on VNC.
Underlying X server release 70100000, The X.Org Foundation