performs far better on OS X and Solaris-style kernels
Will Bryant (<will.bryant@gmail.com>) writes:
On 21/09/2011, at 20:14 , Christian Grothoff wrote:
> > On the OpenIndiana performance, do you have something like 'strace' where you
> > could monitor what's going on? Most likely the code hangs blocking in some
> > syscall for longer than it should...
It has truss, and for the record, it showed no slow calls.
The problem turned out to be the tests using "localhost", which was picking the primary interface for the box, not the loopback interface.
Changing perf_get_concurrent.c to use 127.0.0.1 makes it vastly faster - it went from ~35 requests/sec to 12,000-15,000 requests/sec.
It makes a big difference on OS X too - from 115-140 requests/sec on my old macbook to 600-3600 requests/sec (variation here mainly due to the large amount of other stuff I have running).
Patch attached.
Will Bryant <will.bryant@gmail.com>
To:
libmicrohttpd development and user mailinglist <libmicrohttpd@gnu.org>
Date:
Today 03:01:54 AM
Attachments:
0001-use-separate-ports-for-subsequent-tests-in-the-perf-.patch
So patch attached to use sequential port number assignments in those two perf test programs - with that and the earlier pipe shutdown patch, OS X passes all tests now.
OpenIndiana also passes all the perf tests, leaving just the SIGPIPE matter. Incidentally, performance is terrible there. I would be interested to know why - my OpenIndiana box has a modern Intel Q9505 and gets in the region of 35 requests/s in each perf test, whereas my aging Intel Core 2 Duo macbook gets 600-900 despite having half the cores. Of course we only expect the non-concurrent test to use about 1 of the cores, but both that and the concurrent test actually use only about 1% of a single CPU. This is puzzling as OpenIndiana has the very performant and scalable sunos/solaris-derived kernel and libc, so something odd is definitely going on.
Regarding the SIGPIPE, do you think a signal handler should be installed in all test programs, to implement the recommended behavior for applications, or only in those that need it?
I am sitting on the fence. I think the argument for the latter would be that one would not normally expect SIGPIPE to occur during tests that do not test out error/abort behavior, but I don't think it would normally be harmful.
(I haven't implemented the configure script integration to set the HAVE_LISTEN_SHUTDOWN conditional define for Linux etc. - can you help there? I have, for what it's worth, checked that Linux does also work using the pipe technique instead, so that seems well-portable.)
Cheers,
Will
Will Bryant <will.bryant@gmail.com>
To:
libmicrohttpd development and user mailinglist <libmicrohttpd@gnu.org>
Date:
Yesterday 03:59:23 PM
Attachments:
0001-use-a-pipe-to-signal-shutdown-to-select-and-poll.patch
I attach my current work. I have attempted to preserve and restore as much of the 'old' pipe shutdown code that was present in earlier versions of the library, but have removed the sections that aren't relevant now that there is a socket-by-socket client shutdown procedure.
On OS X, all the 'regular' tests pass. The two perf test programs both fail on a random subset of the tests with "Failed to bind to port 1081: Address already in use" (or the other ports that the perf tests use).
I initially thought that this was a problem with this patch, but after I couldn't find any problem with it I tried backporting the perf test programs to the older 0.9.7 version of libmicrohttpd, which is before the listening socket shutdown code went in, and it turned out that the older versions failed in the same way.
On OpenIndiana (oi_151a - but should behave the same as other SunOS/Solaris/OpenSolaris/Illumos kernels), the same problem occurs. (I haven't tried the backport there too.)
Linux is known to have slightly different behavior for the SOL_REUSEADDR option which might explain this, but it's quite possible that this is just a timing issue with the cleanup of the client threads in these perf test programs (if you wait for 1s between the tests then the problem goes away).
On OS X, using both debugging printfs and shelling out to lsof -itcp, I have confirmed that the listening socket has been closed successfully before the error occurs. So I am starting to suspect that it is not possible to bind to an address that was used as a server socket until all the client sockets have also been cleaned up - but that's just a theory at this point.
Anyway, as far as this patch goes, OS X looks a lot healthier - as healthy as it was in 0.9.7 (0.9.8 and 0.9.9 both failed to build entirely).
There is one further test failure on openindiana, which I believe is not related to this patch - all the same 'regular' tests pass except daemontest_get_response_cleanup, which dies due to an unhandled SIGPIPE when trying to write to the socket after killing the curl process:
Program received signal SIGPIPE, Broken pipe.
[Switching to Thread 2 (LWP 2)]
0xfed634f7 in __so_send () from /lib/libc.so.1
(gdb) bt
#0 0xfed634f7 in __so_send () from /lib/libc.so.1
#1 0xfed511d7 in _so_send () from /lib/libc.so.1
#2 0xfe7dcb26 in send () from /lib/libsocket.so.1
#3 0xfef56d29 in send_param_adapter (connection=0x20, other=0xfe99080f, i=0) at daemon.c:764
#4 0xfef54a0b in do_write (connection=0x8073ad0) at connection.c:1495
#5 0xfef54f6f in MHD_connection_handle_write (connection=0x8073ad0) at connection.c:1909
#6 0xfef57b3c in MHD_select (daemon=0x80737a8, may_block=0) at daemon.c:1367
#7 0xfef57ff4 in MHD_select_thread (cls=0x80737a8) at daemon.c:1615
#8 0xfed5f0f3 in _thrp_setup () from /lib/libc.so.1
#9 0xfed5f3a0 in ?? () from /lib/libc.so.1
#10 0xfe9a0240 in ?? ()
#11 0x00000000 in ?? ()
(gdb) info threads
* 4 Thread 2 (LWP 2) 0xfed634f7 in __so_send () from /lib/libc.so.1
3 LWP 2 0xfed634f7 in __so_send () from /lib/libc.so.1
2 Thread 1 (LWP 1) 0xfed63e87 in __waitid () from /lib/libc.so.1
1 LWP 1 0xfed63e87 in __waitid () from /lib/libc.so.1
(gdb) thread 2
[Switching to thread 2 (Thread 1 (LWP 1))]#0 0xfed63e87 in __waitid () from /lib/libc.so.1
(gdb) bt
#0 0xfed63e87 in __waitid () from /lib/libc.so.1
#1 0xfed515bd in waitid () from /lib/libc.so.1
#2 0xfed00035 in waitpid () from /lib/libc.so.1
#3 0x08051423 in kill_curl (pid=4479) at daemontest_get_response_cleanup.c:70
#4 0x0805157d in main (argc=1, argv=0x6b) at daemontest_get_response_cleanup.c:141
0001-use-a-pipe-to-signal-shutdown-to-select-and-poll.patch
Nils Durner
Christian Grothoff
Matthias Wachs