windows (mingw-w64) support for libh2o #380

martell · 2015-06-25T23:37:01Z

Hey guys,

I'm not putting this up for merging.
Everything compiles with mingw-w64 but it still doesn't quite run.
I'm leaving this here for everyone to use, debug, test :)

martell · 2015-06-25T23:38:14Z

#2 This could be considered as an updated version :)

kazuho · 2015-06-27T02:23:15Z

👍

mattn · 2015-06-27T14:12:32Z

Wow!! 👍

martell · 2015-06-27T14:27:41Z

Okay so I have done some more work on this.
I will update shortly with what I think can be merged :)

I will try to add mmap functions and readdir_r emulation to mingw-w64.

I think supporting MSVC would be a big overhaul that destroys the neatness of the project which is why I went for mingw-w64 as that it much closer to h2o because it supports posix in most places and we can keep the changes to a minimal and mostly about header includes.

martell · 2015-06-27T14:56:51Z

@kazuho: this should be good to merge.
I will find better solutions to the hackery parts in a different PR.
At least we have the ball rolling now on windows support.

Please review :)

martell · 2015-06-27T16:22:14Z

relative readdir_r patch thread for mingw-w64
https://www.mail-archive.com/mingw-w64-public@lists.sourceforge.net/msg11426.html

martell · 2015-06-28T01:23:09Z

I was wondering do you guys have an IRC channel when I could chat with @mattn and @kazuho

Currently when running the http1client example i get the following output.
"I/O timeout"

I could do with some help in fixing h2o in what it is passing

kazuho · 2015-06-29T02:47:01Z

lib/common/multithread.c

    if (cloexec_pipe(fds) != 0) {
        perror("pipe");
        abort();
    }
    fcntl(fds[1], F_SETFL, O_NONBLOCK);
+#else
+    u_long nonblock = 1;


Call to cloexec_pipe (or an equivalent function) is missing.

I presume that this is the reason why you are seeing timeout errors.

If you look a few lines above this is only relevant for the evloop implementation
#if H2O_USE_LIBUV
#else

With reguard to the evloop implementation I do need some advice here :)
In cloexec to setup fds[0] and fds[1] on windows we have to create the socket before hand
This means we need to know if it is tcp or udp a stream etc beforehand which I don't think we know at this point ?
This is done in the set_cloexec function

I think I should focus on supporting doing 1 of the 2 first and not both at the same time.
libuv seemed like the easiest one as it already supports windows and mingw-w64.
Your thoughts ?

In cloexec to setup fds[0] and fds[1] on windows we have to create the socket before hand
This means we need to know if it is tcp or udp a stream etc beforehand which I don't think we know at this point ?
This is done in the set_cloexec function

In the unix-side, fds[0] and fds[1] are initialized by calling cloexec_pipe (which in turn calls pipe(2)). So on the Windows-side, the descriptors should be initialized as well, before setting them to non-blocking mode.

The file descriptors being created are used for signalling between the threads, and the threads listen to the sockets using the event loop. So I think on Windows you would need to create a pair of TCP sockets connected to each other under my understanding that the HANDLEs created by CreatePipe cannot be passed to select.

libuv seemed like the easiest one as it already supports windows and mingw-w64.
Your thoughts ?

The H2O standalone server only supports the embedded event loop (evloop). Libuv binding exists for users of libh2o (since it is a popular event loop that is used, with rich documentation and other protocol bindings that application developers can use).

I agree that porting the libuv-side would be faster, but if your intention is to make the H2O standalone server runnable on Windows, you would need to port evloop (and it seems like that you have already done that to some extent!).

So I think on Windows you would need to create a pair of TCP sockets connected to each other under my understanding that the HANDLEs created by CreatePipe cannot be passed to select.

Yes I did this yesterday evening using the socket function and making it a stream socket with ipv4 support and tcp but it didn't change much in that the in ev-loop mode example-httpclient hangs (still stuck in the loop)

I agree that porting the libuv-side would be faster, but if your intention is to make the H2O standalone server runnable on Windows, you would need to port evloop (and it seems like that you have already done that to some extent!).

Yes I started working on that because I couldn't get the examples that use libuv working :(
I would like to get libh2o working first if possible because I want to create a few mockups with it.

I still intend to get the full h2o package working on windows but I would like to focus on the lib aspect first.

Yes I did this yesterday evening using the socket function and making it a stream socket with ipv4 support and tcp but it didn't change much in that the in ev-loop mode example-httpclient hangs (still stuck in the loop)

To which part of the callbacks does the client proceed? exmaples/libh2o/http1client.c defines a number of callbacks that are called one by one. on_connect is called first, then on_head, and on_body. You could also set breakpoints to (or emit logs in) the callback functions within socketpool.c or http1client.c to determine which callbacks are called, and which are not.

That would help you analyze the cause of the issue.

Should I focus on the evloop version first or the libuv ?

That depends on what you want to do.

If your intention is to use libh2o, then you should better concentrate on libh2o and the examples under examples/libh2o. OTOH if your intention is porting the standalone server to Windows, I would suggest concentrating on evloop and then working on the libuv-side.

Personally I did not expect somebody to start porting the standalone server to Windows, and am fascinated by how far you have achieved so far.

To which part of the callbacks does the client proceed? exmaples/libh2o/http1client.c defines a number of callbacks that are called one by one. on_connect is called first, then on_head, and on_body. You could also set breakpoints to (or emit logs in) the callback functions within socketpool.c or http1client.c to determine which callbacks are called, and which are not.

When using libuv on_connect is called and it doesn't complain and that comes from a call stack of

1 on_connect
2 call_connect_cb
3 uv_process_tcp_connect_req
4 uv_process_reqs
5 uv_run

then after that it hits on_head and exits on line 108 after fprintf "I/O timeout"

the stack for this is

1 on_head
2 on_error_before_head
3 h2o_timeout_run
4 on_timeout

After debugging in fiddler2 the http req debugger I can see that the http request is infact never made to argv[1] in my case hxxp://google.com (xx = tt)

now the only strange piece of code left is this
/uv-binding.c.h:147:25: warning: passing argument 1 of 'dup' makes integer from pointer without a cast
if ((info->fd = dup(fd)) == -1)

I tried using _get_osfhanle but it doesn't change anything :/

I'm not quite sure whats going on here exactly.
The handle is being duplicated for some reason to hold on to after close ?

If your intention is to use libh2o, then you should better concentrate on libh2o and the examples under examples/libh2o. OTOH if your intention is porting the standalone server to Windows, I would suggest concentrating on evloop and then working on the libuv-side.

I would like to get the lib working first, but I do intend to get the server working after that :)

Personally I did not expect somebody to start porting the standalone server to Windows, and am fascinated by how far you have achieved so far.

Someone was bound to try it sooner or later :D

kazuho · 2015-06-29T03:24:00Z

In addition to the in-line comments, I see following high-level issues.

configuration directives that cannot be supported on Windows (e.g. fastcgi.spawn) should not be removed; they should kept as they are and report configuration error if used
upon the start-up modes (see -m option), do we need to support daemon and master?
- master mode relies on server-starter (i.e. the start_server command) which will never work on Windows, and IMO the mode and related code should be #ifndefed for Windows
- I am not sure if daemon mode can be implemented (or how it should be implemented) on Windows
- if the modes are to be unsupported, they should also report the unsupported error in case of being used
please check that the test-suite passes, or please report the ones that cannot pass
- the easiest way to find out how it can be run is by reading .travis.yml
- EDIT: it might be better to use Cygwin for running the tests (binaries can be built using MingW)

martell · 2015-06-29T15:13:29Z

After some though I would like to narrow the scope of the PR to libh2o and libh2o-evloop.
First with focus on libh2o and then libh2o-evlopp after libh2o is working and the examples :)
I will do a seperate PR for h2o itself

All commits will be tidied squashed force re pushed once we get the lib working

EDIT: it might be better to use Cygwin for running the tests (binaries can be built using MingW)

I use msys2 for this as I am one of the maintainers http://msys2.github.io/
It is essentially a newer fork of cygwin with a linux package manager
We use pacman the arch linux package manager.
All my depends come from this and most packages are based on archlinux
Currently the h2o package is not available but the code for it is here
https://github.com/Alexpux/MINGW-packages/blob/master/mingw-w64-h2o/PKGBUILD

martell · 2015-06-29T15:44:07Z

Attached is a log of t-00unit-libuv.t.exe

https://gist.github.com/martell/30fc1fc5570f24c49673

looking at the tests for file.c

ok 31 - E:/msys64/home/Martell/h2o/t/00unit/lib/handler/file.c 703

{
    h2o_loopback_conn_t *conn = h2o_loopback_create(&ctx, ctx.globalconf->hosts);
    conn->req.input.method = h2o_iovec_init(H2O_STRLIT("HEAD"));
    conn->req.input.path = h2o_iovec_init(H2O_STRLIT("/index_txt/"));
    h2o_loopback_run_loop(conn);
    ok(conn->req.res.status == 200);
    ok(check_header(&conn->req.res, H2O_TOKEN_CONTENT_TYPE, "text/plain"));
    ok(conn->body->size == 0);
    h2o_loopback_destroy(conn);
}

not ok 33 - E:/msys64/home/Martell/h2o/t/00unit/lib/handler/file.c 712

{
    h2o_loopback_conn_t *conn = h2o_loopback_create(&ctx, ctx.globalconf->hosts);
    conn->req.input.method = h2o_iovec_init(H2O_STRLIT("HEAD"));
    conn->req.input.path = h2o_iovec_init(H2O_STRLIT("/index_txt"));
    h2o_loopback_run_loop(conn);
    ok(conn->req.res.status == 301);
    ok(check_header(&conn->req.res, H2O_TOKEN_LOCATION, "/index_txt/"));
    h2o_loopback_destroy(conn);
}

200 ok requests succeed while 301 redirects fail

conn->req.res.status is actually returned as 403 instead :(

when it hits line 112 in request.c

{
                if (req->path_normalized.len == confpath_wo_slash) {
}

this essintially becomes if(10 == 0) which is of no use to us on windows.
comparing the differences between the 2 OS's will require me to setup a linux build env

kazuho · 2015-06-30T02:53:53Z

not ok 33 - E:/msys64/home/Martell/h2o/t/00unit/lib/handler/file.c 712

The test case is asserting that the server redirects the client to a path with / appended, if the path specified by the request is a directory.

So it might be the case that S_ISDIR(st.st_mode) is not returning true as expected.

when it hits line 112 in request.c

No, that not the path that is taken by the test case. It is one of the calls to h2o_send_redirect within lib/handler/file.c that is generating the response.

comparing the differences between the 2 OS's will require me to setup a linux build env

I understand how you feel. I do not have an Windows machine, and have the same issue when I look into your code.
You can use travis to check the behavior by pushing code with debug logs.

kazuho · 2015-07-02T09:41:15Z

FYI, in #385 the select backend has been replaced by a poll-based backend. I believe that you can use WSAPoll on Windows.

martell · 2015-07-11T16:24:15Z

After doing some research into WSAPoll I came across these

curl/curl@8bad5f2a6169e2e
http://curl.haxx.se/mail/lib-2012-07/0311.html
http://daniel.haxx.se/blog/2012/10/10/wsapoll-is-broken/

This probably is an issue on cygwin also if it uses WSAPoll to emulate the poll function

kazuho · 2015-07-14T03:00:33Z

@martell Thank you for the information.

If I read correctly the difference of the behavior is limited to when connecting to a non-responding server. If that is right then the impact of the problem to H2O will be fairly limited, since it mostly runs as a server (and if it connects to an upstream server when acting as a proxy, it connects to a predefined server that can be expected to be running and up).

That said, I would not mind if you keep the select-based implementation for Windows considering the fact that the platform is capable of modifying FD_SETSIZE at compile time (cf. https://msdn.microsoft.com/en-us/library/ms740141.aspx). We would need to increase the number to something very large though.

martell · 2015-11-28T23:48:01Z

Just following up on this old PR to update the status.
I got it working at one point in the end but it seems like its overkill to the code base of the project to support win32 natively so I've abandoned this approach in favour of a new one.

I've been working with a new linux emulation layer called midipix which is like cygwin but it aparently performs even better then mingw-w64 for networked projects like this. It also uses musl libc so it already works. I'll post an update when that project gets closer to release.

kazuho · 2015-11-30T00:02:01Z

I've been working with a new linux emulation layer called midipix which is like cygwin but it aparently performs even better then mingw-w64 for networked projects like this. It also uses musl libc so it already works. I'll post an update when that project gets closer to release.

Sounds interesting! Looking forward to hearing the updates.

martell added 3 commits June 27, 2015 15:39

basic windows support

57464da

fix gmtime_r for mingw-w64

99a4510

use gnu_printf and gnu_scanf for mingw-w64

52dbba6

martell force-pushed the mingw branch from dd233e3 to 52dbba6 Compare June 27, 2015 14:41

kazuho reviewed Jun 29, 2015
View reviewed changes

martell added 2 commits June 29, 2015 13:25

ignore O_CLOEXEC in file.c for windows

a5f2eba

address review

c2c9664

martell changed the title ~~windows (mingw-w64) support for h2o~~ windows (mingw-w64) support for libh2o Jun 29, 2015

martell closed this Nov 28, 2015

LakhanRathi92 mentioned this pull request Mar 21, 2016

Visual studio2015 martell/h2o#1

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

windows (mingw-w64) support for libh2o #380

windows (mingw-w64) support for libh2o #380

martell commented Jun 25, 2015

martell commented Jun 25, 2015

kazuho commented Jun 27, 2015

mattn commented Jun 27, 2015

martell commented Jun 27, 2015

martell commented Jun 27, 2015

martell commented Jun 27, 2015

martell commented Jun 28, 2015

kazuho Jun 29, 2015

martell Jun 29, 2015

kazuho Jun 29, 2015

kazuho Jun 29, 2015

martell Jun 29, 2015

martell Jun 29, 2015

kazuho Jun 29, 2015

martell Jun 29, 2015

kazuho commented Jun 29, 2015

martell commented Jun 29, 2015

martell commented Jun 29, 2015

kazuho commented Jun 30, 2015

kazuho commented Jul 2, 2015

martell commented Jul 11, 2015

kazuho commented Jul 14, 2015

martell commented Nov 28, 2015

kazuho commented Nov 30, 2015

windows (mingw-w64) support for libh2o #380

windows (mingw-w64) support for libh2o #380

Conversation

martell commented Jun 25, 2015

martell commented Jun 25, 2015

kazuho commented Jun 27, 2015

mattn commented Jun 27, 2015

martell commented Jun 27, 2015

martell commented Jun 27, 2015

martell commented Jun 27, 2015

martell commented Jun 28, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kazuho commented Jun 29, 2015

martell commented Jun 29, 2015

martell commented Jun 29, 2015

kazuho commented Jun 30, 2015

kazuho commented Jul 2, 2015

martell commented Jul 11, 2015

kazuho commented Jul 14, 2015

martell commented Nov 28, 2015

kazuho commented Nov 30, 2015