allow read of chunked request in lua #488

rzenker · 2015-04-10T19:10:08Z

No description provided.

agentzh · 2015-04-10T23:02:34Z

@rzenker Thanks for looking into this. Unfortunately this is not sufficient. We need to parse the chunked encoded data ourselves (we could reuse the chunked request body parser in the nginx, but not much else).

rzenker · 2015-04-11T02:44:10Z

@agentzh Does it need to be? Your resty-http chunked body reader already works with it this way. Also, in my case I need the raw encoded body to forward on as unmolested as possible. If it forces parse outside of lua it removes the option for people to use that.

agentzh · 2015-04-11T03:47:02Z

@rzenker Well,

I never wrote any resty-http libraries. You must be referring to some 3rd-party libraries.
I don't think it's really useful to expose the chunked encoding to the Lua land. I think 99% of the time it just just complicates the caller's code.
If you really need raw chunked request bodies, then you should use the raw request cosocket API instead (i.e., ngx.req.socket(true)). See https://github.com/openresty/lua-nginx-module/blob/master/t/116-raw-req-socket.t#L786

rzenker · 2015-04-11T16:31:19Z

@agentzh

Ah, I was confused, but referring to https://github.com/pintsized/lua-resty-http.

I did try to use raw socket first, but then it requires me to handle all the response headers differently then I do with a non-chunked request. If there was a read-only raw that still let the normal nginx responses to be sent that would be great.

I am not understanding the downside to at least allow the body to be read via the non-raw socket right now as opposed to returning the no chunked support error. I could see a case for not wanting to handle the raw chunked body in lua for ease-of-use, but in my case I am just forwarding the request to the backend (mostly unmodified besides new headers). If it returned just the body, my forwarding code would have to support re-chunking the request to the backend.

If I can't convince you of the utility of this option, would you accept a PR that took another argument to ngx.req.socket(false, true) style to either give you the raw body (2nd arg true), or a un-chunk-encoded version (2nd arg false, default)?

agentzh · 2015-04-12T22:24:04Z

@rzenker That lua-resty-http library is written by @pintsized and is a 3rd-party OpenResty library. It was used for upstream HTTP communications, instead of downstream (and we're talking about downstream here).

Will you explain why you want to parse the chunked request bodies yourself instead of letting ngx_lua (or nginx core) taking care of that automatically for you?

Regarding the ngx.req.socket(false, true) proposal, no, it looks ugly to me without outstanding reasons. If you want raw downstream cosockets, then you should want both directions (requests and responses), as in the case of the WebSocket protocol.

rzenker · 2015-04-13T15:29:52Z

@agentzh My use-case is that i'm proxying the request to a backend server in lua. If I can just read the still-chunk encoded response off and then no need to re-chunk-encode it for the backend.

I was pointing out in the https://github.com/pintsized/lua-resty-http/blob/master/lib/resty/http.lua#L286 it already has a compatible reader for chunked if ngx_lua would allow it to be read (as opposed to just refusing). If users need to read a chunked body, but not the chunk encoded version, they could just use that.

If the only option is to force the chunk decoding inside nginx, then a dev needs to create another lua class to re-chunk encode it if they're sending to a backend via lua like in my case.

agentzh · 2015-04-13T22:12:00Z

@rzenker For your use case, you can just assemble the chunked body yourself. That overhead is minimal. I don't see a reason to give you the original chunked data just for that. And ngx_lua still needs to parse the chunked request body somehow if even it does give you the raw chunked data because otherwise it has no ways to determine the end of the request body stream, which MAY cause problems in pipelined or keep-alive requests.

mergify · 2020-06-26T00:41:53Z