Skip to content

Accept raw result without parsing JSON #194

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
uasan opened this issue Jun 21, 2021 · 12 comments
Closed

Accept raw result without parsing JSON #194

uasan opened this issue Jun 21, 2021 · 12 comments
Labels
enhancement New feature or request

Comments

@uasan
Copy link

uasan commented Jun 21, 2021

Hello.
We have many endpoints that do not need to parse the json type and receive it as an object, because these endpoints send this values as a JSON.stringify(values) to the client socket.

SELECT json_agg(values) AS values FROM ...

In this case, two expensive blocked operations, JSON.parse and JSON.stringify, are constantly performed.
Ideally, I would like to be able to receive the response in the form of a binary buffer, without parsing json, to send this buffer to the client.

How do you like the idea?
Thanks.

@uasan uasan changed the title Accept raw response without parsing JSON Accept raw result without parsing JSON Jun 21, 2021
@porsager
Copy link
Owner

It's right along what I just played with regarding sending bytea as binary, the issue is deciding return results as binary or text or mixed.

I'll play a bit more with it, but until then you could get a long way simply casting json(b) to text in your query ::text and passing it directly.

@porsager
Copy link
Owner

You can also override the json types and set the serializer to null which will disable JSON.parse

@uasan
Copy link
Author

uasan commented Jun 21, 2021

I'll play a bit more with it, but until then you could get a long way simply casting json(b) to text in your query ::text and passing it directly.

I know about this ugly hack )

You can also override the json types and set the serializer to null which will disable JSON.parse

I can do this, only for some queries, or only settings for the connection?

@porsager
Copy link
Owner

types are set per instance.

Why do you find casting to text ugly? Now I'm curious if there's any overhead compared to skipping JSON.parse 😋

@uasan
Copy link
Author

uasan commented Jun 21, 2021

  1. because I have to edit the original queries
  2. this will worsen the understanding of the SQL code for people who will support it and there may be a situation when I need to execute the same prepared request in different places and somewhere I want to get the result in the form of an object, I need to call JSON.parse there )

@porsager
Copy link
Owner

porsager commented Jun 22, 2021

That sounds reasonable.

How about adding the following .raw() method?

sql`select 1 as x, 2 as x`.raw()

That would resolve to this:

[
  [ <Buffer 31>, <Buffer 32> ],
  count: 1,
  columns: [
    { name: 'x', parser: [Function: parse], type: 23 },
    { name: 'x', parser: [Function: parse], type: 23 }
  ],
  command: 'SELECT',
  state: { status: 'I', pid: 62257, secret: 1364597943 }
]

This would help some other requests as well who wanted rows as arrays instead of objects, and allow same named columns to be retrieved.

For json like in your example

sql`select '{"a":"b"}'::jsonb`.raw()

you'd get this:

[
  [ <Buffer 7b 22 61 22 3a 20 22 62 22 7d> ],
  count: 1,
  columns: [ { name: 'jsonb', parser: [Function: parse], type: 3802 } ],
  command: 'SELECT',
  state: { status: 'I', pid: 62208, secret: -1778630945 }
]

@uasan
Copy link
Author

uasan commented Jun 23, 2021

Great, I like your interfaces

porsager added a commit that referenced this issue Jun 23, 2021
@porsager
Copy link
Owner

Would you mind trying out the #raw branch to see if it works well for you?

@uasan
Copy link
Author

uasan commented Jun 23, 2021

How about unsafe method, how to get buffer?

@porsager
Copy link
Owner

porsager commented Jun 23, 2021

Same way :) the methods on a tagged query and an unsafe query are the same:

const [[buffer]] = await sql.unsafe('select '{"a":"b"}'::jsonb').raw()

@uasan
Copy link
Author

uasan commented Jun 23, 2021

Would you mind trying out the #raw branch to see if it works well for you?

I checked in my test version:

const [[buffer]] = await sql`SELECT '{"key": "value"}'::jsonb`.raw()
console.log(JSON.parse(buffer))

Everything works fine, thanks!

P.S.
Unfortunately, I cannot check the performance in real conditions, we do not have enough of this functionality for production #170

@porsager
Copy link
Owner

porsager commented Jun 25, 2021

I still need some performance improvements around connection handling, but copy from / to works now https://github.com/porsager/postgres/tree/copy

@porsager porsager added the enhancement New feature or request label Jul 30, 2021
porsager added a commit that referenced this issue Aug 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants