Skip to content

JSON documents numeric field values are automatically rounding #1216

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nhealthlive opened this issue Jul 10, 2024 · 6 comments
Open

JSON documents numeric field values are automatically rounding #1216

nhealthlive opened this issue Jul 10, 2024 · 6 comments

Comments

@nhealthlive
Copy link

I have a JSON document in the Redis stack, There's a field surId which holds a long-range numeric value, When I set the value, it automatically rounds to x000

That is if I set field surId as 8301034833169298414 it rounds to 8301034833169298000

If I set surId as 8301034833169297919 it rounds to 8301034833169297000

This can be tried from the Redis insight.

image

Is there any range limit for NumericField? Any workarounds that you can suggest?

  • Module versions
    RediSearch v. 2.8.9
    RedisJSON v. 2.6.7
    Redis Standalone Version: 7.2.3
@LiorKogan
Copy link
Member

LiorKogan commented Jul 10, 2024

That's the expected behavior.

Note the JSON standard specifies:

This specification allows implementations to set limits on the range and precision of numbers accepted. Since software that implements IEEE 754 binary64 (double precision) numbers [IEEE754] is generally available and widely used, good interoperability can be achieved by implementations that expect no more precision or range than these provide, in the sense that implementations will approximate JSON numbers within the expected precision.

For RedisJSON, numeric values are stored as a 64-bit floating point, a 64-bit signed integer, or a 64-bit unsigned integer. The exact behavior is explained here.

You should upgrade RedisJSON to 2.6.10 or later (or Redis Stack to 7.2.0-v10 or later) to avoid a possible crash described in that issue.

If you use integers larger than 2^64-1 and can't afford to lose precision, consider storing them as strings.

@domints
Copy link

domints commented Oct 16, 2024

@LiorKogan Hi, thanks for this explanation. However I think this functionality of storing af f64, i64 or u64 depending on needs might be not working correctly.

I'm on RedisJSON 2.6.10 and I am seeing this behaviour:
obraz
Wolfram Alpha confirms that this number is in bounds of signed 64 bit integer, it also comes from .NET's long type, so in any case it'd be cut or otherwise caught in .NET.

Here's the command, so you can copy-paste it in your instance or something:

"JSON.SET" "GtfsProvider:Vehicle_:Krakow:Tram:-1188950299261208742" "." "{"Id":"Krakow:Tram:-1188950299261208742","City":1,"UniqueId":-1188950299261208742,"GtfsId":129,"SideNo":"HW129","IsHeuristic":false,"HeuristicScore":0,"ModelName":"E1","ModelLowFloor":1,"ModelType":1}"

Is there any way to make this work or do I really have to make it string and parse/unparse each time? I also assume int would take less space?

@LiorKogan
Copy link
Member

@domints thank you, this seems to be an issue with Redis Insight's browser.
I checked from redis-cli and Redis Insight's Workbench and the result is correct.
I'll report it to the Redis Insight team.

> JSON.SET x . '{"Id":"Krakow:Tram:-1188950299261208742","City":1,"UniqueId":-1188950299261208742,"GtfsId":129,"SideNo":"HW129","IsHeuristic":false,"HeuristicScore":0,"ModelName":"E1","ModelLowFloor":1,"ModelType":1}'
OK

> JSON.GET x .UniqueId
"-1188950299261208742"

@domints
Copy link

domints commented Oct 16, 2024

@LiorKogan that is indeed true! Thank you. Although this worries me, because I thought this was source of my problems. Maybe Redis.OM library also misinterprets it...

@domints
Copy link

domints commented Oct 16, 2024

@LiorKogan Would this be a problem with RediSearch or RedisJSON? :D

> "FT.SEARCH" "storevehicle-idx" "(((@City:[1 1]) (@ModelType:[1 1])) (@UniqueId:[-1188950299261208216 -1188950299261208216]))" "LIMIT" "0" "1"
   1) "62"
   2) "GtfsProvider:Vehicle_:Krakow:Tram:-1188950299261208446"
   3) 1) "$"
      2) "{\"Id\":\"Krakow:Tram:-1188950299261208446\",\"City\":1,\"UniqueId\":-1188950299261208446,\"GtfsId\":715,\"SideNo\":\"HY715\",\"IsHeuristic\":false,\"HeuristicScore\":0,\"ModelName\":\"Stadler Tango II\",\"ModelLowFloor\":3,\"ModelType\":1}"
      
> JSON.GET GtfsProvider:Vehicle_:Krakow:Tram:-1188950299261208216
"{\"Id\":\"Krakow:Tram:-1188950299261208216\",\"City\":1,\"UniqueId\":-1188950299261208216,\"GtfsId\":909,\"SideNo\":\"RG909\",\"IsHeuristic\":false,\"HeuristicScore\":0,\"ModelName\":\"2014N\",\"ModelLowFloor\":3,\"ModelType\":1}"

Essentially, I'm (what I mean by this is Redis.OM library) searching by this UniqueID field, it returns different entry than what I get if I do direct key fetch. Is there something wrong?
I think it's probably related to this float thingy too.

EDIT:
I'm pretty sure it's related to the float thingy :D Here's how JS in browser treats those two numbers:
obraz

@LiorKogan
Copy link
Member

LiorKogan commented Oct 17, 2024

I would assume RediSearch's numeric indexes do not use the same numeric types, I64/U64/F64, as in RedisJSON.

@MeirShpilraien @oshadmi - can you please provide more details?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants