Skip to content

[BUG] After creating index Redis Stack 7.2.3 crashed #1199

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
ghczko opened this issue Mar 28, 2024 · 4 comments
Open

[BUG] After creating index Redis Stack 7.2.3 crashed #1199

ghczko opened this issue Mar 28, 2024 · 4 comments

Comments

@ghczko
Copy link

ghczko commented Mar 28, 2024

Hello,

I am facing a crashing Redis stack when creating index for searching in JSON. In Redis stack I have around 100k JSON records.
When I create a index with NodeJS library the Redis Stack crash

13509:M 28 Mar 2024 12:50:43.973 * <module> Scanning index pool:index in background
thread '<unnamed>' panicked at json_path/src/json_node.rs:238:24:
called `Option::unwrap()` on a `None` value
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
fatal runtime error: failed to initiate panic, error 5
=== REDIS BUG REPORT START: Cut & paste starting from here ===
13509:M 28 Mar 2024 12:50:44.169 # Redis 7.2.3 crashed by signal: 6, si_code: 0
13509:M 28 Mar 2024 12:50:44.169 # Crashed running the instruction at: 0x1872060dc

------ STACK TRACE ------
EIP:
0   libsystem_kernel.dylib              0x00000001872060dc __pthread_kill + 8

Backtrace:
0   libsystem_platform.dylib            0x000000018726da24 _sigtramp + 56
1   libsystem_pthread.dylib             0x000000018723dcc0 pthread_kill + 288
2   libsystem_c.dylib                   0x0000000187149a40 abort + 180
3   rejson.so                           0x0000000101f62f9c _ZN108_$LT$std..sys..unix..locks..pthread_mutex..AllocatedMutex$u20$as$u20$std..sys_common..lazy_box..LazyInit$GT$4init17hc1d68b7de7fd100eE + 0
4   rejson.so                           0x0000000101f616cc _ZN3std12backtrace_rs9symbolize6Symbol4name17hc8326d09aeb77feaE + 0
5   rejson.so                           0x0000000101f61610 _ZN3std9panicking20rust_panic_with_hook17h84c8637cb6e56008E + 596
6   rejson.so                           0x0000000101f61368 _ZN3std9panicking19begin_panic_handler28_$u7b$$u7b$closure$u7d$$u7d$17h25482adda06c7b7fE + 144
7   rejson.so                           0x0000000101f60518 _ZN3std10sys_common9backtrace15output_filename17h60eb3d725954b704E + 0
8   rejson.so                           0x0000000101f61108 _ZN90_$LT$std..panicking..begin_panic_handler..PanicPayload$u20$as$u20$core..panic..BoxMeUp$GT$8take_box17hdcefff1827414461E + 0
9   rejson.so                           0x0000000101f8cca8 _ZN4core9panicking18panic_nounwind_fmt17ha4f6bbf8fe9fb5aeE + 0
10  rejson.so                           0x0000000101f8cd18 _ZN4core9panicking14panic_nounwind17ha756411eedbc9594E + 0
11  rejson.so                           0x0000000101e72bb4 _ZN9json_path9json_node87_$LT$impl$u20$json_path..select_value..SelectValue$u20$for$u20$ijson..value..IValue$GT$10get_double17h5b3fea498d2eb9bfE + 0
12  rejson.so                           0x0000000101e3ee98 JSONAPI_getInt + 72
13  redisearch.so                       0x000000010191d004 JSON_StoreInDocField + 208
14  redisearch.so                       0x000000010191d4b4 JSON_LoadDocumentField + 424
15  redisearch.so                       0x00000001019046e0 Document_LoadSchemaFieldJson + 304
16  redisearch.so                       0x0000000101947ed8 IndexSpec_UpdateDoc + 392
17  redisearch.so                       0x0000000101949f9c Indexes_ScanProc + 308
18  redis-server                        0x0000000100633998 moduleScanCallback + 296
19  redis-server                        0x00000001005350b4 dictScan + 328
20  redis-server                        0x0000000100633840 RM_Scan + 88
21  redisearch.so                       0x0000000101945ee8 Indexes_ScanAndReindexTask + 276
22  redisearch.so                       0x00000001018de34c thread_do + 752
23  libsystem_pthread.dylib             0x000000018723e034 _pthread_start + 136
24  libsystem_pthread.dylib             0x0000000187238e3c thread_start + 8

------ REGISTERS ------
13509:M 28 Mar 2024 12:50:44.176 #
x0:0000000000000000 x1:0000000000000000 x2:0000000000000000 x3:0000000000000000
x4:0000000000000000 x5:0000000000000001 x6:0000000000000005 x7:0000000000000150
x8:27bfbc5ce9aa4c4a x9:27bfbc5d99d73c4a x10:0000000000110000 x11:0000000102067970
x12:0000000000000000 x13:0000000000000000 x14:0000000000000000 x15:0000000187243cc4
x16:0000000000000148 x17:00000001e6a8f340 x18:0000000000000000 x19:0000000000000006
x20:00000001707d7000 x21:0000000000001303 x22:00000001707d70e0 x23:000000010206c340
x24:0000000187246620 x25:0000000101fed590 x26:0000000000000007 x27:0000000101c03dd0
x28:000000000000000a fp:00000001707d6770 lr:000000018723dcc0
sp:00000001707d6750 pc:00000001872060dc cpsr:40001000

13509:M 28 Mar 2024 12:50:44.176 # (00000001707d675f) -> 0000000101f616cc
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d675e) -> 00000001707d6820
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d675d) -> 315b800101f62f9c
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d675c) -> 00000001707d67c0
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d675b) -> 0000000000000001
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d675a) -> 00000001020678c0
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d6759) -> 00000000fffff9df
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d6758) -> 0000000000000000
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d6757) -> ffffffff01f5e65c
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d6756) -> 00000001707d67c0
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d6755) -> ce2b000187149a40
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d6754) -> 00000001707d67b0
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d6753) -> 00000001707d7000
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d6752) -> 00000001707d6798
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d6751) -> 00000001707d68f0
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d6750) -> 000000010206c180

------ INFO OUTPUT ------
# Server
redis_version:7.2.3
redis_git_sha1:7f4bae81
redis_git_dirty:0
redis_build_id:a97542f11b1bfce4
redis_mode:standalone
os:Darwin 23.2.0 arm64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:kqueue
atomicvar_api:c11-builtin
gcc_version:4.2.1
process_id:13509
process_supervised:no
run_id:2110b56b87d13088e006830fcefdc9df242de31b
tcp_port:6379
server_time_usec:1711626644167971
uptime_in_seconds:37
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:350612
executable:/opt/homebrew/Caskroom/redis-stack-server/7.2.0-v6/bin/redis-server
config_file:/opt/homebrew/Caskroom/redis-stack-server/7.2.0-v6/etc/redis-stack.conf
io_threads_active:0
listener0:name=tcp,bind=*,bind=-::*,port=6379

# Clients
connected_clients:2
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:16
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0

# Memory
used_memory:89174336
used_memory_human:85.04M
used_memory_rss:93552640
used_memory_rss_human:89.22M
used_memory_peak:89174336
used_memory_peak_human:85.04M
used_memory_peak_perc:102.29%
used_memory_overhead:7427472
used_memory_startup:1435440
used_memory_dataset:81746864
used_memory_dataset_perc:93.17%
allocator_allocated:87174880
allocator_active:93520896
allocator_resident:93520896
total_system_memory:17179869184
total_system_memory_human:16.00G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:216
used_memory_scripts:216
used_memory_scripts_human:216B
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.07
allocator_frag_bytes:6346016
allocator_rss_ratio:1.00
allocator_rss_bytes:0
rss_overhead_ratio:1.00
rss_overhead_bytes:31744
mem_fragmentation_ratio:1.07
mem_fragmentation_bytes:6377760
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:3648
mem_cluster_links:0
mem_aof_buffer:0
mem_allocator:libc
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:4
rdb_bgsave_in_progress:0
rdb_last_save_time:1711626607
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:0
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:123487
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:3
total_commands_processed:22
instantaneous_ops_per_sec:6
total_net_input_bytes:2334
total_net_output_bytes:177296
total_net_repl_input_bytes:0
total_net_repl_output_bytes:0
instantaneous_input_kbps:1.07
instantaneous_output_kbps:0.05
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:254007
keyspace_misses:2
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:3
dump_payload_sanitizations:0
total_reads_processed:13
total_writes_processed:15
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:3
reply_buffer_expands:0
eventloop_cycles:370
eventloop_duration_sum:60962
eventloop_duration_cmd_sum:17170
instantaneous_eventloop_cycles_per_sec:13
instantaneous_eventloop_duration_usec:118
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0

# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:91623a63f06557c641a812556aac70046b3eeaf5
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:0.096157
used_cpu_user:0.474580
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000

# Modules
module:name=search,ver=20809,api=1,filters=0,usedby=[],using=[ReJSON],options=[handle-io-errors]
module:name=bf,ver=20608,api=1,filters=0,usedby=[],using=[],options=[]
module:name=RedisCompat,ver=1,api=1,filters=0,usedby=[],using=[],options=[]
module:name=ReJSON,ver=20607,api=1,filters=0,usedby=[search],using=[],options=[handle-io-errors]
module:name=redisgears_2,ver=20014,api=1,filters=0,usedby=[],using=[],options=[]
module:name=timeseries,ver=11009,api=1,filters=0,usedby=[],using=[],options=[]

# Commandstats
cmdstat_config|get:calls=1,usec=6,usec_per_call=6.00,rejected_calls=0,failed_calls=0
cmdstat_quit:calls=1,usec=0,usec_per_call=0.00,rejected_calls=0,failed_calls=0
cmdstat_client|setinfo:calls=2,usec=1,usec_per_call=0.50,rejected_calls=0,failed_calls=0
cmdstat_client|setname:calls=1,usec=2,usec_per_call=2.00,rejected_calls=0,failed_calls=0
cmdstat_dbsize:calls=1,usec=1339,usec_per_call=1339.00,rejected_calls=0,failed_calls=0
cmdstat_FT.SEARCH:calls=0,usec=0,usec_per_call=0.00,rejected_calls=1,failed_calls=0
cmdstat_info:calls=5,usec=1196,usec_per_call=239.20,rejected_calls=0,failed_calls=0
cmdstat_FT.CREATE:calls=2,usec=2596,usec_per_call=1298.00,rejected_calls=0,failed_calls=0
cmdstat_scan:calls=1,usec=10625,usec_per_call=10625.00,rejected_calls=0,failed_calls=0
cmdstat_set:calls=2,usec=86,usec_per_call=43.00,rejected_calls=0,failed_calls=0
cmdstat_unlink:calls=2,usec=9,usec_per_call=4.50,rejected_calls=0,failed_calls=0
cmdstat_get:calls=2,usec=1326,usec_per_call=663.00,rejected_calls=0,failed_calls=0
cmdstat_FT.DROPINDEX:calls=2,usec=69,usec_per_call=34.50,rejected_calls=0,failed_calls=2

# Errorstats
errorstat_LOADING:count=1
errorstat_Unknown:count=2

# Latencystats
latency_percentiles_usec_config|get:p50=6.015,p99=6.015,p99.9=6.015
latency_percentiles_usec_quit:p50=0.001,p99=0.001,p99.9=0.001
latency_percentiles_usec_client|setinfo:p50=0.001,p99=1.003,p99.9=1.003
latency_percentiles_usec_client|setname:p50=2.007,p99=2.007,p99.9=2.007
latency_percentiles_usec_dbsize:p50=1343.487,p99=1343.487,p99.9=1343.487
latency_percentiles_usec_info:p50=8.031,p99=1122.303,p99.9=1122.303
latency_percentiles_usec_FT.CREATE:p50=290.815,p99=2310.143,p99.9=2310.143
latency_percentiles_usec_scan:p50=10682.367,p99=10682.367,p99.9=10682.367
latency_percentiles_usec_set:p50=17.023,p99=69.119,p99.9=69.119
latency_percentiles_usec_unlink:p50=1.003,p99=8.031,p99.9=8.031
latency_percentiles_usec_get:p50=9.023,p99=1318.911,p99.9=1318.911
latency_percentiles_usec_FT.DROPINDEX:p50=5.023,p99=64.255,p99.9=64.255

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=123489,expires=0,avg_ttl=0

------ CLIENT LIST OUTPUT ------
id=14 addr=[::1]:49949 laddr=[::1]:6379 fd=12 name= age=36 idle=36 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1824 events=r cmd=FT.SEARCH user=default redir=-1 resp=2 lib-name= lib-ver=
id=15 addr=127.0.0.1:50001 laddr=127.0.0.1:6379 fd=10 name=redisinsight-browser-93b67723 age=25 idle=25 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1824 events=r cmd=scan user=default redir=-1 resp=2 lib-name= lib-ver=

------ MODULES INFO OUTPUT ------
# search_version
search_version:2.8.9
search_redis_version:7.2.3 - oss

# search_index
search_number_of_indexes:2

# search_fields_statistics
search_fields_text:Text=3
search_fields_numeric:Numeric=2,Sortable=2
search_fields_tag:Tag=12

# search_dialect_statistics
search_dialect_1:0
search_dialect_2:0
search_dialect_3:0
search_dialect_4:0

# search_runtime_configurations
search_concurrent_mode:OFF
search_enableGC:ON
search_minimal_term_prefix:2
search_maximal_prefix_expansions:200
search_query_timeout_ms:500
search_timeout_policy:return
search_cursor_read_size:1000
search_cursor_max_idle_time:300000
search_max_doc_table_size:1000000
search_max_search_results:10000
search_max_aggregate_results:10000
search_search_pool_size:20
search_index_pool_size:8
search_gc_scan_size:100
search_min_phonetic_term_length:3

# ReJSON_trace
ReJSON_backtrace:   0: backtrace::capture::Backtrace::new
   1: redis_module::basic_info_command_handler
   2: rejson::__info_func
   3: _modulesCollectInfo
   4: _printCrashReport
   5: _sigsegvHandler
   6: __platform_memmove
   7: __pthread_atfork_prepare_handlers
   8: <unknown>
   9: std::sys::unix::abort_internal
  10: _rust_panic
  11: std::panicking::rust_panic_with_hook
  12: std::panicking::begin_panic_handler::{{closure}}
  13: std::sys_common::backtrace::__rust_end_short_backtrace
  14: _rust_begin_unwind
  15: core::panicking::panic_fmt
  16: core::panicking::panic
  17: json_path::json_node::<impl json_path::select_value::SelectValue for ijson::value::IValue>::get_long
  18: _JSONAPI_getInt
  19: _JSON_StoreInDocField
  20: _JSON_LoadDocumentField
  21: _Document_LoadSchemaFieldJson
  22: _IndexSpec_UpdateDoc
  23: _Indexes_ScanProc
  24: _moduleScanCallback
  25: _dictScan
  26: _RM_Scan
  27: _Indexes_ScanAndReindexTask
  28: _thread_do
  29: __pthread_joiner_wake


# redisgears_2_trace
redisgears_2_backtrace:   0: backtrace::capture::Backtrace::new
   1: redis_module::basic_info_command_handler
   2: redisgears::gears_module::__info_func
   3: _modulesCollectInfo
   4: _printCrashReport
   5: _sigsegvHandler
   6: __platform_memmove
   7: __pthread_atfork_prepare_handlers
   8: <unknown>
   9: std::sys::unix::abort_internal
  10: _rust_panic
  11: std::panicking::rust_panic_with_hook
  12: std::panicking::begin_panic_handler::{{closure}}
  13: std::sys_common::backtrace::__rust_end_short_backtrace
  14: _rust_begin_unwind
  15: core::panicking::panic_fmt
  16: core::panicking::panic
  17: json_path::json_node::<impl json_path::select_value::SelectValue for ijson::value::IValue>::get_long
  18: _JSONAPI_getInt
  19: _JSON_StoreInDocField
  20: _JSON_LoadDocumentField
  21: _Document_LoadSchemaFieldJson
  22: _IndexSpec_UpdateDoc
  23: _Indexes_ScanProc
  24: _moduleScanCallback
  25: _dictScan
  26: _RM_Scan
  27: _Indexes_ScanAndReindexTask
  28: _thread_do
  29: __pthread_joiner_wake


# redisgears_2_UninitialisedBackends
redisgears_2_backend_name:js

------ CONFIG DEBUG OUTPUT ------
replica-read-only yes
io-threads-do-reads no
slave-read-only yes
proto-max-bulk-len 512mb
lazyfree-lazy-server-del no
lazyfree-lazy-expire no
lazyfree-lazy-user-del no
client-query-buffer-limit 1gb
lazyfree-lazy-eviction no
repl-diskless-load disabled
sanitize-dump-payload no
io-threads 1
repl-diskless-sync yes
lazyfree-lazy-user-flush no
list-compress-depth 0
activedefrag no

------ DUMPING CODE AROUND EIP ------
Symbol: __pthread_kill (base: 0x1872060d4)
Module: /usr/lib/system/libsystem_kernel.dylib (base 0x1871fc000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x1872060d4 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
13509:M 28 Mar 2024 12:50:44.200 # dump of function (hexdump of 136 bytes):
102980d2011000d4030100547f2303d5fd7bbfa9fd03009151e0ff97bf030091fd7bc1a8ff0f5fd6c0035fd6700a80d2011000d4030100547f2303d5fd7bbfa9fd03009146e0ff97bf030091fd7bc1a8ff0f5fd6c0035fd6703080d2011000d4030100547f2303d5fd7bbfa9fd0300913be0ff97bf030091fd7bc1a8ff0f5fd6c0035fd664010034

=== REDIS BUG REPORT END. Make sure to include from START to END. ===
@ghczko ghczko mentioned this issue Mar 28, 2024
Closed
@meiravgri
Copy link
Contributor

Hi @ghczko
we suspect it's related to the datatype.
Can you please provide an example of a json record and the command that caused the crash?
Thanks

@ephraimfeldblum
Copy link
Collaborator

ephraimfeldblum commented Mar 29, 2024 •
edited
Loading

If there are numbers in the document within the range (INT64_MAX, UINT64_MAX], this issue is known and should be fixed in the next stack release.

@ghczko
Copy link
Author

ghczko commented Mar 29, 2024

I confirm that the issue is related with range (INT64_MAX, UINT64_MAX). INT64 is limitation in my use case, is it planned to change it?

@mezga0153
Copy link

it seems i'm also hitting this bug, i can reproduce it with:

FLUSHALL
JSON.SET test:entry $ '{"bigNumber":9223372036854775808}'
FT.CREATE test:index ON JSON PREFIX 1 test: SCHEMA '$["bigNumber"]' AS bigNumber NUMERIC NOINDEX

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mezga0153 @ghczko @ephraimfeldblum @meiravgri