numpy crash #15603

wadeliuyi · 2020-02-19T04:02:57Z

(gdb) bt
#0  PyObject_Malloc (nbytes=80) at /usr/src/debug/Python-2.7.5/Objects/obmalloc.c:804
#1  0x00007f1d75c8b792 in array_alloc (type=0x7f1d761e9dc0 <PyArray_Type>, __NPY_UNUSED_TAGGEDnitems=<optimized out>) at numpy/core/src/multiarray/arrayobject.c:1767
#2  0x00007f1d75cece12 in PyArray_NewFromDescr_int (subtype=0x7f1d761e9dc0 <PyArray_Type>, descr=0x7f1d761eab00 <DOUBLE_Descr>, nd=1, dims=0x7f1b94180e80, strides=<optimized out>, data=0x0, flags=0,
    obj=0x0, base=0x0, zeroed=0, allow_emptystring=0) at numpy/core/src/multiarray/ctors.c:1003
#3  0x00007f1d75ced393 in PyArray_NewFromDescrAndBase (base=0x0, obj=obj@entry=0x0, flags=0, data=data@entry=0x0, strides=strides@entry=0x0, dims=<optimized out>, nd=<optimized out>,
    descr=<optimized out>, subtype=<optimized out>) at numpy/core/src/multiarray/ctors.c:1181
#4  PyArray_NewFromDescr (subtype=<optimized out>, descr=<optimized out>, nd=<optimized out>, dims=<optimized out>, strides=strides@entry=0x0, data=data@entry=0x0, flags=0, obj=obj@entry=0x0)
    at numpy/core/src/multiarray/ctors.c:1166
#5  0x00007f1d75eea8d7 in execute_legacy_ufunc_loop (ufunc=0x18a6620, trivial_loop_ok=trivial_loop_ok@entry=1, op=op@entry=0x7f1b9fffd480, dtypes=dtypes@entry=0x7f1b9fffc1e0, order=NPY_KEEPORDER,
    buffersize=8192, arr_prep=arr_prep@entry=0x7f1b9fffc2e0, full_args=..., op_flags=op_flags@entry=0x7f1b9fffc0e0) at numpy/core/src/umath/ufunc_object.c:1691
#6  0x00007f1d75eeceae in PyUFunc_GenericFunction (ufunc=ufunc@entry=0x18a6620, args=args@entry=(<numpy.ndarray at remote 0x7f1c3c06ab6f>, <numpy.float64 at remote 0x7f1d88fb3b40>), kwds=kwds@entry=0x0,
    op=op@entry=0x7f1b9fffd480) at numpy/core/src/umath/ufunc_object.c:3262
#7  0x00007f1d75eedd5e in ufunc_generic_call (ufunc=0x18a6620, args=(<numpy.ndarray at remote 0x7f1c3c06ab6f>, <numpy.float64 at remote 0x7f1d88fb3b40>), kwds=0x0)
    at numpy/core/src/umath/ufunc_object.c:4688
#8  0x00007f1d88a85a63 in PyObject_Call (func=func@entry=<numpy.ufunc at remote 0x18a6620>, arg=arg@entry=(<numpy.ndarray at remote 0x7f1c3c06ab6f>, <numpy.float64 at remote 0x7f1d88fb3b40>),
    kw=kw@entry=0x0) at /usr/src/debug/Python-2.7.5/Objects/abstract.c:2529
#9  0x00007f1d88a8635c in PyObject_CallFunctionObjArgs (callable=<numpy.ufunc at remote 0x18a6620>) at /usr/src/debug/Python-2.7.5/Objects/abstract.c:2760
#10 0x00007f1d75dad2db in PyArray_GenericBinaryFunction (op=<optimized out>, m2=<numpy.float64 at remote 0x7f1d88fb3b40>, m1=0x7f1c3c06ab6f) at numpy/core/src/multiarray/number.c:284
#11 array_divide (m1=0x7f1c3c06ab6f, m2=<numpy.float64 at remote 0x7f1d88fb3b40>) at numpy/core/src/multiarray/number.c:364
#12 0x00007f1d88a8195c in binary_op1 (v=v@entry=<numpy.ndarray at remote 0x7f1c3c06ab6f>, w=w@entry=<numpy.float64 at remote 0x7f1d88fb3b40>, op_slot=24)
    at /usr/src/debug/Python-2.7.5/Objects/abstract.c:945
#13 0x00007f1d88a81d82 in binary_op (v=<numpy.ndarray at remote 0x7f1c3c06ab6f>, w=<numpy.float64 at remote 0x7f1d88fb3b40>, op_slot=<optimized out>, op_name=0x7f1d88b66466 "/")
    at /usr/src/debug/Python-2.7.5/Objects/abstract.c:997
#14 0x00007f1d88b1de2c in PyEval_EvalFrameEx (
    f=f@entry=Frame 0x7f1b000d0750, for file function_cache/eggs_extract/uc_audio_audio_vggish_feature/vggish_feature-0.1-py2.7/vggish_feature/mel_features.py, line 180, in spectrogram_to_mel_matrix (num_mel_bins=64, num_spectrogram_bins=257, audio_sample_rate=16000, lower_edge_hertz=125, upper_edge_hertz=7500, nyquist_hertz=<float at remote 0x7f1b7c13e948>, spectrogram_bins_hertz=<numpy.ndarray at remote 0x7f1c3c06a710>, spectrogram_bins_mel=<numpy.ndarray at remote 0x7f1c3c06a850>, band_edges_mel=<numpy.ndarray at remote 0x7f1c3c06aa30>, mel_weights_matrix=<numpy.ndarray at remote 0x7f1c3c06acb0>, i=17, lower_edge_mel=<numpy.float64 at remote 0x7f1d7a6bdd68>, center_mel=<numpy.float64 at remote 0x7f1d7a6bdf00>, upper_edge_mel=<numpy.float64 at remote 0x7f1d7a6bd4b0>, lower_slope=<numpy.ndarray at remote 0x7f1c3c06a120>, upper_slope=<numpy.ndarray at remote 0x7f1c3c06aee0>), throwflag=throwflag@entry=0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:1481
#15 0x00007f1d88b2103d in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x7f1d88fe1068, argcount=0, kws=kws@entry=0x7f1c4c05f4a8,
    kwcount=kwcount@entry=5, defs=defs@entry=0x7f1d7d7b7d28, defcount=defcount@entry=5, closure=0x0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:3640
#16 0x00007f1d88aaaa6d in function_call (func=<function at remote 0x7f1c540acf50>, arg=(),
    kw={'num_mel_bins': 64, 'upper_edge_hertz': 7500, 'lower_edge_hertz': 125, 'num_spectrogram_bins': 257, 'audio_sample_rate': 16000}) at /usr/src/debug/Python-2.7.5/Objects/funcobject.c:526
#17 0x00007f1d88a85a63 in PyObject_Call (func=func@entry=<function at remote 0x7f1c540acf50>, arg=arg@entry=(),
    kw=kw@entry={'num_mel_bins': 64, 'upper_edge_hertz': 7500, 'lower_edge_hertz': 125, 'num_spectrogram_bins': 257, 'audio_sample_rate': 16000}) at /usr/src/debug/Python-2.7.5/Objects/abstract.c:2529
#18 0x00007f1d88b196fd in ext_do_call (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7f1b9fffdd38, func=<function at remote 0x7f1c540acf50>)
    at /usr/src/debug/Python-2.7.5/Python/ceval.c:4721
#19 PyEval_EvalFrameEx (
    f=f@entry=Frame 0x7f1d3c016e80, for file function_cache/eggs_extract/uc_audio_audio_vggish_feature/vggish_feature-0.1-py2.7/vggish_feature/mel_features.py, line 222, in log_mel_spectrogram (data=<numpy.ndarray at remote 0x7f1c64133c60>, audio_sample_rate=16000, log_offset=<float at remote 0x7f1d3c00b4e0>, window_length_secs=<float at remote 0x7f1d3c00b5a0>, hop_length_secs=<float at remote 0x7f1d3c00b4e0>, kwargs={'num_mel_bins': 64, 'upper_edge_hertz': 7500, 'lower_edge_hertz': 125}, window_length_samples=400, hop_length_samples=160, fft_length=512, spectrogram=<numpy.ndarray at remote 0x7f1c3c06a990>), throwflag=throwflag@entry=0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:3079
#20 0x00007f1d88b2103d in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=1, kws=0x7f1d38001f10, kwcount=7, defs=0x7f1c540e9fe0,
    defcount=4, closure=closure@entry=0x0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:3640
#21 0x00007f1d88b1e53c in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7f1b9fffdf40, func=<optimized out>) at /usr/src/debug/Python-2.7.5/Python/ceval.c:4504
#22 call_function (oparg=<optimized out>, pp_stack=0x7f1b9fffdf40) at /usr/src/debug/Python-2.7.5/Python/ceval.c:4429
#23 PyEval_EvalFrameEx (
---Type <return> to continue, or q <return> to quit---q
f=f@entry=Frame 0x7f1d38001d50, for file function_cache/eggs_extract/uc_audio_audio_vggish_feature/vggish_feature-0.1-py2.7/vggish_feature/vggish_input.py, line 60, in waveform_to_Quit
(gdb)

the python exec is :
lower_slope = ((spectrogram_bins_mel - lower_edge_mel) /
                   (center_mel - lower_edge_mel))

The text was updated successfully, but these errors were encountered:

wadeliuyi · 2020-02-19T04:06:17Z

(gdb) list
799	             * Pick up the head block of its free list.
800	             */
801	            ++pool->ref.count;
802	            bp = pool->freeblock;
803	            assert(bp != NULL);
804	            if ((pool->freeblock = *(block **)bp) != NULL) {
805	                UNLOCK();
806	                return (void *)bp;
807	            }
808	            /*
(gdb) p *pool
$1 = {
  ref = {
    _padding = 0x32 <Address 0x32 out of bounds>,
    count = 50
  },
  freeblock = 0x7f1c3c06ae9000 <Address 0x7f1c3c06ae9000 out of bounds>,
  nextpool = 0x7f1c3c09b000,
  prevpool = 0x7f1d88dcfe60 <usedpools+128>,
  arenaindex = 127,
  szidx = 9,
  nextoffset = 4048,
  maxnextoffset = 4016
}
(gdb) p bp
$2 = (block *) 0x7f1c3c06ae9000 <Address 0x7f1c3c06ae9000 out of bounds>
(gdb)
(gdb) p pool
$3 = (struct pool_header *) 0x7f1c3c06a000
(gdb) x/10a 0x7f1c3c06ae90
0x7f1c3c06ae90:	0x0	0x7f1d761e9dc0 <PyArray_Type>
0x7f1c3c06aea0:	0x7f1d38574c20	0xffffffff00000000
0x7f1c3c06aeb0:	0x0	0x0
0x7f1c3c06aec0:	0x0	0x7f1d761eab00 <DOUBLE_Descr>
0x7f1c3c06aed0:	0x7379656b00000507	0x0

we can see bp is 0x7f1c3c06ae9000 and out of bounds, the 0x7f1c3c06ae90 is in current pool, it look like that update freeblock with 0x7f1c3c06ae90<<8

seberg · 2020-02-19T04:20:30Z

@wadeliuyi can you try to create a full working example. Also which version of NumPy are yo using exactly?

wadeliuyi · 2020-02-19T11:42:31Z

@wadeliuyi can you try to create a full working example. Also which version of NumPy are yo using exactly?

numpy version is 1.16.4.

wadeliuyi · 2020-02-19T11:52:25Z

the full sample is tensorflow script:
https://github.com/tensorflow/models/blob/master/research/audioset/vggish/mel_features.py

and crash in function spectrogram_to_mel_matrix line 180.

seberg · 2020-02-19T16:04:30Z

I tried to reproduce it with 1.16.5, which should be the same, on Python 2.7. I cannot reproduce it. We will need a workable example.
From where the traceback from, there is no way telling what causes it directly. For all I know it could be tensorflow, or other (probably) C extension code that seems completely unrelated.

wadeliuyi · 2020-02-20T03:04:17Z

I tried to reproduce it with 1.16.5, which should be the same, on Python 2.7. I cannot reproduce it. We will need a workable example.
From where the traceback from, there is no way telling what causes it directly. For all I know it could be tensorflow, or other (probably) C extension code that seems completely unrelated.

thanks for your reply. My program reproduce this program every time after it run more than one hour, and the stack is same with each core file.

And I will try it with new version of numpy, thank you again.

seberg · 2020-02-20T05:35:22Z

@wadeliuyi the problem is that there definitely is a bug somewhere.

From where it comes from, I could imagine that PYTHOMALLOC=malloc changes the way it crashes (not a solution, just a silly try). The crash seems like it is inside the Python memory management system, and right now I am not sure if there are any patterns/misuses that are known to cause this type of error.

EDIT: Opst, python 2.7 does not support PYTHONMALLOC anyway.

rossbar · 2020-07-17T19:35:29Z

I'm closing this due to inactivity, more information and a reproducible example that isolated the problem in NumPy (if it is even a NumPy issue) would likely be required to make any progress here.

rossbar added the 33 - Question Question about NumPy usage or development label Jul 17, 2020

rossbar closed this as completed Jul 17, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

numpy crash #15603

numpy crash #15603

wadeliuyi commented Feb 19, 2020 •

edited

Loading

wadeliuyi commented Feb 19, 2020 •

edited

Loading

seberg commented Feb 19, 2020

wadeliuyi commented Feb 19, 2020

wadeliuyi commented Feb 19, 2020 •

edited

Loading

seberg commented Feb 19, 2020

wadeliuyi commented Feb 20, 2020

seberg commented Feb 20, 2020 •

edited

Loading

rossbar commented Jul 17, 2020

numpy crash #15603

numpy crash #15603

Comments

wadeliuyi commented Feb 19, 2020 • edited Loading

wadeliuyi commented Feb 19, 2020 • edited Loading

seberg commented Feb 19, 2020

wadeliuyi commented Feb 19, 2020

wadeliuyi commented Feb 19, 2020 • edited Loading

seberg commented Feb 19, 2020

wadeliuyi commented Feb 20, 2020

seberg commented Feb 20, 2020 • edited Loading

rossbar commented Jul 17, 2020

wadeliuyi commented Feb 19, 2020 •

edited

Loading

wadeliuyi commented Feb 19, 2020 •

edited

Loading

wadeliuyi commented Feb 19, 2020 •

edited

Loading

seberg commented Feb 20, 2020 •

edited

Loading