Pytest gives segmentation fault on fresh clone #17582

NeilBotelho · 2020-06-13T15:54:35Z

Describe the bug

on running pytest on a fresh clone of the scikit-learn repository in a virtual env(properly setup), I get

Fatal Python error: Segmentation fault

Steps/Code to Reproduce

clone the scikit-learn repository
python3.7 -m venv venv
source venv/bin/activate
run pip install -e in the scikit-learn repo.
run pytest

Platform:
Debian 10
4gb RAM
python 3.7

Expected Results

pytest runs without error (or one or two known errors)

Actual Results

pytest throws a segmentation fault.
The traceback is as follows:

Traceback

Fatal Python error: Segmentation fault

Thread 0x00007f931f3a5700 (most recent call first):
  File "/usr/lib/python3.7/threading.py", line 296 in wait
  File "/home/neil/testing/venv/lib/python3.7/site-packages/joblib/externals/loky/backend/queues.py", line 138 in _feed
  File "/usr/lib/python3.7/threading.py", line 865 in run
  File "/usr/lib/python3.7/threading.py", line 917 in _bootstrap_inner
  File "/usr/lib/python3.7/threading.py", line 885 in _bootstrap

Thread 0x00007f931eba4700 (most recent call first):
  File "/usr/lib/python3.7/selectors.py", line 415 in select
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 920 in wait
  File "/home/neil/testing/venv/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py", line 609 in wait_result_broken_or_wakeup
  File "/home/neil/testing/venv/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py", line 555 in run
  File "/usr/lib/python3.7/threading.py", line 917 in _bootstrap_inner
  File "/usr/lib/python3.7/threading.py", line 885 in _bootstrap

Current thread 0x00007f9342504740 (most recent call first):
  File "/home/neil/testing/scikit-learn/sklearn/svm/_base.py", line 276 in _dense_fit
  File "/home/neil/testing/scikit-learn/sklearn/svm/_base.py", line 217 in fit
  File "/home/neil/testing/scikit-learn/sklearn/utils/estimator_checks.py", line 1834 in check_classifiers_train
  File "/home/neil/testing/scikit-learn/sklearn/utils/_testing.py", line 299 in wrapper
  File "/home/neil/testing/scikit-learn/sklearn/tests/test_common.py", line 85 in test_estimators
  File "/home/neil/testing/venv/lib/python3.7/site-packages/_pytest/python.py", line 182 in pytest_pyfunc_call
  File "/home/neil/testing/venv/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/neil/testing/venv/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/neil/testing/venv/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/neil/testing/venv/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/neil/testing/venv/lib/python3.7/site-packages/_pytest/python.py", line 1477 in runtest
  File "/home/neil/testing/venv/lib/python3.7/site-packages/_pytest/runner.py", line 135 in pytest_runtest_call
  File "/home/neil/testing/venv/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/neil/testing/venv/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/neil/testing/venv/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/neil/testing/venv/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/neil/testing/venv/lib/python3.7/site-packages/_pytest/runner.py", line 217 in <lambda>
  File "/home/neil/testing/venv/lib/python3.7/site-packages/_pytest/runner.py", line 244 in from_call
  File "/home/neil/testing/venv/lib/python3.7/site-packages/_pytest/runner.py", line 217 in call_runtest_hook
  File "/home/neil/testing/venv/lib/python3.7/site-packages/_pytest/runner.py", line 186 in call_and_report
  File "/home/neil/testing/venv/lib/python3.7/site-packages/_pytest/runner.py", line 100 in runtestprotocol
  File "/home/neil/testing/venv/lib/python3.7/site-packages/_pytest/runner.py", line 85 in pytest_runtest_protocol
  File "/home/neil/testing/venv/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/neil/testing/venv/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/neil/testing/venv/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/neil/testing/venv/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/neil/testing/venv/lib/python3.7/site-packages/_pytest/main.py", line 272 in pytest_runtestloop
  File "/home/neil/testing/venv/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/neil/testing/venv/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/neil/testing/venv/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/neil/testing/venv/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/neil/testing/venv/lib/python3.7/site-packages/_pytest/main.py", line 247 in _main
  File "/home/neil/testing/venv/lib/python3.7/site-packages/_pytest/main.py", line 191 in wrap_session
  File "/home/neil/testing/venv/lib/python3.7/site-packages/_pytest/main.py", line 240 in pytest_cmdline_main
  File "/home/neil/testing/venv/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/neil/testing/venv/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/neil/testing/venv/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/neil/testing/venv/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/neil/testing/venv/lib/python3.7/site-packages/_pytest/config/__init__.py", line 125 in main
  File "/home/neil/testing/venv/bin/pytest", line 10 in <module>

I've also uploaded the entire output of running the pytest command(as it is too long to paste in the issue) here:
pytest_output.txt

The text was updated successfully, but these errors were encountered:

rth · 2020-06-13T16:13:08Z

Thanks for the report. Could you please re-run pytest with the -v option to see which test segfaults exactly?

NeilBotelho · 2020-06-13T16:41:59Z

It seems like its failing in

sklearn/tests/test_common.py::test_estimators[NuSVC()-check_classifiers_train(readonly_memmap=True)]

Full output: pytest-v__output.txt

glemaitre · 2020-06-14T13:35:58Z

@NeilBotelho Are you running on an ARM architecture?

NeilBotelho · 2020-06-14T13:55:07Z

@NeilBotelho Are you running on an ARM architecture?

No. I've got an AMD A6-7310

TomDLT · 2020-06-15T18:59:23Z

May be related to misaligned arrays in joblib (joblib/joblib#563).

Numpy checks for misaligned data in every function, so it is not a problem in most estimators, but C expects arrays to be aligned to a multiple of their size. NuSVC might directly call C code, hence the segfault.

import numpy as np

def create_memmap_backed_data(data):
    import os.path
    import tempfile
    import joblib
    temp_folder = tempfile.mkdtemp(prefix='testing_')
    filename = os.path.join(temp_folder, 'data.pkl')
    joblib.dump(data, filename)
    return joblib.load(filename, mmap_mode='r')

y = create_memmap_backed_data(np.random.randn(20, 1))
print(y.flags['ALIGNED'])

Hedaia-Ghannam-2020 · 2021-08-11T16:20:36Z

I have the same issue where I have scipy version 1.6.2
Pytest stage in my CI/CD pipeline in gitlab fails with the following error on my AWS linux machine:

$ pytest tests/
Fatal Python error: Segmentation fault
Current thread 0x00007f1b36afb740 (most recent call first):
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1166 in create_module
  File "<frozen importlib._bootstrap>", line 556 in module_from_spec
  File "<frozen importlib._bootstrap>", line 657 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "/usr/local/lib/python3.8/site-packages/sklearn/__check_build/__init__.py", line 44 in <module>

ogrisel · 2021-11-15T16:54:51Z

It might be related to the version of the C/C++ compiler used to build scikit-learn compiled extensions, in particular the vendored libsvm C++ code.

@Hedaia-Ghannam-2020 @NeilBotelho could you please tell us which compiler and which version did you use to reproduce this segfault?

ogrisel · 2021-11-15T16:58:26Z

It would also be interesting to get the pytest log when pytest is run with the -v option to see which test parametrization is causing the segfault.

thomasjpfan · 2022-04-18T01:29:22Z

The memmap aligned issue in the tests suite is now fixed in main with #21654. With that in mind, I am closing this issue.

NeilBotelho added the Bug: triage label Jun 13, 2020

rth added Blocker Bug and removed Bug: triage Blocker labels Jun 17, 2020

adrinjalali added Hard Hard level of difficulty help wanted labels Jul 10, 2020

cmarmo added the module:test-suite everything related to our tests label Sep 15, 2020

tricktron mentioned this issue May 11, 2021

scikit: random segmentation faults in hydra ( NixOS/nixpkgs#121988

Closed

thomasjpfan closed this as completed Apr 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pytest gives segmentation fault on fresh clone #17582

Pytest gives segmentation fault on fresh clone #17582

NeilBotelho commented Jun 13, 2020

rth commented Jun 13, 2020

NeilBotelho commented Jun 13, 2020

glemaitre commented Jun 14, 2020

NeilBotelho commented Jun 14, 2020

TomDLT commented Jun 15, 2020 •

edited

Loading

Hedaia-Ghannam-2020 commented Aug 11, 2021 •

edited by ogrisel

Loading

ogrisel commented Nov 15, 2021

ogrisel commented Nov 15, 2021

thomasjpfan commented Apr 18, 2022

Pytest gives segmentation fault on fresh clone #17582

Pytest gives segmentation fault on fresh clone #17582

Comments

NeilBotelho commented Jun 13, 2020

Describe the bug

Steps/Code to Reproduce

Expected Results

Actual Results

rth commented Jun 13, 2020

NeilBotelho commented Jun 13, 2020

glemaitre commented Jun 14, 2020

NeilBotelho commented Jun 14, 2020

TomDLT commented Jun 15, 2020 • edited Loading

Hedaia-Ghannam-2020 commented Aug 11, 2021 • edited by ogrisel Loading

ogrisel commented Nov 15, 2021

ogrisel commented Nov 15, 2021

thomasjpfan commented Apr 18, 2022

TomDLT commented Jun 15, 2020 •

edited

Loading

Hedaia-Ghannam-2020 commented Aug 11, 2021 •

edited by ogrisel

Loading