Skip to content

Commit 062300b

Browse files
patrickvonplatenpatrickvonplaten
authored
[Testing] Add Flax Tests on GPU, Add Speech and Vision to Flax & TF tests (huggingface#13313)
* up * finish * Apply suggestions from code review * apply Lysandres suggestions * adapt circle ci as well * finish * Update setup.py
1 parent 8b2de0e commit 062300b

File tree

4 files changed

+215
-24
lines changed

4 files changed

+215
-24
lines changed

‎.circleci/config.yml

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ jobs:
8080
- v0.4-{{ checksum "setup.py" }}
8181
- run: sudo apt-get -y update && sudo apt-get install -y libsndfile1-dev
8282
- run: pip install --upgrade pip
83-
- run: pip install .[sklearn,tf-cpu,torch,testing,sentencepiece,speech,vision]
83+
- run: pip install .[sklearn,tf-cpu,torch,testing,sentencepiece,torch-speech,vision]
8484
- run: pip install torch-scatter -f https://pytorch-geometric.com/whl/torch-1.9.0+cpu.html
8585
- save_cache:
8686
key: v0.4-{{ checksum "setup.py" }}
@@ -116,7 +116,7 @@ jobs:
116116
- v0.4-{{ checksum "setup.py" }}
117117
- run: sudo apt-get -y update && sudo apt-get install -y libsndfile1-dev
118118
- run: pip install --upgrade pip
119-
- run: pip install .[sklearn,tf-cpu,torch,testing,sentencepiece,speech,vision]
119+
- run: pip install .[sklearn,tf-cpu,torch,testing,sentencepiece,torch-speech,vision]
120120
- run: pip install torch-scatter -f https://pytorch-geometric.com/whl/torch-1.9.0+cpu.html
121121
- save_cache:
122122
key: v0.4-{{ checksum "setup.py" }}
@@ -147,7 +147,7 @@ jobs:
147147
- v0.4-{{ checksum "setup.py" }}
148148
- run: sudo apt-get -y update && sudo apt-get install -y libsndfile1-dev
149149
- run: pip install --upgrade pip
150-
- run: pip install .[sklearn,flax,torch,testing,sentencepiece,speech,vision]
150+
- run: pip install .[sklearn,flax,torch,testing,sentencepiece,torch-speech,vision]
151151
- run: pip install torch-scatter -f https://pytorch-geometric.com/whl/torch-1.9.0+cpu.html
152152
- save_cache:
153153
key: v0.4-{{ checksum "setup.py" }}
@@ -183,7 +183,7 @@ jobs:
183183
- v0.4-{{ checksum "setup.py" }}
184184
- run: sudo apt-get -y update && sudo apt-get install -y libsndfile1-dev
185185
- run: pip install --upgrade pip
186-
- run: pip install .[sklearn,flax,torch,testing,sentencepiece,speech,vision]
186+
- run: pip install .[sklearn,flax,torch,testing,sentencepiece,torch-speech,vision]
187187
- run: pip install torch-scatter -f https://pytorch-geometric.com/whl/torch-1.9.0+cpu.html
188188
- save_cache:
189189
key: v0.4-{{ checksum "setup.py" }}
@@ -213,7 +213,7 @@ jobs:
213213
- v0.4-{{ checksum "setup.py" }}
214214
- run: sudo apt-get -y update && sudo apt-get install -y libsndfile1-dev
215215
- run: pip install --upgrade pip
216-
- run: pip install .[sklearn,torch,testing,sentencepiece,speech,vision,timm]
216+
- run: pip install .[sklearn,torch,testing,sentencepiece,torch-speech,vision,timm]
217217
- run: pip install torch-scatter -f https://pytorch-geometric.com/whl/torch-1.9.0+cpu.html
218218
- save_cache:
219219
key: v0.4-torch-{{ checksum "setup.py" }}
@@ -248,7 +248,7 @@ jobs:
248248
- v0.4-{{ checksum "setup.py" }}
249249
- run: sudo apt-get -y update && sudo apt-get install -y libsndfile1-dev
250250
- run: pip install --upgrade pip
251-
- run: pip install .[sklearn,torch,testing,sentencepiece,speech,vision,timm]
251+
- run: pip install .[sklearn,torch,testing,sentencepiece,torch-speech,vision,timm]
252252
- run: pip install torch-scatter -f https://pytorch-geometric.com/whl/torch-1.9.0+cpu.html
253253
- save_cache:
254254
key: v0.4-torch-{{ checksum "setup.py" }}
@@ -277,7 +277,7 @@ jobs:
277277
- v0.4-tf-{{ checksum "setup.py" }}
278278
- v0.4-{{ checksum "setup.py" }}
279279
- run: pip install --upgrade pip
280-
- run: pip install .[sklearn,tf-cpu,testing,sentencepiece]
280+
- run: pip install .[sklearn,tf-cpu,testing,sentencepiece,tf-speech]
281281
- save_cache:
282282
key: v0.4-tf-{{ checksum "setup.py" }}
283283
paths:
@@ -310,7 +310,7 @@ jobs:
310310
- v0.4-tf-{{ checksum "setup.py" }}
311311
- v0.4-{{ checksum "setup.py" }}
312312
- run: pip install --upgrade pip
313-
- run: pip install .[sklearn,tf-cpu,testing,sentencepiece]
313+
- run: pip install .[sklearn,tf-cpu,testing,sentencepiece,tf-speech]
314314
- save_cache:
315315
key: v0.4-tf-{{ checksum "setup.py" }}
316316
paths:
@@ -338,7 +338,7 @@ jobs:
338338
- v0.4-flax-{{ checksum "setup.py" }}
339339
- v0.4-{{ checksum "setup.py" }}
340340
- run: pip install --upgrade pip
341-
- run: sudo pip install .[flax,testing,sentencepiece]
341+
- run: sudo pip install .[flax,testing,sentencepiece,flax-speech,vision]
342342
- save_cache:
343343
key: v0.4-flax-{{ checksum "setup.py" }}
344344
paths:
@@ -371,7 +371,7 @@ jobs:
371371
- v0.4-flax-{{ checksum "setup.py" }}
372372
- v0.4-{{ checksum "setup.py" }}
373373
- run: pip install --upgrade pip
374-
- run: sudo pip install .[flax,testing,sentencepiece]
374+
- run: sudo pip install .[flax,testing,sentencepiece,vision,flax-speech]
375375
- save_cache:
376376
key: v0.4-flax-{{ checksum "setup.py" }}
377377
paths:
@@ -401,7 +401,7 @@ jobs:
401401
- v0.4-{{ checksum "setup.py" }}
402402
- run: sudo apt-get -y update && sudo apt-get install -y libsndfile1-dev
403403
- run: pip install --upgrade pip
404-
- run: pip install .[sklearn,torch,testing,sentencepiece,speech,vision]
404+
- run: pip install .[sklearn,torch,testing,sentencepiece,torch-speech,vision]
405405
- run: pip install torch-scatter -f https://pytorch-geometric.com/whl/torch-1.9.0+cpu.html
406406
- save_cache:
407407
key: v0.4-torch-{{ checksum "setup.py" }}
@@ -437,7 +437,7 @@ jobs:
437437
- v0.4-{{ checksum "setup.py" }}
438438
- run: sudo apt-get -y update && sudo apt-get install -y libsndfile1-dev
439439
- run: pip install --upgrade pip
440-
- run: pip install .[sklearn,torch,testing,sentencepiece,speech,vision]
440+
- run: pip install .[sklearn,torch,testing,sentencepiece,torch-speech,vision]
441441
- run: pip install torch-scatter -f https://pytorch-geometric.com/whl/torch-1.9.0+cpu.html
442442
- save_cache:
443443
key: v0.4-torch-{{ checksum "setup.py" }}

‎.github/workflows/self-push.yml

Lines changed: 114 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ jobs:
3333
apt -y update && apt install -y software-properties-common && apt -y update && add-apt-repository -y ppa:git-core/ppa && apt -y update && apt install -y git
3434
apt install -y libsndfile1-dev
3535
pip install --upgrade pip
36-
pip install .[sklearn,testing,onnxruntime,sentencepiece,speech,vision,timm]
36+
pip install .[sklearn,testing,onnxruntime,sentencepiece,torch-speech,vision,timm]
3737
3838
- name: Launcher docker
3939
uses: actions/checkout@v2
@@ -78,6 +78,61 @@ jobs:
7878
name: run_all_tests_torch_gpu_test_reports
7979
path: reports
8080

81+
run_tests_flax_gpu:
82+
runs-on: [self-hosted, docker-gpu, single-gpu]
83+
container:
84+
image: tensorflow/tensorflow:2.4.1-gpu
85+
options: --gpus 0 --shm-size "16gb" --ipc host -v /mnt/cache/.cache/huggingface:/mnt/cache/
86+
steps:
87+
- name: Install dependencies
88+
run: |
89+
apt -y update && apt install -y software-properties-common && apt -y update && add-apt-repository -y ppa:git-core/ppa && apt -y update && apt install -y git
90+
pip install --upgrade "jax[cuda111]" -f https://storage.googleapis.com/jax-releases/jax_releases.html
91+
pip install --upgrade pip
92+
pip install .[sklearn,testing,sentencepiece,flax,flax-speech,vision]
93+
94+
- name: Launcher docker
95+
uses: actions/checkout@v2
96+
with:
97+
fetch-depth: 2
98+
99+
- name: NVIDIA-SMI
100+
run: |
101+
nvidia-smi
102+
103+
- name: Are GPUs recognized by our DL frameworks
104+
run: |
105+
python -c "from jax.lib import xla_bridge; print('GPU available:', xla_bridge.get_backend().platform)"
106+
python -c "import jax; print('Number of GPUs available:', len(jax.local_devices()))"
107+
108+
# - name: Fetch the tests to run
109+
# run: |
110+
# python utils/tests_fetcher.py --diff_with_last_commit | tee test_preparation.txt
111+
112+
- name: Report fetched tests
113+
uses: actions/upload-artifact@v2
114+
with:
115+
name: test_fetched
116+
path: test_preparation.txt
117+
118+
- name: Run all non-slow tests on GPU
119+
run: |
120+
python -m pytest -n 2 --dist=loadfile -v --make-reports=tests_flax_gpu
121+
# if [ -f test_list.txt ]; then
122+
# python -m pytest -n 2 --dist=loadfile -v --make-reports=tests_flax_gpu $(cat test_list.txt)
123+
# fi
124+
125+
- name: Failure short reports
126+
if: ${{ failure() }}
127+
run: cat reports/tests_flax_gpu_failures_short.txt
128+
129+
- name: Test suite reports artifacts
130+
if: ${{ always() }}
131+
uses: actions/upload-artifact@v2
132+
with:
133+
name: run_all_tests_flax_gpu_test_reports
134+
path: reports
135+
81136
# run_tests_tf_gpu:
82137
# runs-on: [self-hosted, docker-gpu, single-gpu]
83138
# timeout-minutes: 120
@@ -89,7 +144,7 @@ jobs:
89144
# run: |
90145
# apt -y update && apt install -y software-properties-common && apt -y update && add-apt-repository -y ppa:git-core/ppa && apt -y update && apt install -y git
91146
# pip install --upgrade pip
92-
# pip install .[sklearn,testing,onnxruntime,sentencepiece]
147+
# pip install .[sklearn,testing,onnxruntime,sentencepiece,tf-speech]
93148
#
94149
# - name: Launcher docker
95150
# uses: actions/checkout@v2
@@ -147,7 +202,7 @@ jobs:
147202
apt -y update && apt install -y software-properties-common && apt -y update && add-apt-repository -y ppa:git-core/ppa && apt -y update && apt install -y git
148203
apt install -y libsndfile1-dev
149204
pip install --upgrade pip
150-
pip install .[sklearn,testing,onnxruntime,sentencepiece,speech,vision,timm]
205+
pip install .[sklearn,testing,onnxruntime,sentencepiece,torch-speech,vision,timm]
151206
152207
- name: Launcher docker
153208
uses: actions/checkout@v2
@@ -195,6 +250,61 @@ jobs:
195250
name: run_all_tests_torch_multi_gpu_test_reports
196251
path: reports
197252

253+
run_tests_flax_multi_gpu:
254+
runs-on: [self-hosted, docker-gpu, multi-gpu]
255+
container:
256+
image: tensorflow/tensorflow:2.4.1-gpu
257+
options: --gpus 0 --shm-size "16gb" --ipc host -v /mnt/cache/.cache/huggingface:/mnt/cache/
258+
steps:
259+
- name: Install dependencies
260+
run: |
261+
apt -y update && apt install -y software-properties-common && apt -y update && add-apt-repository -y ppa:git-core/ppa && apt -y update && apt install -y git
262+
pip install --upgrade "jax[cuda111]" -f https://storage.googleapis.com/jax-releases/jax_releases.html
263+
pip install --upgrade pip
264+
pip install .[sklearn,testing,sentencepiece,flax,flax-speech,vision]
265+
266+
- name: Launcher docker
267+
uses: actions/checkout@v2
268+
with:
269+
fetch-depth: 2
270+
271+
- name: NVIDIA-SMI
272+
run: |
273+
nvidia-smi
274+
275+
- name: Are GPUs recognized by our DL frameworks
276+
run: |
277+
python -c "from jax.lib import xla_bridge; print('GPU available:', xla_bridge.get_backend().platform)"
278+
python -c "import jax; print('Number of GPUs available:', len(jax.local_devices()))"
279+
280+
# - name: Fetch the tests to run
281+
# run: |
282+
# python utils/tests_fetcher.py --diff_with_last_commit | tee test_preparation.txt
283+
284+
- name: Report fetched tests
285+
uses: actions/upload-artifact@v2
286+
with:
287+
name: test_fetched
288+
path: test_preparation.txt
289+
290+
- name: Run all non-slow tests on GPU
291+
run: |
292+
python -m pytest -n 2 --dist=loadfile -v --make-reports=tests_flax_multi_gpu
293+
# if [ -f test_list.txt ]; then
294+
# python -m pytest -n 2 --dist=loadfile -v --make-reports=tests_flax_multi_gpu $(cat test_list.txt)
295+
# fi
296+
297+
- name: Failure short reports
298+
if: ${{ failure() }}
299+
run: cat reports/tests_flax_multi_gpu_failures_short.txt
300+
301+
- name: Test suite reports artifacts
302+
if: ${{ always() }}
303+
uses: actions/upload-artifact@v2
304+
with:
305+
name: run_all_tests_flax_multi_gpu_test_reports
306+
path: reports
307+
198308
# run_tests_tf_multi_gpu:
199309
# runs-on: [self-hosted, docker-gpu, multi-gpu]
200310
# timeout-minutes: 120
@@ -206,7 +316,7 @@ jobs:
206316
# run: |
207317
# apt -y update && apt install -y software-properties-common && apt -y update && add-apt-repository -y ppa:git-core/ppa && apt -y update && apt install -y git
208318
# pip install --upgrade pip
209-
# pip install .[sklearn,testing,onnxruntime,sentencepiece]
319+
# pip install .[sklearn,testing,onnxruntime,sentencepiece,tf-speech]
210320
#
211321
# - name: Launcher docker
212322
# uses: actions/checkout@v2

‎.github/workflows/self-scheduled.yml

Lines changed: 82 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ jobs:
3434
run: |
3535
apt -y update && apt install -y libsndfile1-dev git
3636
pip install --upgrade pip
37-
pip install .[integrations,sklearn,testing,onnxruntime,sentencepiece,speech,vision,timm]
37+
pip install .[integrations,sklearn,testing,onnxruntime,sentencepiece,torch-speech,vision,timm]
3838
3939
- name: Are GPUs recognized by our DL frameworks
4040
run: |
@@ -85,6 +85,45 @@ jobs:
8585
name: run_all_tests_torch_gpu_test_reports
8686
path: reports
8787

88+
run_all_tests_flax_gpu:
89+
runs-on: [self-hosted, docker-gpu, single-gpu]
90+
container:
91+
image: tensorflow/tensorflow:2.4.1-gpu
92+
options: --gpus 0 --shm-size "16gb" --ipc host -v /mnt/cache/.cache/huggingface:/mnt/cache/
93+
steps:
94+
- name: Launcher docker
95+
uses: actions/checkout@v2
96+
97+
- name: NVIDIA-SMI
98+
run: |
99+
nvidia-smi
100+
101+
- name: Install dependencies
102+
run: |
103+
pip install --upgrade pip
104+
pip install --upgrade "jax[cuda111]" -f https://storage.googleapis.com/jax-releases/jax_releases.html
105+
pip install .[flax,integrations,sklearn,testing,sentencepiece,flax-speech,vision]
106+
107+
- name: Are GPUs recognized by our DL frameworks
108+
run: |
109+
python -c "from jax.lib import xla_bridge; print('GPU available:', xla_bridge.get_backend().platform)"
110+
python -c "import jax; print('Number of GPUs available:', len(jax.local_devices()))"
111+
112+
- name: Run all tests on GPU
113+
run: |
114+
python -m pytest -n 1 -v --dist=loadfile --make-reports=tests_flax_gpu tests
115+
116+
- name: Failure short reports
117+
if: ${{ always() }}
118+
run: cat reports/tests_flax_gpu_failures_short.txt
119+
120+
- name: Test suite reports artifacts
121+
if: ${{ always() }}
122+
uses: actions/upload-artifact@v2
123+
with:
124+
name: run_all_tests_flax_gpu_test_reports
125+
path: reports
126+
88127
run_all_tests_tf_gpu:
89128
runs-on: [self-hosted, docker-gpu, single-gpu]
90129
container:
@@ -102,7 +141,7 @@ jobs:
102141
run: |
103142
apt -y update && apt install -y git
104143
pip install --upgrade pip
105-
pip install .[sklearn,testing,onnx,sentencepiece]
144+
pip install .[sklearn,testing,onnx,sentencepiece,tf-speech]
106145
107146
- name: Are GPUs recognized by our DL frameworks
108147
run: |
@@ -158,7 +197,7 @@ jobs:
158197
run: |
159198
apt -y update && apt install -y libsndfile1-dev git
160199
pip install --upgrade pip
161-
pip install .[integrations,sklearn,testing,onnxruntime,sentencepiece,speech,vision,timm]
200+
pip install .[integrations,sklearn,testing,onnxruntime,sentencepiece,torch-speech,vision,timm]
162201
163202
- name: Are GPUs recognized by our DL frameworks
164203
run: |
@@ -213,7 +252,7 @@ jobs:
213252
run: |
214253
apt -y update && apt install -y git
215254
pip install --upgrade pip
216-
pip install .[sklearn,testing,onnx,sentencepiece]
255+
pip install .[sklearn,testing,onnx,sentencepiece,tf-speech]
217256
218257
- name: Are GPUs recognized by our DL frameworks
219258
run: |
@@ -251,6 +290,45 @@ jobs:
251290
name: run_all_tests_tf_multi_gpu_test_reports
252291
path: reports
253292

293+
run_all_tests_flax_multi_gpu:
294+
runs-on: [self-hosted, docker-gpu, multi-gpu]
295+
container:
296+
image: tensorflow/tensorflow:2.4.1-gpu
297+
options: --gpus 0 --shm-size "16gb" --ipc host -v /mnt/cache/.cache/huggingface:/mnt/cache/
298+
steps:
299+
- name: Launcher docker
300+
uses: actions/checkout@v2
301+
302+
- name: NVIDIA-SMI
303+
run: |
304+
nvidia-smi
305+
306+
- name: Install dependencies
307+
run: |
308+
pip install --upgrade pip
309+
pip install --upgrade "jax[cuda111]" -f https://storage.googleapis.com/jax-releases/jax_releases.html
310+
pip install .[flax,integrations,sklearn,testing,sentencepiece,flax-speech,vision]
311+
312+
- name: Are GPUs recognized by our DL frameworks
313+
run: |
314+
python -c "from jax.lib import xla_bridge; print('GPU available:', xla_bridge.get_backend().platform)"
315+
python -c "import jax; print('Number of GPUs available:', len(jax.local_devices()))"
316+
317+
- name: Run all tests on GPU
318+
run: |
319+
python -m pytest -n 1 -v --dist=loadfile --make-reports=tests_flax_gpu tests
320+
321+
- name: Failure short reports
322+
if: ${{ always() }}
323+
run: cat reports/tests_flax_gpu_failures_short.txt
324+
325+
- name: Test suite reports artifacts
326+
if: ${{ always() }}
327+
uses: actions/upload-artifact@v2
328+
with:
329+
name: run_all_tests_flax_gpu_test_reports
330+
path: reports
331+
254332
run_all_tests_torch_cuda_extensions_gpu:
255333
runs-on: [self-hosted, docker-gpu, single-gpu]
256334
container:

0 commit comments

Comments
 (0)