-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Leverage Docker image caching #2009
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Labels
Comments
AndreMiras
added a commit
to AndreMiras/python-for-android
that referenced
this issue
Oct 26, 2019
Revamping Travis and Docker setup introducing a `Makefile`. The idea is to have a single entry point via `make` and reproducible builds via Docker. This pull request also starts introducing some docker layer cache optimization as needed by kivy#2009 to speed up docker pull/push and rebuilds from cache.
AndreMiras
added a commit
to AndreMiras/python-for-android
that referenced
this issue
Oct 26, 2019
Revamping Travis and Docker setup introducing a `Makefile`. The idea is to have a single entry point via `make` and reproducible builds via Docker. This pull request also starts introducing some docker layer cache optimization as needed by kivy#2009 to speed up docker pull/push and rebuilds from cache.
AndreMiras
added a commit
to AndreMiras/python-for-android
that referenced
this issue
Oct 27, 2019
Revamping Travis and Docker setup introducing a `Makefile`. The idea is to have a single entry point via `make` and reproducible builds via Docker. This pull request also starts introducing some docker layer cache optimization as needed by kivy#2009 to speed up docker pull/push and rebuilds from cache.
AndreMiras
added a commit
to AndreMiras/python-for-android
that referenced
this issue
Oct 27, 2019
Revamping Travis and Docker setup introducing a `Makefile`. The idea is to have a single entry point via `make` and reproducible builds via Docker. This pull request also starts introducing some docker layer cache optimization as needed by kivy#2009 to speed up docker pull/push and rebuilds from cache.
AndreMiras
added a commit
to AndreMiras/python-for-android
that referenced
this issue
Oct 27, 2019
Revamping Travis and Docker setup introducing a `Makefile`. The idea is to move the CI complexity from .travis.yml to `Makefile`. That makes a single entry point via `make` command and reproducible builds via Docker. It makes it easy to run some commands outside docker, such as: ```sh make testapps/python3/armeabi-v7a ``` Or the same command inside docker: ```sh make docker/run/make/testapps/python3/armeabi-v7a ``` This pull request also starts introducing some docker layer cache optimization as needed by kivy#2009 to speed up docker pull/push and rebuilds from cache. It also introduces other Docker images good practices like ordering dependencies alphabetically or always enforcing `apt update` prior install, refs: https://docs.docker.com/develop/develop-images/dockerfile_best-practices/ Subsequent pull requests would simplify the process furthermore and leverage the cache to speed up builds.
This was referenced Oct 27, 2019
AndreMiras
added a commit
to AndreMiras/python-for-android
that referenced
this issue
Oct 28, 2019
The `docker build` stage tries to pull existing image related to the on-going branch or pull request. Then this stage would build the new image from this and push it for subsequent builds. `DOCKER_USERNAME` is set as normal Travis environment variable while `DOCKER_PASSWORD` as an encrypted one. `TRAVIS_PULL_REQUEST` and `TRAVIS_BRANCH` are being setup by Travis. Note that we're passing variables to `Makefile` explicitly on `make` call rather than using the `--environment-overrides` flag. While it's more verbose, it makes it easier to follow what's being used and when. Closes kivy#2009
AndreMiras
added a commit
to AndreMiras/python-for-android
that referenced
this issue
Oct 28, 2019
The `docker build` stage tries to pull existing image related to the on-going branch or pull request. Then this stage would build the new image from this and push it for subsequent builds. `DOCKER_USERNAME` is set as normal Travis environment variable while `DOCKER_PASSWORD` as an encrypted one. `TRAVIS_PULL_REQUEST` and `TRAVIS_BRANCH` are being setup by Travis. Note that we're passing variables to `Makefile` explicitly on `make` call rather than using the `--environment-overrides` flag. While it's more verbose, it makes it easier to follow what's being used and when. Closes kivy#2009
AndreMiras
added a commit
to AndreMiras/python-for-android
that referenced
this issue
Oct 28, 2019
The `docker build` stage tries to pull existing image related to the on-going branch or pull request. Then this stage would build the new image from this and push it for subsequent builds. `DOCKER_USERNAME` is set as normal Travis environment variable while `DOCKER_PASSWORD` as an encrypted one. `TRAVIS_PULL_REQUEST` and `TRAVIS_BRANCH` are being setup by Travis. Note that we're passing variables to `Makefile` explicitly on `make` call rather than using the `--environment-overrides` flag. While it's more verbose, it makes it easier to follow what's being used and when. Closes kivy#2009
AndreMiras
added a commit
to AndreMiras/python-for-android
that referenced
this issue
Oct 28, 2019
The `docker build` stage tries to pull existing image related to the on-going branch or pull request. Then this stage would build the new image from this and push it for subsequent builds. `DOCKER_USERNAME` is set as normal Travis environment variable while `DOCKER_PASSWORD` as an encrypted one. `TRAVIS_PULL_REQUEST` and `TRAVIS_BRANCH` are being setup by Travis. Note that we're passing variables to `Makefile` explicitly on `make` call rather than using the `--environment-overrides` flag. While it's more verbose, it makes it easier to follow what's being used and when. Closes kivy#2009
AndreMiras
added a commit
to AndreMiras/python-for-android
that referenced
this issue
Oct 28, 2019
The `docker build` stage tries to pull existing image related to the on-going branch or pull request. Then this stage would build the new image from this and push it for subsequent builds. `DOCKER_USERNAME` is set as normal Travis environment variable while `DOCKER_PASSWORD` as an encrypted one. `TRAVIS_PULL_REQUEST` and `TRAVIS_BRANCH` are being setup by Travis. Note that we're passing variables to `Makefile` explicitly on `make` call rather than using the `--environment-overrides` flag. While it's more verbose, it makes it easier to follow what's being used and when. Closes kivy#2009
AndreMiras
added a commit
to AndreMiras/python-for-android
that referenced
this issue
Oct 28, 2019
The `docker build` stage tries to pull existing image related to the on-going branch or pull request. Then this stage would build the new image from this and push it for subsequent builds. `DOCKER_USERNAME` is set as normal Travis environment variable while `DOCKER_PASSWORD` as an encrypted one. `TRAVIS_PULL_REQUEST` and `TRAVIS_BRANCH` are being setup by Travis. Note that we're passing variables to `Makefile` explicitly on `make` call rather than using the `--environment-overrides` flag. While it's more verbose, it makes it easier to follow what's being used and when. Closes kivy#2009
AndreMiras
added a commit
to AndreMiras/python-for-android
that referenced
this issue
Oct 28, 2019
The `docker build` stage tries to pull existing image related to the on-going branch or pull request. Then this stage would build the new image from this and push it for subsequent builds. `DOCKER_USERNAME` is set as normal Travis environment variable while `DOCKER_PASSWORD` as an encrypted one. `TRAVIS_PULL_REQUEST` and `TRAVIS_BRANCH` are being setup by Travis. Note that we're passing variables to `Makefile` explicitly on `make` call rather than using the `--environment-overrides` flag. While it's more verbose, it makes it easier to follow what's being used and when. Closes kivy#2009
AndreMiras
added a commit
to AndreMiras/python-for-android
that referenced
this issue
Oct 28, 2019
The `docker build` stage tries to pull existing image related to the on-going branch or pull request. Then this stage would build the new image from this and push it for subsequent builds. `DOCKER_USERNAME` is set as normal Travis environment variable while `DOCKER_PASSWORD` as an encrypted one. `TRAVIS_PULL_REQUEST` and `TRAVIS_BRANCH` are being setup by Travis. Note that we're passing variables to `Makefile` explicitly on `make` call rather than using the `--environment-overrides` flag. While it's more verbose, it makes it easier to follow what's being used and when. Closes kivy#2009
AndreMiras
added a commit
to AndreMiras/python-for-android
that referenced
this issue
Oct 28, 2019
The `docker build` stage tries to pull existing image related to the on-going branch or pull request. Then this stage would build the new image from this and push it for subsequent builds. `DOCKER_USERNAME` is set as normal Travis environment variable while `DOCKER_PASSWORD` as an encrypted one. `TRAVIS_PULL_REQUEST` and `TRAVIS_BRANCH` are being setup by Travis. Note that we're passing variables to `Makefile` explicitly on `make` call rather than using the `--environment-overrides` flag. While it's more verbose, it makes it easier to follow what's being used and when. Closes kivy#2009
AndreMiras
added a commit
to AndreMiras/python-for-android
that referenced
this issue
Oct 28, 2019
The `docker build` is now running alongside the `tox` testing as part of the `pre checks` stage. It also fully leverages the docker image cache. The newly built image is pushed to be available for subsequent build stages and pull requests. Two environment variables `DOCKER_USERNAME` and `DOCKER_PASSWORD` got added to the Travis UI for making image pushing possible. Both `push` and `pull` features fail silently with `|| true` as they should be optional for a valid build. `TRAVIS_PULL_REQUEST` and `TRAVIS_BRANCH` are built-in Travis variables. Note that we're passing variables to `Makefile` explicitly on `make` call rather than using the `--environment-overrides` flag. While it's more verbose, it makes it easier to follow what's being used and when. With this change, `Python 3 arm64-v8a` the longest `testapps` build that was taking ~26 minutes to complete is now taking ~20 minutes when hitting docker layers cache (most of the time). The shorter build `Rebuild updated recipes` was taking ~9 minutes and is now down to ~3 minutes. Closes kivy#2009
AndreMiras
added a commit
to AndreMiras/python-for-android
that referenced
this issue
Oct 29, 2019
Revamping Travis and Docker setup introducing a `Makefile`. The idea is to move the CI complexity from .travis.yml to `Makefile`. That makes a single entry point via `make` command and reproducible builds via Docker. It makes it easy to run some commands outside docker, such as: ```sh make testapps/python3/armeabi-v7a ``` Or the same command inside docker: ```sh make docker/run/make/testapps/python3/armeabi-v7a ``` This pull request also starts introducing some docker layer cache optimization as needed by kivy#2009 to speed up docker pull/push and rebuilds from cache. It also introduces other Docker images good practices like ordering dependencies alphabetically or always enforcing `apt update` prior install, refs: https://docs.docker.com/develop/develop-images/dockerfile_best-practices/ Subsequent pull requests would simplify the process furthermore and leverage the cache to speed up builds.
@AndreMiras: Is this still an issue? |
We haven't solved it in p4a, but I think it's better documented in stackoverflow now. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The idea would be to speed up docker build (currently 10 minutes) by using the
--cache-from
flag.Thing is we also need to
docker push
our successful builds ourselves as the DockerHub autobuild seems to mess up withCOPY
command leading to 100% cache bust when using this image.The text was updated successfully, but these errors were encountered: