Skip to content

Leverage Docker image caching #2009

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
AndreMiras opened this issue Oct 20, 2019 · 2 comments
Closed

Leverage Docker image caching #2009

AndreMiras opened this issue Oct 20, 2019 · 2 comments
Assignees

Comments

@AndreMiras
Copy link
Member

The idea would be to speed up docker build (currently 10 minutes) by using the --cache-from flag.
Thing is we also need to docker push our successful builds ourselves as the DockerHub autobuild seems to mess up with COPY command leading to 100% cache bust when using this image.

@AndreMiras AndreMiras self-assigned this Oct 20, 2019
AndreMiras added a commit to AndreMiras/python-for-android that referenced this issue Oct 26, 2019
Revamping Travis and Docker setup introducing a `Makefile`.
The idea is to have a single entry point via `make` and reproducible
builds via Docker.
This pull request also starts introducing some docker layer cache
optimization as needed by kivy#2009 to speed up docker pull/push and
rebuilds from cache.
AndreMiras added a commit to AndreMiras/python-for-android that referenced this issue Oct 26, 2019
Revamping Travis and Docker setup introducing a `Makefile`.
The idea is to have a single entry point via `make` and reproducible
builds via Docker.
This pull request also starts introducing some docker layer cache
optimization as needed by kivy#2009 to speed up docker pull/push and
rebuilds from cache.
AndreMiras added a commit to AndreMiras/python-for-android that referenced this issue Oct 27, 2019
Revamping Travis and Docker setup introducing a `Makefile`.
The idea is to have a single entry point via `make` and reproducible
builds via Docker.
This pull request also starts introducing some docker layer cache
optimization as needed by kivy#2009 to speed up docker pull/push and
rebuilds from cache.
AndreMiras added a commit to AndreMiras/python-for-android that referenced this issue Oct 27, 2019
Revamping Travis and Docker setup introducing a `Makefile`.
The idea is to have a single entry point via `make` and reproducible
builds via Docker.
This pull request also starts introducing some docker layer cache
optimization as needed by kivy#2009 to speed up docker pull/push and
rebuilds from cache.
AndreMiras added a commit to AndreMiras/python-for-android that referenced this issue Oct 27, 2019
Revamping Travis and Docker setup introducing a `Makefile`.
The idea is to move the CI complexity from .travis.yml to `Makefile`.
That makes a single entry point via `make` command and reproducible
builds via Docker.
It makes it easy to run some commands outside docker, such as:
```sh
make testapps/python3/armeabi-v7a
```
Or the same command inside docker:
```sh
make docker/run/make/testapps/python3/armeabi-v7a
```
This pull request also starts introducing some docker layer cache
optimization as needed by kivy#2009 to speed up docker pull/push and
rebuilds from cache.
It also introduces other Docker images good practices like ordering
dependencies alphabetically or always enforcing `apt update` prior
install, refs:
https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
Subsequent pull requests would simplify the process furthermore and
leverage the cache to speed up builds.
AndreMiras added a commit to AndreMiras/python-for-android that referenced this issue Oct 28, 2019
The `docker build` stage tries to pull existing image related to the
on-going branch or pull request. Then this stage would build the new
image from this and push it for subsequent builds.
`DOCKER_USERNAME` is set as normal Travis environment variable while
`DOCKER_PASSWORD` as an encrypted one.
`TRAVIS_PULL_REQUEST` and `TRAVIS_BRANCH` are being setup by Travis.
Note that we're passing variables to `Makefile` explicitly on `make`
call rather than using the `--environment-overrides` flag. While it's
more verbose, it makes it easier to follow what's being used and when.
Closes kivy#2009
AndreMiras added a commit to AndreMiras/python-for-android that referenced this issue Oct 28, 2019
The `docker build` stage tries to pull existing image related to the
on-going branch or pull request. Then this stage would build the new
image from this and push it for subsequent builds.
`DOCKER_USERNAME` is set as normal Travis environment variable while
`DOCKER_PASSWORD` as an encrypted one.
`TRAVIS_PULL_REQUEST` and `TRAVIS_BRANCH` are being setup by Travis.
Note that we're passing variables to `Makefile` explicitly on `make`
call rather than using the `--environment-overrides` flag. While it's
more verbose, it makes it easier to follow what's being used and when.
Closes kivy#2009
AndreMiras added a commit to AndreMiras/python-for-android that referenced this issue Oct 28, 2019
The `docker build` stage tries to pull existing image related to the
on-going branch or pull request. Then this stage would build the new
image from this and push it for subsequent builds.
`DOCKER_USERNAME` is set as normal Travis environment variable while
`DOCKER_PASSWORD` as an encrypted one.
`TRAVIS_PULL_REQUEST` and `TRAVIS_BRANCH` are being setup by Travis.
Note that we're passing variables to `Makefile` explicitly on `make`
call rather than using the `--environment-overrides` flag. While it's
more verbose, it makes it easier to follow what's being used and when.
Closes kivy#2009
AndreMiras added a commit to AndreMiras/python-for-android that referenced this issue Oct 28, 2019
The `docker build` stage tries to pull existing image related to the
on-going branch or pull request. Then this stage would build the new
image from this and push it for subsequent builds.
`DOCKER_USERNAME` is set as normal Travis environment variable while
`DOCKER_PASSWORD` as an encrypted one.
`TRAVIS_PULL_REQUEST` and `TRAVIS_BRANCH` are being setup by Travis.
Note that we're passing variables to `Makefile` explicitly on `make`
call rather than using the `--environment-overrides` flag. While it's
more verbose, it makes it easier to follow what's being used and when.
Closes kivy#2009
AndreMiras added a commit to AndreMiras/python-for-android that referenced this issue Oct 28, 2019
The `docker build` stage tries to pull existing image related to the
on-going branch or pull request. Then this stage would build the new
image from this and push it for subsequent builds.
`DOCKER_USERNAME` is set as normal Travis environment variable while
`DOCKER_PASSWORD` as an encrypted one.
`TRAVIS_PULL_REQUEST` and `TRAVIS_BRANCH` are being setup by Travis.
Note that we're passing variables to `Makefile` explicitly on `make`
call rather than using the `--environment-overrides` flag. While it's
more verbose, it makes it easier to follow what's being used and when.
Closes kivy#2009
AndreMiras added a commit to AndreMiras/python-for-android that referenced this issue Oct 28, 2019
The `docker build` stage tries to pull existing image related to the
on-going branch or pull request. Then this stage would build the new
image from this and push it for subsequent builds.
`DOCKER_USERNAME` is set as normal Travis environment variable while
`DOCKER_PASSWORD` as an encrypted one.
`TRAVIS_PULL_REQUEST` and `TRAVIS_BRANCH` are being setup by Travis.
Note that we're passing variables to `Makefile` explicitly on `make`
call rather than using the `--environment-overrides` flag. While it's
more verbose, it makes it easier to follow what's being used and when.
Closes kivy#2009
AndreMiras added a commit to AndreMiras/python-for-android that referenced this issue Oct 28, 2019
The `docker build` stage tries to pull existing image related to the
on-going branch or pull request. Then this stage would build the new
image from this and push it for subsequent builds.
`DOCKER_USERNAME` is set as normal Travis environment variable while
`DOCKER_PASSWORD` as an encrypted one.
`TRAVIS_PULL_REQUEST` and `TRAVIS_BRANCH` are being setup by Travis.
Note that we're passing variables to `Makefile` explicitly on `make`
call rather than using the `--environment-overrides` flag. While it's
more verbose, it makes it easier to follow what's being used and when.
Closes kivy#2009
AndreMiras added a commit to AndreMiras/python-for-android that referenced this issue Oct 28, 2019
The `docker build` stage tries to pull existing image related to the
on-going branch or pull request. Then this stage would build the new
image from this and push it for subsequent builds.
`DOCKER_USERNAME` is set as normal Travis environment variable while
`DOCKER_PASSWORD` as an encrypted one.
`TRAVIS_PULL_REQUEST` and `TRAVIS_BRANCH` are being setup by Travis.
Note that we're passing variables to `Makefile` explicitly on `make`
call rather than using the `--environment-overrides` flag. While it's
more verbose, it makes it easier to follow what's being used and when.
Closes kivy#2009
AndreMiras added a commit to AndreMiras/python-for-android that referenced this issue Oct 28, 2019
The `docker build` stage tries to pull existing image related to the
on-going branch or pull request. Then this stage would build the new
image from this and push it for subsequent builds.
`DOCKER_USERNAME` is set as normal Travis environment variable while
`DOCKER_PASSWORD` as an encrypted one.
`TRAVIS_PULL_REQUEST` and `TRAVIS_BRANCH` are being setup by Travis.
Note that we're passing variables to `Makefile` explicitly on `make`
call rather than using the `--environment-overrides` flag. While it's
more verbose, it makes it easier to follow what's being used and when.
Closes kivy#2009
AndreMiras added a commit to AndreMiras/python-for-android that referenced this issue Oct 28, 2019
The `docker build` is now running alongside the `tox` testing as part of
the `pre checks` stage. It also fully leverages the docker image cache.
The newly built image is pushed to be available for subsequent build
stages and pull requests.
Two environment variables `DOCKER_USERNAME` and `DOCKER_PASSWORD` got
added to the Travis UI for making image pushing possible. Both `push`
and `pull` features fail silently with `|| true` as they should be
optional for a valid build.
`TRAVIS_PULL_REQUEST` and `TRAVIS_BRANCH` are built-in Travis variables.
Note that we're passing variables to `Makefile` explicitly on `make`
call rather than using the `--environment-overrides` flag. While it's
more verbose, it makes it easier to follow what's being used and when.
With this change, `Python 3 arm64-v8a` the longest `testapps` build that
was taking ~26 minutes to complete is now taking ~20 minutes when
hitting docker layers cache (most of the time). The shorter build
`Rebuild updated recipes` was taking ~9 minutes and is now down to ~3
minutes.
Closes kivy#2009
AndreMiras added a commit to AndreMiras/python-for-android that referenced this issue Oct 29, 2019
Revamping Travis and Docker setup introducing a `Makefile`.
The idea is to move the CI complexity from .travis.yml to `Makefile`.
That makes a single entry point via `make` command and reproducible
builds via Docker.
It makes it easy to run some commands outside docker, such as:
```sh
make testapps/python3/armeabi-v7a
```
Or the same command inside docker:
```sh
make docker/run/make/testapps/python3/armeabi-v7a
```
This pull request also starts introducing some docker layer cache
optimization as needed by kivy#2009 to speed up docker pull/push and
rebuilds from cache.
It also introduces other Docker images good practices like ordering
dependencies alphabetically or always enforcing `apt update` prior
install, refs:
https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
Subsequent pull requests would simplify the process furthermore and
leverage the cache to speed up builds.
@Julian-O
Copy link
Contributor

@AndreMiras: Is this still an issue?

@AndreMiras
Copy link
Member Author

We haven't solved it in p4a, but I think it's better documented in stackoverflow now.
https://stackoverflow.com/questions/54574821/docker-build-not-using-cache-when-copying-gemfile-while-using-cache-from
Also it's probably not critical and could be closed

@Julian-O Julian-O closed this as not planned Won't fix, can't repro, duplicate, stale Nov 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants