Skip to content

WIP Push Docker image and build from it in sub-stages #2013

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

Conversation

AndreMiras
Copy link
Member

This is a follow-up for #2011 it also supersedes it as it's built on top. See description below:

The docker build is now running alongside the tox testing as part of
the pre checks stage. It also fully leverages the docker image cache.
The newly built image is pushed to be available for subsequent build
stages and pull requests.
Two environment variables DOCKER_USERNAME and DOCKER_PASSWORD got
added to the Travis UI for making image pushing possible. Both push
and pull features fail silently with || true as they should be
optional for a valid build.
TRAVIS_PULL_REQUEST and TRAVIS_BRANCH are built-in Travis variables.
Note that we're passing variables to Makefile explicitly on make
call rather than using the --environment-overrides flag. While it's
more verbose, it makes it easier to follow what's being used and when.
With this change, Python 3 arm64-v8a the longest testapps build that
was taking ~26 minutes to complete is now taking ~20 minutes when
hitting docker layers cache (most of the time). The shorter build
Rebuild updated recipes was taking ~9 minutes and is now down to ~3
minutes.
Closes #2009

Revamping Travis and Docker setup introducing a `Makefile`.
The idea is to move the CI complexity from .travis.yml to `Makefile`.
That makes a single entry point via `make` command and reproducible
builds via Docker.
It makes it easy to run some commands outside docker, such as:
```sh
make testapps/python3/armeabi-v7a
```
Or the same command inside docker:
```sh
make docker/run/make/testapps/python3/armeabi-v7a
```
This pull request also starts introducing some docker layer cache
optimization as needed by kivy#2009 to speed up docker pull/push and
rebuilds from cache.
It also introduces other Docker images good practices like ordering
dependencies alphabetically or always enforcing `apt update` prior
install, refs:
https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
Subsequent pull requests would simplify the process furthermore and
leverage the cache to speed up builds.
The `docker build` is now running alongside the `tox` testing as part of
the `pre checks` stage. It also fully leverages the docker image cache.
The newly built image is pushed to be available for subsequent build
stages and pull requests.
Two environment variables `DOCKER_USERNAME` and `DOCKER_PASSWORD` got
added to the Travis UI for making image pushing possible. Both `push`
and `pull` features fail silently with `|| true` as they should be
optional for a valid build.
`TRAVIS_PULL_REQUEST` and `TRAVIS_BRANCH` are built-in Travis variables.
Note that we're passing variables to `Makefile` explicitly on `make`
call rather than using the `--environment-overrides` flag. While it's
more verbose, it makes it easier to follow what's being used and when.
With this change, `Python 3 arm64-v8a` the longest `testapps` build that
was taking ~26 minutes to complete is now taking ~20 minutes when
hitting docker layers cache (most of the time). The shorter build
`Rebuild updated recipes` was taking ~9 minutes and is now down to ~3
minutes.
Closes kivy#2009
@AndreMiras
Copy link
Member Author

Build flow and time before the PR:
image
image

And after (on my fork):
image
Note that all the builds are going faster by 6 minutes except the osx one which doesn't run in Docker.
There's still room for improvements after this one gets merged.
Not only we're building faster, but also this code change makes it easier to play with docker locally the same way Travis is doing.

@AndreMiras AndreMiras changed the title Push Docker image and build from it in sub-stages WIP Push Docker image and build from it in sub-stages Oct 29, 2019
@AndreMiras
Copy link
Member Author

Just changed to WIP as it seems we cannot access secure variables from pull requests due to security restrictions https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
That's a real pity, but it makes sense it's disabled. I'll try to modify the pull request to take it into account

@AndreMiras
Copy link
Member Author

OK so let's not deal with docker push for on-going pull request for now since docker login is not possible in pull requests by Travis security restrictions.
The only thing we may do is to docker push only on merge to develop. But we can deal with this in a completely different pull request.
I adjusted #2011 to leverage the cache to some good enough level for now.

@AndreMiras AndreMiras closed this Oct 29, 2019
@AndreMiras AndreMiras deleted the feature/revamp_travis_file2 branch October 29, 2019 23:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Leverage Docker image caching
1 participant