Skip to content

Commit 9801865

Browse files
opacamopacam
authored
[LIBS - PART I] Initial refactor of library recipes (kivy#1944)
This commit will introduce a new cls attribute: `built_libraries`. This `built_libraries` attribute must be a dictionary. The keys are the full library name, e.g.: `libffi.so`. And the values must be the relative path of the library, e.g: `.libs`. So the this cls attribute for libffi recipe it would look like: ```built_libraries = {'libffi.so': '.libs'}``` This new cls attribute will allow us to detect library recipes and to refactor common operations that we do for those kind of recipes: - copy library into right location - implement `should_build` for library recipes Those two things will make that our rebuilds to be more consistent and I hope that it will be easier to maintain. **So, once explained a little this PR, in here we do:** - [x] Add attribute `built_libraries` - [x] Add methods to refactor common library operations: - [x] install_libraries - [x] get_libraries - [x] implement `should_build` for library recipes - [x] implement basic tests for the newly introduced `Recipe` methods - [x] Make use of those Library attributes/methods for some basic set of recipes: - [x] libffi - [x] openssl - [x] png - [x] jpeg - [x] freetype - [x] harfbuzz - [x] libcurl - [x] libzbar - [x] libiconv - [x] libexpat - [x] libogg - [x] libxml2 - [x] libxslt - [x] libshine - [x] libx264 - [x] libglob - [x] libsodium - [x] libsecp256k1 **Things to do in separate PRs**, to be easy to review and because those points will start to conflict with the `NDK r19 migration` thing just merged: - Make use of Library attributes/methods introduced in this PR for un covered library recipes in here. Here we have two situations: - library recipes that depends on android's STL library (almost all the work done in here will depend of the `NDK r19 migration` thing) - all remaining library recipes, which are not STL lib dependent and that are not implemented in here (because I was unable to perform a successful build with them using arch `arm64-v8a`...so I think it would be better to deal with those recipes in separate PRs and later...with the `NDK r19 migration` thing merged) **Notes about changed recipes:** all the recipes touched in here almost have the same changes: - remove `should_build` method (to make use of the one implemented in base class for library recipes) - remove the copy of the library (because that will be done automatically by the method `install_libraries` implemented in base class) - fixed the imports due to refactoring * [recipe-lib] Add attribute Recipe.built_libraries so we can refactor some essential operations performed for library recipes (like copy the library into the right location, or decide if we should build the library recipe again) * [recipe-lib] Add method Recipe.get_libraries This function will allows us to get a list of the built/context libraries. This will be useful to check if we have to built a library recipe or which library should be rebuild * [recipe-lib] Make `Recipe.should_build` decide the library build In case that the attribute built_libraries has been set, we will check the need of the recipe build, otherwise we will act as usual, forcing the build unless overwrote in subclass * [recipe-lib] Add method `Recipe.install_libraries` So we can: - control the copy of the library recipe - refactor the install of the lib for all library recipes * [recipe-lib] Make libffi a library recipe * [recipe-lib] Make openssl a library recipe and ... also make the imports from the right module * [recipe-lib] Make png a library recipe * [recipe-lib] Make jpeg a library recipe * [recipe-lib] Make freetype a library recipe and ... also make the imports from the right module * [recipe-lib] Make harfbuzz a library recipe and ... also make the imports from the right module * [recipe-lib] Make libcurl a library recipe and ... also make the imports from the right module * [recipe-lib] Make libzbar a library recipe and ... also make the imports from the right module * [recipe-lib] Make libiconv a library recipe and ... also make the imports from the right module * [recipe-lib] Make libexpat a library recipe and ... also: - make the imports from the right module - remove hardcoded arch * [recipe-lib] Make libogg a library recipe * [recipe-lib] Make libxml2 a library recipe and ... also make the imports from the right module * [recipe-lib] Make libxslt a library recipe and ... also make the imports from the right module * [recipe-lib] Make libshine a library recipe and ... also: - make the imports from the right module - remove the hardcoded cpu count when compiling * [recipe-lib] Make libx264 a library recipe and ... also:   - make the imports from the right module   - remove the hardcoded cpu count when compiling * [recipe-lib] Make libglob a library recipe * [recipe-lib] Make libsodium a library recipe and ... also:   - make the imports from the right module   - fix hardcoded arch   - enable cpu count when compiling * [recipe-lib] Make libsecp256k1 a library recipe and ... also make the imports from the right module * [tests] Add tests for library recipe * [NDK19] Fix libglob for android NDK r19 - change the `ARG_MAX` define, because it's already defined at `sysroot/usr/include/linux/limits.h` - Replaced `size_t` by Including the <stddef.h> header. Because found the solution at stackoverflow: As per C99, §7.17, size_t is not a builtin type but defined in <stddef.h> See also: - https://travis-ci.org/kivy/python-for-android/jobs/576392711#L5992-L6013 - https://stackoverflow.com/questions/26410466/gcc-linaro-compiler-throws-error-unknown-type-name-size-t
1 parent 3887d2b commit 9801865

File tree

22 files changed

+214
-163
lines changed

22 files changed

+214
-163
lines changed

‎pythonforandroid/build.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -575,6 +575,7 @@ def build_recipes(build_order, python_modules, ctx, project_dir,
575575
info_main('Building {} for {}'.format(recipe.name, arch.arch))
576576
if recipe.should_build(arch):
577577
recipe.build_arch(arch)
578+
recipe.install_libraries(arch)
578579
else:
579580
info('{} said it is already built, skipping'
580581
.format(recipe.name))

‎pythonforandroid/recipe.py

Lines changed: 59 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,25 @@ class Recipe(with_metaclass(RecipeMeta)):
106106

107107
archs = ['armeabi'] # Not currently implemented properly
108108

109+
built_libraries = {}
110+
"""Each recipe that builds a system library (e.g.:libffi, openssl, etc...)
111+
should contain a dict holding the relevant information of the library. The
112+
keys should be the generated libraries and the values the relative path of
113+
the library inside his build folder. This dict will be used to perform
114+
different operations:
115+
- copy the library into the right location, depending on if it's shared
116+
or static)
117+
- check if we have to rebuild the library
118+
119+
Here an example of how it would look like for `libffi` recipe:
120+
121+
- `built_libraries = {'libffi.so': '.libs'}`
122+
123+
.. note:: in case that the built library resides in recipe's build
124+
directory, you can set the following values for the relative
125+
path: `'.', None or ''`
126+
"""
127+
109128
@property
110129
def version(self):
111130
key = 'VERSION_' + self.name
@@ -479,9 +498,14 @@ def apply_patches(self, arch, build_dir=None):
479498

480499
def should_build(self, arch):
481500
'''Should perform any necessary test and return True only if it needs
482-
building again.
501+
building again. Per default we implement a library test, in case that
502+
we detect so.
483503
484504
'''
505+
if self.built_libraries:
506+
return not all(
507+
exists(lib) for lib in self.get_libraries(arch.arch)
508+
)
485509
return True
486510

487511
def build_arch(self, arch):
@@ -492,6 +516,19 @@ def build_arch(self, arch):
492516
if hasattr(self, build):
493517
getattr(self, build)()
494518

519+
def install_libraries(self, arch):
520+
'''This method is always called after `build_arch`. In case that we
521+
detect a library recipe, defined by the class attribute
522+
`built_libraries`, we will copy all defined libraries into the
523+
right location.
524+
'''
525+
if not self.built_libraries:
526+
return
527+
shared_libs = [
528+
lib for lib in self.get_libraries(arch) if lib.endswith(".so")
529+
]
530+
self.install_libs(arch, *shared_libs)
531+
495532
def postbuild_arch(self, arch):
496533
'''Run any post-build tasks for the Recipe. By default, this checks if
497534
any postbuild_archname methods exist for the archname of the
@@ -554,6 +591,27 @@ def install_libs(self, arch, *libs):
554591
def has_libs(self, arch, *libs):
555592
return all(map(lambda l: self.ctx.has_lib(arch.arch, l), libs))
556593

594+
def get_libraries(self, arch_name, in_context=False):
595+
"""Return the full path of the library depending on the architecture.
596+
Per default, the build library path it will be returned, unless
597+
`get_libraries` has been called with kwarg `in_context` set to
598+
True.
599+
600+
.. note:: this method should be used for library recipes only
601+
"""
602+
recipe_libs = set()
603+
if not self.built_libraries:
604+
return recipe_libs
605+
for lib, rel_path in self.built_libraries.items():
606+
if not in_context:
607+
abs_path = join(self.get_build_dir(arch_name), rel_path, lib)
608+
if rel_path in {".", "", None}:
609+
abs_path = join(self.get_build_dir(arch_name), lib)
610+
else:
611+
abs_path = join(self.ctx.get_libs_dir(arch_name), lib)
612+
recipe_libs.add(abs_path)
613+
return recipe_libs
614+
557615
@classmethod
558616
def recipe_dirs(cls, ctx):
559617
recipe_dirs = []

‎pythonforandroid/recipes/freetype/__init__.py

Lines changed: 11 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
1-
from pythonforandroid.toolchain import Recipe
1+
from pythonforandroid.recipe import Recipe
22
from pythonforandroid.logger import shprint, info
33
from pythonforandroid.util import current_directory
4-
from os.path import exists, join
4+
from os.path import join, exists
55
from multiprocessing import cpu_count
66
import sh
77

@@ -26,16 +26,7 @@ class FreetypeRecipe(Recipe):
2626

2727
version = '2.5.5'
2828
url = 'http://download.savannah.gnu.org/releases/freetype/freetype-{version}.tar.gz' # noqa
29-
30-
def should_build(self, arch):
31-
return not exists(
32-
join(
33-
self.get_build_dir(arch.arch),
34-
'objs',
35-
'.libs',
36-
'libfreetype.so',
37-
)
38-
)
29+
built_libraries = {'libfreetype.so': 'objs/.libs'}
3930

4031
def get_recipe_env(self, arch=None, with_harfbuzz=False):
4132
env = super(FreetypeRecipe, self).get_recipe_env(arch)
@@ -111,11 +102,14 @@ def build_arch(self, arch, with_harfbuzz=False):
111102
# First build, install the compiled lib, and clean build env
112103
shprint(sh.make, 'install', _env=env)
113104
shprint(sh.make, 'distclean', _env=env)
114-
else:
115-
# Second build (or the first if harfbuzz not enabled), now we
116-
# copy definitive libs to libs collection. Be sure to link your
117-
# recipes to the definitive library, located at: objs/.libs
118-
self.install_libs(arch, 'objs/.libs/libfreetype.so')
105+
106+
def install_libraries(self, arch):
107+
# This library it's special because the first time we built it may not
108+
# generate the expected library, because it can depend on harfbuzz, so
109+
# we will make sure to only install it when the library exists
110+
if not exists(list(self.get_libraries(arch))[0]):
111+
return
112+
self.install_libs(arch, *self.get_libraries(arch))
119113

120114

121115
recipe = FreetypeRecipe()

‎pythonforandroid/recipes/harfbuzz/__init__.py

Lines changed: 5 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
1-
from pythonforandroid.toolchain import Recipe
1+
from pythonforandroid.recipe import Recipe
22
from pythonforandroid.util import current_directory
33
from pythonforandroid.logger import shprint
44
from multiprocessing import cpu_count
5-
from os.path import exists, join
5+
from os.path import join
66
import sh
77

88

@@ -23,13 +23,7 @@ class HarfbuzzRecipe(Recipe):
2323
version = '0.9.40'
2424
url = 'http://www.freedesktop.org/software/harfbuzz/release/harfbuzz-{version}.tar.bz2' # noqa
2525
opt_depends = ['freetype']
26-
27-
def should_build(self, arch):
28-
return not exists(
29-
join(
30-
self.get_build_dir(arch.arch), 'src', '.libs', 'libharfbuzz.so'
31-
)
32-
)
26+
built_libraries = {'libharfbuzz.so': 'src/.libs'}
3327

3428
def get_recipe_env(self, arch=None):
3529
env = super(HarfbuzzRecipe, self).get_recipe_env(arch)
@@ -68,12 +62,12 @@ def build_arch(self, arch):
6862
_env=env,
6963
)
7064
shprint(sh.make, '-j', str(cpu_count()), _env=env)
71-
self.install_libs(arch, join('src', '.libs', 'libharfbuzz.so'))
7265

7366
if 'freetype' in self.ctx.recipe_build_order:
74-
# Rebuild freetype with harfbuzz support
67+
# Rebuild/install freetype with harfbuzz support
7568
freetype = self.get_recipe('freetype', self.ctx)
7669
freetype.build_arch(arch, with_harfbuzz=True)
70+
freetype.install_libraries(arch)
7771

7872

7973
recipe = HarfbuzzRecipe()

‎pythonforandroid/recipes/jpeg/__init__.py

Lines changed: 2 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,8 @@
11
from pythonforandroid.recipe import Recipe
22
from pythonforandroid.logger import shprint
33
from pythonforandroid.util import current_directory
4-
from os.path import join, exists
4+
from os.path import join
55
from os import environ, uname
6-
from glob import glob
76
import sh
87

98

@@ -16,15 +15,11 @@ class JpegRecipe(Recipe):
1615
name = 'jpeg'
1716
version = '2.0.1'
1817
url = 'https://github.com/libjpeg-turbo/libjpeg-turbo/archive/{version}.tar.gz' # noqa
18+
built_libraries = {'libjpeg.a': '.', 'libturbojpeg.a': '.'}
1919
# we will require this below patch to build the shared library
2020
# patches = ['remove-version.patch']
2121

22-
def should_build(self, arch):
23-
return not exists(join(self.get_build_dir(arch.arch),
24-
'libturbojpeg.a'))
25-
2622
def build_arch(self, arch):
27-
super(JpegRecipe, self).build_arch(arch)
2823
build_dir = self.get_build_dir(arch.arch)
2924

3025
# TODO: Fix simd/neon
@@ -59,10 +54,6 @@ def build_arch(self, arch):
5954
_env=env)
6055
shprint(sh.make, _env=env)
6156

62-
# copy static libs to libs collection
63-
for lib in glob(join(build_dir, '*.a')):
64-
shprint(sh.cp, '-L', lib, self.ctx.libs_dir)
65-
6657
def get_recipe_env(self, arch=None, with_flags_in_cc=False):
6758
env = environ.copy()
6859

‎pythonforandroid/recipes/libcurl/__init__.py

Lines changed: 5 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,18 @@
11
import sh
2-
from pythonforandroid.toolchain import Recipe, shprint, shutil, current_directory
3-
from os.path import exists, join
2+
from pythonforandroid.recipe import Recipe
3+
from pythonforandroid.util import current_directory
4+
from pythonforandroid.logger import shprint
5+
from os.path import join
46
from multiprocessing import cpu_count
57

68

79
class LibcurlRecipe(Recipe):
810
version = '7.55.1'
911
url = 'https://curl.haxx.se/download/curl-7.55.1.tar.gz'
12+
built_libraries = {'libcurl.so': 'dist/lib'}
1013
depends = ['openssl']
1114

12-
def should_build(self, arch):
13-
super(LibcurlRecipe, self).should_build(arch)
14-
return not exists(join(self.ctx.get_libs_dir(arch.arch), 'libcurl.so'))
15-
1615
def build_arch(self, arch):
17-
super(LibcurlRecipe, self).build_arch(arch)
1816
env = self.get_recipe_env(arch)
1917

2018
r = self.get_recipe('openssl', self.ctx)
@@ -31,10 +29,6 @@ def build_arch(self, arch):
3129
_env=env)
3230
shprint(sh.make, '-j', str(cpu_count()), _env=env)
3331
shprint(sh.make, 'install', _env=env)
34-
shutil.copyfile('{}/lib/libcurl.so'.format(dst_dir),
35-
join(
36-
self.ctx.get_libs_dir(arch.arch),
37-
'libcurl.so'))
3832

3933

4034
recipe = LibcurlRecipe()

‎pythonforandroid/recipes/libexpat/__init__.py

Lines changed: 6 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,38 +1,32 @@
11

22
import sh
3-
from pythonforandroid.toolchain import Recipe, shprint, shutil, current_directory
4-
from os.path import exists, join
3+
from pythonforandroid.recipe import Recipe
4+
from pythonforandroid.util import current_directory
5+
from pythonforandroid.logger import shprint
6+
from os.path import join
57
from multiprocessing import cpu_count
68

79

810
class LibexpatRecipe(Recipe):
911
version = 'master'
1012
url = 'https://github.com/libexpat/libexpat/archive/{version}.zip'
13+
built_libraries = {'libexpat.so': 'dist/lib'}
1114
depends = []
1215

13-
def should_build(self, arch):
14-
super(LibexpatRecipe, self).should_build(arch)
15-
return not exists(
16-
join(self.ctx.get_libs_dir(arch.arch), 'libexpat.so'))
17-
1816
def build_arch(self, arch):
19-
super(LibexpatRecipe, self).build_arch(arch)
2017
env = self.get_recipe_env(arch)
2118
with current_directory(join(self.get_build_dir(arch.arch), 'expat')):
2219
dst_dir = join(self.get_build_dir(arch.arch), 'dist')
2320
shprint(sh.Command('./buildconf.sh'), _env=env)
2421
shprint(
2522
sh.Command('./configure'),
26-
'--host=arm-linux-androideabi',
23+
'--host={}'.format(arch.command_prefix),
2724
'--enable-shared',
2825
'--without-xmlwf',
2926
'--prefix={}'.format(dst_dir),
3027
_env=env)
3128
shprint(sh.make, '-j', str(cpu_count()), _env=env)
3229
shprint(sh.make, 'install', _env=env)
33-
shutil.copyfile(
34-
'{}/lib/libexpat.so'.format(dst_dir),
35-
join(self.ctx.get_libs_dir(arch.arch), 'libexpat.so'))
3630

3731

3832
recipe = LibexpatRecipe()

‎pythonforandroid/recipes/libffi/__init__.py

Lines changed: 2 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
from multiprocessing import cpu_count
33
from pythonforandroid.recipe import Recipe
44
from pythonforandroid.logger import shprint
5-
from pythonforandroid.util import current_directory, ensure_dir
5+
from pythonforandroid.util import current_directory
66
import sh
77

88

@@ -31,8 +31,7 @@ class LibffiRecipe(Recipe):
3131

3232
patches = ['remove-version-info.patch']
3333

34-
def should_build(self, arch):
35-
return not exists(join(self.ctx.get_libs_dir(arch.arch), 'libffi.so'))
34+
built_libraries = {'libffi.so': '.libs'}
3635

3736
def build_arch(self, arch):
3837
env = self.get_recipe_env(arch)
@@ -46,10 +45,6 @@ def build_arch(self, arch):
4645
'--disable-builddir',
4746
'--enable-shared', _env=env)
4847
shprint(sh.make, '-j', str(cpu_count()), 'libffi.la', _env=env)
49-
ensure_dir(self.ctx.get_libs_dir(arch.arch))
50-
self.install_libs(
51-
arch, join(self.get_build_dir(arch.arch), '.libs', 'libffi.so')
52-
)
5348

5449
def get_include_dirs(self, arch):
5550
return [join(self.get_build_dir(arch.arch), 'include')]

‎pythonforandroid/recipes/libglob/__init__.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,13 @@
33
available via '-lglob' LDFLAG
44
"""
55
from os.path import exists, join
6-
from pythonforandroid.recipe import CompiledComponentsPythonRecipe
6+
from pythonforandroid.recipe import Recipe
77
from pythonforandroid.toolchain import current_directory
88
from pythonforandroid.logger import info, shprint
99
import sh
1010

1111

12-
class LibGlobRecipe(CompiledComponentsPythonRecipe):
12+
class LibGlobRecipe(Recipe):
1313
"""Make a glob.h and glob.so for the python_install_dir()"""
1414
version = '0.0.1'
1515
url = None
@@ -20,6 +20,7 @@ class LibGlobRecipe(CompiledComponentsPythonRecipe):
2020
# https://raw.githubusercontent.com/white-gecko/TokyoCabinet/master/glob.c
2121
# and pushed in via patch
2222
name = 'libglob'
23+
built_libraries = {'libglob.so': '.'}
2324

2425
depends = [('hostpython2', 'hostpython3')]
2526
patches = ['glob.patch']
@@ -60,7 +61,6 @@ def build_arch(self, arch):
6061
cflags.extend(['-shared', '-I.', 'glob.o', '-o', 'libglob.so'])
6162
cflags.extend(env['LDFLAGS'].split())
6263
shprint(cc, *cflags, _env=env)
63-
shprint(sh.cp, 'libglob.so', join(self.ctx.libs_dir, arch.arch))
6464

6565

6666
recipe = LibGlobRecipe()

‎pythonforandroid/recipes/libglob/glob.patch

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -911,7 +911,7 @@ diff -Nur /tmp/x/glob.c libglob/glob.c
911911
diff -Nur /tmp/x/glob.h libglob/glob.h
912912
--- /tmp/x/glob.h 1969-12-31 19:00:00.000000000 -0500
913913
+++ libglob/glob.h 2017-08-19 15:22:18.367109399 -0400
914-
@@ -0,0 +1,102 @@
914+
@@ -0,0 +1,104 @@
915915
+/*
916916
+ * Copyright (c) 1989, 1993
917917
+ * The Regents of the University of California. All rights reserved.
@@ -952,10 +952,12 @@ diff -Nur /tmp/x/glob.h libglob/glob.h
952952
+
953953
+#include <sys/cdefs.h>
954954
+#include <sys/types.h>
955+
+#ifndef ARG_MAX
955956
+#define ARG_MAX 6553
957+
+#endif
956958
+
957959
+#ifndef _SIZE_T_DECLARED
958-
+typedef __size_t size_t;
960+
+#include <stddef.h>
959961
+#define _SIZE_T_DECLARED
960962
+#endif
961963
+

0 commit comments

Comments
 (0)