Android Graphics

Jim Huang ( )
Developer & Co-Founder, 0xlab
jserv@0xlab.org Sep 24, 2011 / Study-Area

Rights to copy
Copyright 2011 0xlab http://0xlab.org/
Attribution ShareAlike 3.0 Corrections, suggestions, contributions and You are free translations are welcome! to copy, distribute, display, and perform the work to make derivative works Latest update: Nov 20, 2011 to make commercial use of the work Under the following conditions Attribution. You must give the original author credit. Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one. For any reuse or distribution, you must make clear to others the license terms of this work. Any of these conditions can be waived if you get permission from the copyright holder. Your fair use and other rights are in no way affected by the above. License text: http://creativecommons.org/licenses/by-sa/3.0/legalcode

contact@0xlab.org

Agenda

(1) Binder IPC (2) Android Graphics (3) 2D and Accelerations (4) OpenGL|ES

Notice: Before looking into Android Graphics, you should be aware of the design of Binder IPC, otherwise you would get lost!

Binder IPC

Processes running on Android

$ ps ... root system root root root radio root media bluetooth root keystore shell root system system app_24 radio app_18 app_7 app_0 app_14 app_3 app_25 app_26 app_27 app_1 app_19 app_21 app_28 shell $ 37 42 43 44 45 46 47 48 49 50 51 52 53 67 115 124 135 144 165 197 208 219 234 254 266 285 293 301 311 323 1 1 1 1 1 1 1 1 1 1 1 1 1 47 47 47 47 47 47 47 47 47 47 47 47 47 47 47 47 52 248 768 3824 3796 628 4336 62224 16828 1216 776 1704 696 3356 172464 80028 80732 87848 89136 86136 73996 75000 72228 85336 74656 74912 71616 72184 74728 75408 856 156 260 564 560 264 672 27576 3736 572 316 432 336 160 32596 20728 20720 20324 24160 22736 17472 18464 17652 17836 19080 18100 16280 16572 17208 18040 316 c00aef2c c022950c ffffffff ffffffff c02588c0 ffffffff c00aef2c ffffffff c00aef2c c02a8424 c02588c0 c0050934 ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 00000000 0000875c afd0b6fc afd0bdac afd0bdac afd0c0cc afd0bdac afd0b844 afd0b6fc afd0c59c afd0b45c afd0c0cc afd0c3ac 00008294 afd0b6fc afd0c51c afd0c51c afd0c51c afd0c51c afd0c51c afd0c51c afd0c51c afd0c51c afd0c51c afd0c51c afd0c51c afd0c51c afd0c51c afd0c51c afd0c51c afd0b45c S S S S S S S S S S S S S S S S S S S S S S S S S S S S S R /sbin/ueventd /system/bin/servicemanager /system/bin/vold /system/bin/netd /system/bin/debuggerd /system/bin/rild zygote /system/bin/mediaserver /system/bin/dbus-daemon /system/bin/installd /system/bin/keystore /system/bin/sh /sbin/adbd system_server com.android.systemui com.android.inputmethod.latin com.android.phone com.android.launcher android.process.acore com.android.deskclock android.process.media com.android.bluetooth com.android.mms com.android.email com.android.providers.calendar com.android.protips com.android.music com.android.quicksearchbox com.cooliris.media ps

More than 30 processes (200+ threads).

6

IPC = Inter-Process Communication

Activity Activity Manager Window Manager Alarm Manager

Kernel

IPC Abstraction
Intent AIDL
More abstract

Intent The highest level abstraction Inter process method invocation AIDL: Android Interface Definition Language binder: kernel driver ashmem: shared memory

Binder

Method invocation

caller

callee

In the same process

Inter-process method invocation

caller
interface

caller
interface

How?
callee
interface

callee
10

Inter-process method invocation

caller
interface

caller
interface

Proxy
Binder in kernel
Binder Thread

callee

Stub callee

interface

11

android.os.Parcel

Delivering arguments of method

flatten

unflatten

transmit
12

UML Representation

<<interface>> implements

Proxy

Stub

13

UML Representation
caller

<<interface>> implements

calls

Proxy

Stub extends callee

14

AIDL
Auto generated from .aidl file

caller

<<interface>>

Proxy

Stub

callee
15

Use Case: Who calls onPause() in Activity?

2

Send message by Handler

Activity
3

queue Activity Manager Binder Thread #1

Looper

OnPause() is called in main thread

Main Thread

Kernel

Call schedulePauseActivity across process

16

Binder Multi-thread aware

Have internal status per thead Compare to UNIX socket: sockets have internal status per file descriptor (FD)

17

Binder

A pool of threads is associated to each service application to process incoming IPC (Inter-Process Communication). Binder performs mapping of object between two processes. Binder uses an object reference as an address in a processs memory space. Synchronous call, reference couting

Binder

Binder is different from UNIX socket

socket
internal status associated to FD

binder
associated to PID
(FD can be shared among threads in the same process)

read & write operation network transparency

stream I/O Yes

done at once by ioctl No

expected local only

20

Transaction of Binder
binder_write_read write_size write_consumed write_buffer read_size read_consumed read_buffer read buffer write buffer

if (ioctl(fd, BINDER_WRITE_READ, &bwt ) >= 0) err = NO_ERROR; else err = -errno;

21

Transaction of Binder
Process A and B have different memory space. They can not see each other. Kernel Binder Process B Process A

Copy memory by copy_from _user Then, wake up process B Kernel Binder Process B

Process A Copy memory by copy_to_user

Internally, Android uses Binder for graphics data transaction across processes. It is fairly efficient.

22

Real Case

Binder IPC is used for communicating between Graphics client and server. Taken from http://www.cnblogs.com/xl19862005/archive/2011/11/17/2215363.html

23

 Android / Anonymous SHared MEMory subsystem

system/core/cutils/ashmem.h
int ashmem_create_region(const char *name, size_t size) int ashmem_set_prot_region(int fd, int prot) int ashmem_pin_region(int fd, size_t offset, size_t len) int ashmem_unpin_region(int fd, size_t offset, size_t len)

Ashmem

a named memory block shared between processes that the kernel is allowed to free. This is notable as the kernel is not allowed to free standard shared memory. Similar to weak reference of Java. Useful to implement cache. Used in android.os.MemoryFile (Java), 2D memory allocator, etc.
24

Android Graphics

Surface
Source: frameworks/base/core/java/android/view/Surface.java

/* Handle on to a raw buffer that is being managed by the screen compositor */ public class Surface implements Parcelable { public Surface() {
mCanvas = new CompatibleCanvas();

} private class CompatibleCanvas extends Canvas { /* ... */ } }

Surface instances can be written to and restored from a Parcel. Surface instances can be written to and restored from a Parcel.

Delivering arguments of method

flatten

unflatten

transmit

Properties

Android SurfaceFlinger

Can combine 2D/3D surfaces and surfaces from multiple applications Surfaces passed as buffers via Binder IPC calls Can use OpenGL ES and 2D hardware accelerator for its compositions

Double-buffering using page-flip

from SurfaceFlinger to Framebuffer

PixelFlinger : software renderer

Render functions: pointx, linex, recti, trianglex Texture and color buffer: activeTexture, bindTexture,
colorBuffer, readBuffer, depthBuffer, BindTextureLod

Device framebuffer functions: copyPixels,

rasterPos2x, rasterPos2i

Optimizer: codeflinger (JIT assembler)

I/SurfaceFlinger(1931):OpenGLinformations: I/SurfaceFlinger(1931):vendor:Android I/SurfaceFlinger(1931):renderer:AndroidPixelFlinger1.2 I/SurfaceFlinger(1931):version:OpenGLESCM1.0

GGLAssembler GGLAssembler codeflinger's JIT Assembler codeflinger's JIT Assembler

2D and Accelerator

Skia
SkBitmap SkBitmap::Allocator GPU integration in Android 4.0

Case Study: WebKit rendering

Fill rectangle FillRectangles()

Set clip region Acquire font cache Show glyphs BatchBlit() Allocate font cache Surface composite Blit() TileBlit() Prepare / finish composite Clear() Blit(), memcpy()

2D Compositing
SRC

SRC & DST

1 -asrc
DST

Compositing
SRC & NO-DST

NO-DST & NO-SRC 1-adst adst

Enable 2D Accelerator

Android Graphics (HAL view)

Android Framework (Java) libandroid_runtime EventHub Surfaceflinger (service) Copybit (HW accelerated)

libui

LibGLES (libagl)

libpixelflinger

2D Accelerator for Android Graphics

libcopybit provides hareware bitblit operations which includes moving, scaling, rotation, mirroring, and more effects, like blending, dithering, bluring, etc. Removed since Android 2.3 Android has two copybit interfaces: Blit: moving / blending Stretch: scaling besides moving libcopybit is called by libagl which can do swapBuffers to do the framebuffer page flipping that can also be accelerated by libcopybit.

Copybit could improve the performance of page flipping Copybit could improve the performance of page flipping

Copybit operations
Copybit: 2D blitter Copybit: 2D blitter

PMEM:

Manipulate physically continuous memory

Hardware graphics/bitblt operations (blitter) needs physical continuous memory to manipulate. Android use libgralloc to allocate pmem (physical continuous memory) for android native buffer. pmem driver can support up to 12 devices, we have only one for copybit (Android, android native buffer) While running 0xbench, the peak size of the pmem allocated (mapped) is 25194496 bytes.
Take Qualcomm MSM7x25 for example: Take Qualcomm MSM7x25 for example: /dev/pmem /dev/pmem /dev/pmem_adsp /dev/pmem_adsp For multimedia codec, audio, video, camera For multimedia codec, audio, video, camera

gralloc & pmem

gralloc_alloc

Offset = Alloc(size), PMEM_CONNECT, PMEM_MAP, memset If success, new private_handle_t(fd, size, flags); gralloc module

gralloc_free

PMEM_UNMAP, deallocate(offset)

pmem memory
(mmaped virtual address)base 0x49422336, c06c4000, 6c4000 buffer base = base + offset Buffer 2 master_fd (mmap size: 47MB) Buffer 1 fd when being freed, this will be closed

Buffer 3

Quantum

Debug pmem, /sys/kernel/pmem_regions

pmem pmem can have up to 12 devices..., we have only one for copybit (Android native buffer)

gralloc/copybit structure
android_native_buffer_t buffer_handle_t handle; copybit_image_t native_handle_t* handle;

native_handle_t

private_handle_t
Fd (pmem dev) Offset Size Base Flags lockState

buffer_handle_t

libgralloc

virtual address

Android native buffer contains buffer_handle_t Which is native_handle * , defined in system/cutils/

private_handle_t::PRIV_FLAGS_U SES_PMEM

private_handle_t::LOCK_STATE_MAPPED

Functions in copybit HAL

BLIT (moving) Stretch (scaling) Alpha-blending Unused in real case Rotate Unused in real case

OpenGL|ES

Key Concepts
Android is moving to OpenGL|ES accelerated rendering since version 2.x Window systems already comprehend z-order 3D != 2D with depth OpenGL is object based

Describes a scene using its components and properties. Output quality is dependent on the renderer, not source

OpenGL Terminology
OpenGL
An API from Khronos (from SGI), for constructing a 3D object, doing operations on the object, and displaying it

Primitives
Triangles, Lines, Points, that can be specified through vertices to define an arbitrary shape

Texture
Small (!) bitmap to make objects more realistic

EGL
The EGL API defines a portable mechanism for creating GL contexts and windows for rendering into, which may be used in conjunction with different native platform window systems using the WSEGL layer

EGL
EGL (Embedded-System Graphics Library) is an interface between Khronos rendering APIs (such as OpenGL ES or OpenVG) and the underlying native platform window system. EGL handles graphics context management, surface/buffer binding, and rendering synchronization and enables high-performance, accelerated, mixedmode 2D and 3D rendering using other Khronos APIs. EGL Surfaces windows - on-screen rendering pbuffers - off-screen rendering pixmaps - off-screen rendering

from EGL to SurfaceFlinger

hgl = hardware hgl = hardware OpenGL|ES OpenGL|ES

agl = android software agl = android software OpenGL|ES renderer OpenGL|ES renderer

http://0xlab.org