Compiling AOSP in Docker

Jared Dantis -

I've always been into custom ROMs, ever since I got the chance to flash CyanogenMod (now LineageOS) on my mom's Motorola Backflip. Back then you were lucky to get one or two software updates from your phone manufacturer. The Backflip, for example, got an upgrade to Android 2.1 Eclair for the US version (MB300), but remained on Android 1.5 Cupcake for everyone else (ME600), including my mom. CyanogenMod gave that phone a chance to taste 2.3 Gingerbread. At least until my cousin spilled water on it, killing the phone instantly, lol.

Fast-forward to 2023: I've been running PixelExperience on my Redmi Note 11 for a few months now. I've never really been a fan of MIUI. In particular, I never liked how liberally it killed my apps once they were in the background. (The phone has 6GB of RAM – why kill my apps?) Recently, however, the maintainer for PixelExperience on my device quit, and so I've been running a build that's slightly out of date. So I decided to try my hand at compiling PixelExperience from source.

Just thought I'd write a quick post on compiling AOSP in a Docker container, mostly so I have something to refer to if I do this again in the future :-)

Prerequisites

You will need:

  • A working installation of Linux (amd64) on your computer. You can't do this on ARM (sorry, Apple silicon users). You can technically do this in a virtual machine or WSL2 on Windows, but in my experience builds take practically half the time when performed in a bare-metal Linux environment.
  • A lot of free storage space, around 250GB minimum. The Android source code takes about 100GB on my system, and each build generates about 80-100GB of data, including the final ZIP file you flash with your phone's recovery. ccache also caches compiled files, for which 50GB is a good amount of space to allocate at minimum.
  • A lot of RAM, 24-32GB minimum (swap included). I have 16GB on my system and it wouldn't go past the Android.bp files and generating ninja file at out/soong/build.ninja step before running out of memory. I made a 16GB swapfile, enabled it, and the build system seemed to be happy with that. If you're going to be building regularly, however, I recommend getting actual 32GB RAM, as swapping memory to disk will wear out your drive really quickly.
  • I also recommend doing this on an SSD, since the Android build process involves reading and writing a lot of small files, which will be incredibly taxing (read: slow) on a spinning disk drive.

If you've read this far after having read the post title, chances are you already have Docker. If not, follow your Linux distribution's guide to installing Docker and Docker Buildx. (On Arch, for example, it's as easy as sudo pacman -Syu docker docker-buildx.)

Why Docker?

Docker is an extremely easy way to containerize an application. I chose this route primarily because I didn't want to pollute my Arch Linux installation with packages that would only be used for building Android, which I wouldn't be doing often anyway. I also didn't want to triple-boot Windows, Arch, and Ubuntu, so Ubuntu in Docker seemed like a no-brainer.

Building the Ubuntu image

This part is easy. Create a folder where you'll store everything related to AOSP. I'll use /home/jared/android as an example here – make sure to change it to the proper path on your system.

In that folder, create a file called Dockerfile with the following content:

To build the image, run:

cd /home/jared/android
docker buildx build -t aosp-builder \
    --build-arg git_username="your name" \
    --build-arg git_email="your_email@example.com" .

Once this is built, you can reuse this image for your subsequent Android builds.

Starting the container

In the same folder, make a folder where the Android source code will be stored. I called mine pe for PixelExperience:

mkdir /home/jared/android/pe

Also create a folder for ccache to cache compiled code in:

mkdir /home/jared/android/ccache

Finally, start the container:

docker run -it --rm \
  -v /home/jared/android/pe:/opt/android/source \
  -v /home/jared/android/ccache:/opt/android/ccache \
  --security-opt apparmor=unconfined \
  --security-opt seccomp=unconfined \
  --security-opt systempaths=unconfined \
  aosp-builder

This will drop you into a Bash shell in /opt/android/source, ready for you to repo init and repo sync the Android source code. Follow your ROM's build instructions for this. This is also the command you will use every time you want to start an AOSP build container; no need to rebuild the image like in the previous section.

A note on Arch Linux

The official AOSP build guide recommends the latest LTS version of Ubuntu, which is 22.04 as of writing this post. There is also a Docker image for Ubuntu, which for most purposes will do the job if all you want is a minimal Ubuntu environment.

However, there are reasons for recommending a point release distribution like Ubuntu, as compared to a rolling release distribution like Arch, which I used for this. I ran into one of these reasons myself with dex2oatd failing to allocate memory during the latter stages of building Android:

dex2oatd F 05-21 21:23:06 39024 39024 mem_map_arena_pool.cc:65] Check failed: map.IsValid() Failed anonymous mmap((nil), 131072, 0x3, 0x22, -1, 0): Cannot allocate memory. See process maps in the log.

Further investigation revealed that this was probably caused by a regression in Linux 6.x, which Arch has been using for some time now. Ubuntu is still on Linux 5.x, so I tried booting linux-lts515 from the AUR, and the build finished successfully after that. You may not run into this, but it's something to be aware of, especially since Docker still uses the host kernel for the Ubuntu container.