By Tom Granot in Tutorials — Mar 5, 2019

Continuous Integration & Deployment For Android Using CircleCI - Top To Bottom Tutorial

A (not very short) tutorial on building a CI/CD pipeline for your Android apps. If you've never done this before, this guide is probably for you.

Author's note

This was originally published on a Hugo site over at GitHub pages, and was copy-pasted over to Ghost when I moved my blog here. Excuse any formatting discrepancies, and please feel free to email me and ask me to repair something. We might even strike up a conversation about something else altogether. Go figure... :)

Premise

As part of my work for Smart Receipts - an expense tracking app company, I was recently tasked with setting up a full CI (Continuous Integration) & CD (Continuous Deployment) pipeline for the company's Android & iOS apps. This post deals with the Android part, and a follow-up on iOS will be published sometime in the near future.

Contact Me

Feel free to reach out to my email with questions and thoughts - use tom @ this lovely website for that. If your comment merits an edit, I'll probably make a revision and link back to your site. Ah, link juice!

Some Context

Running a mobile app business requires a lot of technical logistics. Other than integrating the code written by your developers into a single place (like a git repo), you'll need a lot of other things done in the process:

Keep track of versioning
Build the app
Run test suites
Publish to app store(s)

And other things that pop up along the way.

It used to be that everyone either did all of that by hand, or ran a dedicated CI server like Jenkins that took care of most of the work for them. Recently, though, a few hosted services became available on the cheap - most notably Travis-CI, CircleCI and JetBrains' TeamCity (the guys behind IntelliJ IDEA and PyCharm, among other things).

For us, there was a manual process that we did each time we had a new release that we wanted to automate. The process was divided roughly to the following steps:

New Code - A push is made to a certain release_VERSION branch in our GitHub repo. This can be a new release, a bugfix or a cosmetical patch, but either way it needs to go live.
Version Bumping - We bump the version number in our app/build.gradle file, and push that change to the repo.
Handling Multiple Changes - In practice, we collected a lot of these changes before we started the manual process - so in each iteration of the process, we had a few commits lined up for deployment.
Building - We create a release build for all build variants of our app.
Testing - We run tests on all of the variants.
Publishing - We then publish all of the variants to the various app stores.
Notifying - Finally we notify a certain e-mail address, reserved for this purpose, about the success or failure of the process. This email should preferably include a complete list of commits that occurred in the current branch, but not in the previous one - to give a quick sense of the release notes that we need to develop

In practice, I achieved full automation on about 90% of this, but I'm happy with the results nonetheless.

Prep Work

Choosing a CI Server/Hosted Service

After a few hours of playing around with Jenkins, I figured out most of what needs to be done aside from publishing to the app store. During the work, though, I realised that we have a cost problem here - Jenkins can rack up quite a bill if left unmonitored.

After looking around for a bit - and noting that Smart Receipts is completely open source - I figured out that CircleCIhas the best offering for our use case. 4 free linux containers (which are great for Android projects) are included in their OSS plan, and that put the nail in the coffin.

Thinking Automatically

The process outlined above could almost be automated as-is. I'll re-write it again, this time as an automated process.

Listen for a GitHub Push - Specifically, a push to a certain release_VERSION branch in our GitHub repo.
Bump Version - Bump version in app/build.gradle file, and push that change to the repo.
Wait for 15 Minutes - To accept multiple commits that are all part of the same release. This is specific to our build process, and can be removed if you don't need it.
Build & Test - Build and test all app flavours.
Publish - Publish all variants to Play Store.
Notify - Send notice to e-mail.

We now have a nice, automatic process template we can start to implement.

Enter CircleCI

Setting Up Your Project

CircleCI is, at the end of the day, a server that spins containers. It waits for something to happen (for example, a push to a GitHub repo the user has privileges in), then spins a containers and runs a bunch of pre-defined tasks in it.

In our case, we'll be waiting for GitHub push, spin up a Linux container that has all the necessary goodies for an Android build, perform all the actions listed above and close the container.

A "Project" is just the name for your GitHub repo on the CircleCI service. When you sign up with GitHub, CircleCI will load up all of your repositories into a projects page, which will be the first thing you see when you sign up.
You can click the blue 'Set Up Project' button to start configuring your repo.

Writing a `config.yml` File

Setting up a project in CircleCI includes adding a .circleci folder, and inside it a .circleci/config.yml file. It will determine some important variables about your CI process, set restrictions and detail all the steps that need to happen during a successful build.

If you're running into permission problems, then clone your repo to your computer, open the repo's folder and do chmod -R 777 .circleci/. This makes sure the config file is accessible to everyone, including CircleCI.

CircleCI recognises your project as an Android/iOS project, and then suggests you use a pre-made template config.yml file, designed for your mobile operating system. Here's the one for Android - you can use it, if you wish.

Important Note: Regardless of which branch you want the build to run from, the original config.yml must be added to the master branch. If you're having problems with it leave a comment below or e-mail me and I'll elaborate further. This helped me quite a bit with understanding how to work with branches in CircleCI.

I chose to modify it quite a bit, and you can see below the full one I use for the Smart Receipts Android app (don't worry, there's a part by part breakdown below).

Note that once you're done writing your config.yml file, you need to push it into your repo - this should have no effect on your circleci build. You can then press "start building" in the CircleCI interface, to actually trigger your first build. It Will Probably Fail. Don't worry - there's a lot of massaging that needs to be done to get your first green build. Drop me a line if you really get stuck below.

Part-By-Part Breakdown

Version 2 is the most up to date version of CircleCI as of this writing.

A job in this context is things the machine needs to do once it's up and running. build, in which we'll write most of our config, encapsulates the entire process we're going to implement.
branches defines on what branches this process should run.See the docs for more options and info - in my case I chose to only run on branches on which the regex release_ returns true. In your case, you might be building from master - change that accordingly. This forum question is also a good resource for info.
The other variables are just used to set up the machine for Android deployments. One thing of note: I'm using the android:api-25 image, while the default config uses the android:api-25-alpha image. The alpha image gave me some trouble, so I opted for the stable one instead. You can see all the possible android images here.

steps are tasks performed during the build. Each run command executes a bash command, with the name attribute specifying the name of the command (you'll see it in the CircleCI interface when the build runs) and the command attribute specifying the actual command.
You can see I chose yo wait for 15 minutes when a commit arrives, to "collect" other, later commits before beginning the builds. In order to make sure no redundant builds are being performed, follow the instructions here to enable auto-cancellations of builds.
checkout goes to GitHub and clones a version of your repo. This will be used later for building & for pushing changes.
caching is crucial for making your builds quicker. I've used the original config settings for my build, but you can get more info here.
Note the comment at the bottom. I did not come across this issue, but I'm leaving it there (commented out) in case on of you do.

Remember I said I need to deal with versioning? Note that gradle allows for XXX.YYY.ZZZ versioning in app/build.gradle. Our versioning policy dictates that whenever a new release is cut, bump YYY by 1 and ZZZ by 20. This little piece of code does that.
The actual bumping is done by a small script I posted here, that performs some regex search & replace in order to bump the version. *Make sure to chmod 777 your script before using it!
Even if you don't need a version bumping script, this is a neat example of how to incorporate a standalone bash script in your build.
We'd also like to push the new version number to GitHub, before building and testing the app. See all the {VAR_NAME} thingies? These are CircleCI Environment Variables that can be set once, and are then available in each build afterwards as regular environment variables. This is a nice way for public / open-source projects to inject secrets into the app - things you'd rather not show on the project's build logs.
For this part of the code, I've set the following environment variables in CircleCI's online interface:
GITHUB_EMAIL - The GitHub e-mail associated with the account that controls the project.
GITHUB_USERNAME - The GitHub username associated with the account that controls the project.
GITHUB_API_TOKEN - A GitHub Personal Access Token with the "Repo" permissions associated with the account that controls the project. This is used to push to GitHub using HTTPS instead of SSH, reading me of the need to play with private/public keys. This question was a lot of help.
CIRCLE_PROJECT_REPONAME - This specifies the repo we're working on, and is also built-in. You might need to get rid of the .git at the end there - I didn't test it on my build, since I hard-coded the URL repo, and edited it in later.
CIRCLE_BRANCH - This is a built-in variable that specifies the branch we're working on.
There is a nice hack going on here with the commit message. Note that the commit message has a [ci skip] part to it ([skip ci] works too). This prevents the version-bumping push to GitHub from triggering another build. CircleCI sees such a note in a commit message or commit description and skips the build triggered by that commit. Here's some more info on the topic.

We have a few secrets (API Keys and the like) we need available on running apps, that we wanted stashed away from the private repo. We did that using a GPG-encrypted tar archive with all of our secrets inside, called secrets.tar. The encrypted file is stored in secrets.tar.gpg. We've also added a line in .gitignore that prevents the secrets.tar file from being checked in accidentally into source control.
You can, totally, just use environment variables for this and then echo them into the files. Heck, I'm doing it here with the GPG key. You can also choose from a few other options CircleCI suggests for dealing with secrets.
Note that in the last part of the tutorial, I will use this gpg-tar combination to hide away an important binary file, so this is worth the trouble.
The GPG key, then, is stored in the GPG_KEY environment variable, and is later echoed into a file and used to decrypt some secrets. It's actually kind of tricky to get the GPG key inside that variable - I've written a small guide here on how to do it properly.
What you do need to do, if you want to publish your app later, is get a service account from the Google Play Console. This will provide you with a few credentials - save them for later use. Specifically for now, it will give you a Play Store Service Key, in JSON format. Open the file in a text editor, copy all of the contents, and save them in an environment variable called PLAYSTORE_SERVICE_KEY. It will be used later in the tutorial.

This is a bit of logistics, and not really interesting. The Android SDK requires that we approve some licenses. If not properly handled, the return message from approving these licenses causes the build to fail. This is a little hack that accepts all the licenses and then catches the output and discards it. You can see more details here.
Every android project has dependencies - the next command takes care of downloading all of them.
Then we're saving the cache for later builds (remember how we restored it earlier?)

Now that we've done all the preparation, we can build the app. Note that we're using the newer Android App bundle format to bundle the two flavours of our app (Free and Plus).
Then, we're using a Gradle plugin called GPP - Gradle Play Publisher that takes care of publishing for us. For each flavour of your app, it creates a new publishFLAVOURNAMEFlavorReleaseBundle Gradle task, that does all the building. We'll configure some other parts of it soon, when we're done with the config.yml file.

This last bit deals with Artifacts, which is a way of persisting data between builds. I haven't touched the original config, and I'm not actively using it. It's there for the future, when I'll get around to using it.

Once you're done, you can add, commit and push that file into your repo. Then, make sure to click "Start Building" in the CircleCI interface to trigger your first build. Feel free to cancel it - we'll check it in the next step anyways.

Testing The Setup - Moving Between Branches

We can now test the setup. This is a bit tricky: Note that we'd need to manually edit a branch to trigger a build (since we decided that only specific branches - like a release_XXXXX branch - merit a build). For now, we'll do it manually - and after this is live, we'll know the functionality is there.

I'm working with a release_4.18.0 branch on my repo, that was laying there before I started working on this. You can of course choose whichever regex you like and whatever branch you like, this is just for the sake of example.

So I'll checkout that branch, and then copy over the files I've created from the master branch so everything will work smoothly:
Make sure you're in the main directory after you checkout the branch!!

git checkout release_4.18.0
git show master:scripts/version-bumper.sh > scripts/version-bumper.sh
chmod -R 777 scripts/
mkdir .circleci
git show master:.circleci/config.yml > .circleci/config.yml
git add .
git commit -m"adding circleci config file and version bumping to release branch"
git push

This should indeed trigger a build, and the build should fail - that's perfectly fine, we're just trying to make sure our initial setup is up and running.

Setting environment variables

As you noticed before, we have a few variables of the form ${VAR_NAME} in .circleci/config.yml. These are just regular environment variables, that are configured inside CircleCI's interface. These are all the ones I've used, so you can check your interface to see you have all of them.

GITHUB_API_TOKEN - A GitHub Personal access token to a GitHub account that has the :repo privilege on the project's repo.
GITHUB_USERNAME - The Github username attached to the account (note that we don't want to use the CIRCLE_USERNAME built-in environment variable since we're always using the same user for every build. If you're interested in having each user build for themselves, you can use that setting instead (but note that you'll need to define additional API tokens for each of the users and write some logic that changes between API tokens based on that).
GITHUB_EMAIL - The GitHub email attached to the account.
GPG_KEY - This is tricky. I used this trick to get it working. It's used for secrets extraction during the build.
PLAYSTORE_SERVICE_KEY - This is a JSON key that we got while opening a Google Play Console service account.
KEY_ALIAS - You chose this value when opening the Service Account.
KEY_PASSWORD - You chose this value when opening the Service Account.
STORE_PASSWORD - You chose this value when opening the Service Account.

We now have everything set up, except for the publishing part of the app. Let's configure it now. On to Deployment!

Final Touches - Getting Ready To Publish

We're almost done - now we just have to configure some things for the Gradle Play Publisher, and sign our app (so the Play Store will not reject it).

Setting Up Gradle Play Publisher

The GPP has great documentation - most of this is taken directly from here.

First, adding the following to the top of your app/build.gradle file (or in the build.gradle file inside your main repo - depending on the way you set up your project):

plugins {
    id("com.android.application")
    id("com.github.triplet.play") version "2.1.0"
}

Note that I used GPP version 2.1.0, and that this is one of the ways you can set plugins for Gradle - there are two possible ways, both of which should work fine.

Now, inside the android block, add the following inner block:

    signingConfigs {
        release {
            // You need to specify either an absolute path or include the
            // keystore file in the same directory as the build.gradle file.
            storeFile file("../upload_key.jks")
            storePassword System.getenv("STORE_PASSWORD")
            keyAlias System.getenv("KEY_ALIAS")
            keyPassword System.getenv("KEY_PASSWORD")
        }
    }

This gets the environment variables required for the signing, as well as the signing key.

Then, at the bottom of the file (but before any preBuild statements you may have) put the following:

play {
    serviceAccountCredentials = file("GPLAY_KEY.json")
    track = "beta"
    userFraction = 1
    releaseStatus = "completed"
}

Remember we saved the JSON key as an environment variable? This uses that key, and places it inside the app folder as needed.

In addition, you can pick which track you want your app to go live on, which release status to assign to it and what percentage of users should get to see it. I chose 100 percent on beta with a release status of completed, by the way - just like in the example above.

If you want to find out more about the nuts and bolts of this, you can read about signing via gradle in the Android docs here.

Protecting The Upload Key

The thing that actually sings your app is a .jks file you received while opening the Google Play Service Account. Since this is a binary, we can't store it inside an environment variable.

Since I already have a tarball containing other secrets, I chose to add this file into that tarball, and just decrypt it with the rest of them.

These are the steps to stash away your upload key, while making sure it's available in the main folder when needed. I've skipped the commands since dealing with GPG keys is out of the scope of this article - see the guide here for more details:

Add the upload_key.jks to the repo's main folder, but do not add it to Git just yet.
Decrypt your secrets.tar archive (see above in the .config.yml for an example).
Add the file to the archive.
Encrypt the archive using your GPG key.
Add a line in .gitignore with the file's name - upload_key.jks.
Add, commit and push all the files to the remote repo.

This should take care of signing the app.

Final testing

Push a minor change to the release_VERSION branch, and make sure your build is green.

One interesting note - if everything seems to work but you're getting a 500 Internal Server Error from the Google Play Store API, make sure there isn't an app already in the track. If there is, remove it and try again - this solved it for me.

Conclusion

This was a long process. It's not easy as I thought, and I've followed a six or so different guides until I've found my mix that's written here. Please, feel free to leave a comment or contact me with questions - I hope I'll be able to help.

Happy CIing!