How I set up a Jenkins node for iOS jobs

Sunday, July 15, 2018

Over my career I have been responsible for setting up and/or maintaining continuous integration environments. Most mobile engineers would recommend you use something like CircleCI or Travis CI but I have always been a big fan of Jenkins. Sure it is a pain in the ass to set up and can be annoying to maintain, but it is worth it to have total control of your continuous integration environment. Some cloud solutions are constrained or become difficult if not impossible to customize whereas with Jenkins the only limitation is how much time you have. With the plethora of plugins and the ability to write your own there really isn't anything Jenkins can't do.

Jenkins supports distributed builds which means you need to create nodes that the master Jenkins server can delegate work to. For web or backend projects this typically means spinning up AWS instances for your nodes. But for iOS projects we need macOS and since AWS doesn't support that yet we need to look elsewhere. There are cloud-based alternatives for macOS, such as MacStadium, but until your company and/or project gets big enough you'll probably end up running a bunch of headless Mac Minis as your Jenkins nodes.

After spending years spinning up Mac Minis for Jenkins I thought it would be beneficial to write the process down so other mobile developers could benefit from my experience. This article will be written using the assumption that you already have a master Jenkins server up and running. I hope to write an article on how to do this someday but I find that most mobile developers inherit a continuous integration environment from the backend where this heavy lifting has already been done.

The process of setting up a Jenkins node for iOS can be split into three distinct sections:

Configuring the Administrator user
Creating and configuring the Jenkins user
Connecting the node to the master Jenkins server

This article was written using a fresh install of macOS High Sierra v10.13.5.

1. Configuring the Administrator user

This is probably incredibly obvious but the first thing we need to do is get some hardware with macOS on it. I recommend a Mac Mini because of its cost/power ratio but anything will do. You should also do a fresh install of macOS because we want to ensure this machine has absolutely nothing running in the background except for what we configure.

If you have physical access to the hardware then you are good to go but I would recommend putting it where it will be permanently housed and use the Screen Sharing app to connect. You may want to give your Mac a static IP to make accessing it easier.

Once you've got your fresh install of macOS up and running and a strong Administrator password set you are ready to move on to the bulk of work.

Clean up the dock

First, let's get rid of all that garbage that is in the dock. We are only going to need:

Terminal: Running commands in the terminal is the most common thing you will do.
Safari: You will often find yourself referencing a website or connecting to the web interface of your master Jenkins server.
App Store: Keeping your nodes up to date is critical. You can attempt to automate this but if you can't then it helps seeing that big red numbered badge when you screen share into your nodes.
System Preferences: Updating or verifying your system preferences is a very common task.

Update Energy Saver settings

Open the "Energy Saver" pane in the System Preferences app and ensure it is configured as follows:

Turn the display off after 5 minutes. The node will be headless most of the time so it doesn't matter if the display goes to sleep.
Enable "Prevent computer from sleeping automatically when the display is off". Because this is a CI node we don't ever want it to sleep. It should be permanently connected to the master Jenkins server listening for work.
Enable "Put hard disks to sleep when possible". I am assuming your hardware has an SSD so it really doesn't matter if it goes to sleep. But if you have a magnetic drive you should really give those spinning disks a break whenever you can because otherwise they will be going 24/7.
Enable "Wake for Wi-Fi network access". Our node never should go to sleep but if somehow does you want to ensure that screen sharing will wake it up.
Disable "Enable Power Nap". We don't want our node doing random work. The only thing it should be doing is listening for jobs and executing them.

Turn off Bluetooth

Typically CI nodes are headless and locked up in a server room. You should disable Bluetooth because it is a possible attack vector. If you need to use the hardware directly you can simply plug in a keyboard and mouse.

Turn off Wi-Fi

Wi-Fi is another possible attack vector. Your CI nodes should be connected via gigabit Ethernet to ensure the fastest and most stable connection possible.

Install the Xcode command line tools

We are going to need some sort of compiler to build the tooling we use such as Homebrew, git, fastlane, etc.

Run the command xcode-select --install in the Terminal.app and press the "Install" button in the dialog that appears. You will then be presented with a progress UI as the tools are downloaded and installed.

Do not install the Xcode command line tools by downloading the full version of Xcode. Later we are going to use a tool to manage our Xcode installs so that we can have multiple at once.

Enable Developer Mode

I don't fully understand why we need to do this but it has something to do with being able to run Xcode correctly.

Run DevToolsSecurity -enable and enter your Administrator password when prompted.

Install Homebrew

If you have ever done any development on macOS odds are you have used Homebrew. It is the premier package manager on macOS and is my goto solution for installing all sorts of dev tools that I use. For our node we are going to use Homebrew to install git (assuming that is the VCS you use). If you have heard of Homebrew before I highly recommend reading about it.

Run /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" to install Homebrew. You will be presented with a number of configuration options but I always leave everything the default. Once Homebrew has finished installing you can run brew doctor and brew update to ensure everything is working properly.

Install your version control system (VCS)

Our nodes are going to need some way to access the source code to run their jobs. The easiest way to do this is to install the same version control system that we use to develop our apps. We installed Homebrew in a previous step to make installing our VCS easier especially if we are using an extremely common one like git.

Run brew install git to get the latest version which should be backwards compatible with any versions your developers are using on their machines. Installing it through Homebrew will ensure that we can update git separate from any other tooling we install, particularly Xcode which bakes a version of git into it.

Install wget

Later on in the process we are going to need to download some files from our master Jenkins server. I find that wget is the easiest way to do this but you are more than welcome to omit it and use whatever tooling you prefer.

Run brew install wget to install it.

Install rbenv

Any iOS developer worth their salt knows that Ruby has become integral to iOS development because of fastlane. Since we are going to install Ruby tooling we should use rbenv to manage our Ruby installations. This way we can ensure that a particular version of Ruby is used for the entire project. Both developers working on their local machines and CI nodes can be in perfect sync.

Run git clone https://github.com/rbenv/rbenv.git ~/.rbenv to install rbenv and then git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build to install the ruby-build plugin.

Set up .bash_profile

Now that we have the building blocks of our toolchain installed we must update (or create) our .bash_profile to ensure we have proper access.

Copy the following into ~/.bash_profile:

rbenv_Path="$HOME/.rbenv/bin"
export PATH="/usr/local/bin:$rbenv_Path:$PATH"

eval "$(rbenv init -)"

This ensures that Homebrew and rbenv are looked at before any other tooling that appears in your PATH environment variable. To verify that it was successful, create a new Terminal shell and run type rbenv. You should see "rbenv is a function".

Install Ruby

With rbenv properly configured it is time to install the version(s) of Ruby that we need for our project(s). At the time of this writing the latest stable release of Ruby is v2.5.1, so if we want to use that we would run rbenv install 2.5.1 to install it and rbenv global 2.5.1 to set it to be the global default.

You should install whatever Ruby version is referenced by the .ruby-version file in your project. If this file doesn't exist then you should go to the root directory of your project and run rbenv local 2.5.1 to create the .ruby-version file. This will ensure that a specific version of Ruby is used whenever a Ruby command is run from inside your project directory.

Install fastlane

With Ruby properly set up we can now install the Swiss Army knife of iOS developer tooling, fastlane, by running the command gem install fastlane. We are installing fastlane as the Administrator user because we are going to use the Fastlane Credentials Manager to hold the username and password of the Apple ID that will be used to download Xcode.

Add user to the Fastlane Credentials Manager

While Xcode is free you need an Apple ID to log in to developer.apple.com to download it. We should create an Apple ID that will only be used by our CI nodes and have absolutely no ability to modify the information associated with our App Store account.

If we assume the email for this Apple ID is "ci.ios@mycompany.com" then we would run fastlane fastlane-credentials add --username ci.ios@mycompany.com and enter the password for this user when prompted. It will be stored in the Keychain so don't worry too much about security.

Install Xcode

To manage our Xcode installations we will use the Ruby gem xcode-install by running gem install xcode-install.

It will not only allow us to install Xcode via the command line but also quickly switch between multiple versions of Xcode that have been installed. This is incredibly useful when you are trying to migrate to the latest version of Xcode.

As of the time of this writing the latest version of Xcode is v9.4.1 which we can install by running xcversion install 9.4.1. This is going to take a long ass time to download so be prepared to wait. Once it finishes downloading and extracting it will ask for an administrator password so it can copy Xcode and configure it to launch.

Remember that Xcode only comes bundled with the latest simulator version so if you test on any older versions you will need to manually install them. xcversion simulators --install="iOS 10.3.1 will get you the latest iOS 10.3 simulator for example.

Install Java Development Kit

Jenkins is written in Java so we have to run a Java program to connect to the master Jenkins server. Simply download the .pkg file from Oracle's website and run it. The installation should be quick and painless.

Because we are installing the Java Development Kit it makes it really easy to repurpose these nodes for Android jobs as well.

Download slave.jar

The Java program that is used to connect to the master Jenkins server is housed in a slave.jar file. Assuming your master Jenkins server is located at https://jenkins.mycompany.com you should be able to easily download the file using the following commands:

mkdir -p /usr/local/lib
wget https://jenkins.mycompany.com/jnlpJars/slave.jar
mv slave.jar /usr/local/lib/slave.jar

We put the slave.jar in /usr/local/lib because it is a "library" that we want to be available to all users, particularly the Jenkins user.

Install any other tooling that your project requires

We are almost done configuring the Administrator user. The last step is to install any other tooling that may be unique to your project. Maybe you need a version of Python, image-magick, SwiftLint or SwiftFormat, Carthage or CocoaPods. The world is your oyster so install everything you need but don't want the Jenkins user to have write access to.

This is also the time to install tools for your code review system. For example, for Phabricator I would run:

git clone https://github.com/phacility/libphutil.git /Applications/phabricator/libphutil
git clone https://github.com/phacility/arcanist.git /Applications/phabricator/arcanist

Which will ensure the arcanist tool is installed on your machine to allow you to arc patch diffs in your Jenkins jobs.

Remember that for a lot of the tooling you install here you will probably have to go back to your ~/.bash_profile and update the PATH environment variable.

2. Creating and configuring the Jenkins user

With the Administrator user completely configured we can move on to creating and configuring the Jenkins user. The idea is to put the Jenkins user into a sandbox where they only have access to the explicit tools they need. If the Jenkin's user or jobs are compromised this will minimize the fallout.

Create the Jenkins user

We are going to create the Jenkins user through the Terminal because creating a user in System Preferences will create a "normal" user which is not what we want.

Create the Jenkins user.
sudo dscl . -create /Users/jenkins

Set the user's shell to bash.
sudo dscl . -create /Users/jenkins UserShell /bin/bash

Set the user's name to "Jenkins".
sudo dscl . -create /Users/jenkins RealName "Jenkins"

Give the user a unique ID. dscl . -list /Users UniqueID | awk '{print $2}' | sort -ug will list all of the current user IDs in descending order. My advice would be increment the last number in the list which will probably be 501. Run sudo dscl . -create /Users/jenkins UniqueID 502 or whatever number you picked to set the user's ID.

Set the user's primary group ID to 20 which is for a "standard user".
sudo dscl . -create /Users/jenkins PrimaryGroupID 20

We are going to use /usr/local/ as the user's home directory because it is not a "real" user but a bot so we don't want them to have anything in the /Users/ directory.
sudo dscl . -create /Users/jenkins NFSHomeDirectory /usr/local/var/jenkins

Set the user's password.
sudo dscl . passwd /Users/jenkins hunter2 but ideally replace "hunter2" with a good password.

Add the user to the developer group so they can access Xcode.
sudo dscl . -append /Groups/_developer GroupMembership jenkins

Create the user's home directory.
sudo mkdir -p /usr/local/var/jenkins

Transfer ownership of that directory to the Jenkins user.
sudo chown -R jenkins:staff /usr/local/var/jenkins

Have the Jenkins user create a folder which will hold all of the data from our Jenkins jobs. sudo -u jenkins mkdir -p /usr/local/var/jenkins/workspace

Restart the machine and log in as the Jenkins user

Now that the Jenkins user has been created we can log in as them and configure the user. You will probably be asked to log in with an Apple ID but just ignore all of that. We want this user to be completely nondescript.

Clean up the dock

Just like with the Administrator user let's clean up the dock. We only need Terminal, Safari, App Store and System Preferences.

Install rbenv

Assuming we are using fastlane to run our tests, we should install rbenv to manage our Ruby installations just like we did with the Administrator user.

Run git clone https://github.com/rbenv/rbenv.git ~/.rbenv to install rbenv and then git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build to install the ruby-build plugin.

Set up .bash_profile

Just like with the Administrator user we must set up our .bash_profile to give us access to our toolchain.

Copy the following to ~/.bash_profile:

rbenv_Path="$HOME/.rbenv/bin"
export PATH="/usr/local/bin:$rbenv_Path:$PATH"

eval "$(rbenv init -)"

Remember to create a new Terminal window so the updated .bash_profile is loaded.

Install Ruby

Install whatever version of Ruby you installed for the Administrator user. Most likely rbenv install 2.5.1 followed by rbenv global 2.5.1.

Install Bundler

To ensure that every developer and CI node is using the same version of fastlane we can use Bundler. gem install bundler will install Bundler, bundle install will install all of the needed Ruby gems and bundle exec fastlane unit_tests can then run our tests.

3. Connecting the node to the master Jenkins server

We have fully configured our Administrator and Jenkins users. In theory we could manually pull down our source code as the Jenkins user, run our tests with bundle exec fastlane unit_tests and they should run fine. But we don't want to go through all that tedium constantly so let's create a new node on our master Jenkins server and have our Jenkins user connect to it.

Create a new node

The easiest way to create a new node is by going to https://jenkins.mycompany.com/computer/new.

Make sure you set the "Remote root directory" to /usr/local/var/jenkins and give your node some labels. I would recommend your labels be the name of the node as well as the platforms it supports, such as "ios" and "android". This allows you to easily blacklist nodes from jobs if you find out that a certain node is buggy and needs to be taken out of rotation but only for a certain platform.

Once the node is created keep the page open because it contains information you are going to need in the coming steps.

Make sure you are logged in as the Administrator user

We want to make some changes to ensure that the node will always connect to Jenkins when the Jenkins user logs in. We need to make these changes as the Administrator because the Jenkins user should not have the power to configure things such as this.

Create deamon plist file

We need to create a specially formatted plist so that macOS knows to launch our slave.jar program whenever the Jenkins user logs in. You can read about the details of the plist at launchd.info but the gist is copy the plist below and change two things to match your configuration:

Create a file at /Library/LaunchAgents/com.mycompany.jenkins-slave and copy the following into it:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>jenkins-slave</string>
    <key>UserName</key>
    <string>jenkins</string>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
    <key>WorkingDirectory</key>
    <string>/usr/local/var/jenkins</string>
    <key>StandardOutPath</key>
    <string>/usr/local/var/jenkins</string>
    <key>StandardErrorPath</key>
    <string>/usr/local/var/jenkins</string>
    <key>EnvironmentVariables</key>
    <dict>
        <key>PATH</key>
        <string>/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>
    </dict>
    <key>ProgramArguments</key>
    <array>
        <string>java</string>
        <string>-jar</string>
        <string>/usr/local/lib/slave.jar</string>
        <string>-jnlpUrl</string>
        <string>https://jenkins.mycompany.com/computer/{Node_Name}/slave-agent.jnlp</string>
        <string>-secret</string>
        <string>{SECRET_TO_REPLACE_FOR_THIS_NODE}</string>
    </array>
</dict>
</plist>

Update the -jnlpUrl value to be the URL for your node. It is probably something like https://jenkins.mycompany.com/computer/{Node_Name}/slave-agent.jnlp

Update the -secret value to be whatever the secret of your node is. This should be found on the page that appeared after you successfully created your node.

Configure launchctl to run the plist

Creating the plist file is not enough to have it automatically run when the Jenkins user logs in. We need to use launchctl to load the plist by running sudo launchctl load /Library/LaunchAgents/com.mycompany.jenkins-slave.

Configure the Jenkins user to log in automatically

Open up the "Users & Groups" pane in the System Preferences app and click on the "Login Options". You want to make the Jenkins user your "Automatic login" so that if there is a power surge or your machine reboots for some reason it will automatically log in as the Jenkins user and reconnect to the master Jenkins server.

Restart the machine

Now restart the machine and it should automatically log in as the Jenkins user and when you see the Java app appear in your Dock it means slave.jar has automatically run. Congratulations you now have a Jenkins node up in running.

If you have any Jenkins jobs configured your node should start getting them dispatched.

Final Thoughts

I'm sure there are some people absolutely screaming at their screen right now asking "why are you doing all of this manually?" and the answer is because I've never had to manage enough nodes that automating them was worth it. If I had to scale this up I would look at using something like Puppet or Ansible and probably switch over to cloud hosting with MacStadium so I could spin up hosts as demand dictated.

But for mobile teams in the single digits it is much easier to find some spare Mac Minis and get your CI infrastructure up ASAP. The sooner you integrate CI into your development process the sooner you can reap the rewards.