Apple prevents their proud owners of shiny new Macintosh computers from installing illegit software via a gatekeeper, that only allows software that has been cryptographically signed by someone who can afford membership in the Apple Development program and that has run through a superficial malware check on the Apple cloud (called notarization).

Introduction

If you want to deliver software to Apple users, you really need to sign and notarize your binaries. Otherwise the system will refrain from running them, unless the user forces the system to do so.

Steps required:

• sign up for the Apple Developer Program
• obtain a certificate
• use the certificate to sign your software
• submit the signed software for notarization
• wait until the software has been notarized
• (optionally) embed a proof of a successful notarization in your application bundle
• distribute

Challenges in a CI system

Apple documents how to sign and notarize software (otherwise, nobody would be able to distribute Apple software). Also, (I think that) a lot of these steps are readily available via Xcode.

What I find undocumented is, how to do that in a fully automated way in the context of a CI system.

The main challenges are:

• the developers might not have a Mac at their disposition
• where to store the sensitive information, given that repositories are publicly viewable?
• the CI is essentially an untrusted machine

The practical problems to solve are:

• create certificates on the cmdline (on non-Apple systems)
• make the certificates available in the CI-job
• sign without user interaction
• notarize without user interaction

Requirements

Software requirements

To generate certificates, you will need openssl (which should be readily available in your Linux distribution).

For notarization, you need the macOS altool, at least version1.1.113, that comes with Xcode 10 (which in turn requires at least macOS10.14 - Mojave)

Social requirements

There are different types of certificates (the distinction is of social nature, rather than of technical nature).

For development purposes a binary ought to be signed with (at least) a Development certificate (that can be created by any member of an Apple Developer Program Group). For notarization, the binary must be signed with a Developer ID Application certificate. Only the Account Holder of the Apple Developer Program Group may add such certificates to the group.

Creating a certificate

A certificate consists of multiple parts, that are stored separately

• a Private Key (only known to you; it is stored in a file I will reference as ${KEYFILE})
• a Public Certificate (issued by a certificate provider, in this case Apple; it is stored in a separate file I will reference as ${CERTFILE})

To obtain a certificate from a certificate provider (Apple), you must submit a Certificate Request which is created with your Private Key. It is stored in a separate file which I will reference as ${CSRFILE}.

The first step is create your Private Key, namely a 2048bit RSA key. (There are other cryptosystems that are more secure, but these were not supported by Apple yet when I tried them):

1openssl genrsa -out "${KEYFILE}" 2048

Then use your Private Key to generate a Certificate Request:

1openssl req -new -sha256 -key "${KEYFILE}" -out "${CSRFILE}" -utf8

This will ask you a lot of questions that you should answer truthfully. When it asks your for your email address, you should probably give the one that is associated with your Apple ID.

Once done, submit the generated CSR (Certificate Signing Request) to https://developer.apple.com/account/resources/certificates to obtain a certificate signed by Apple.

Once you’ve downloaded the certificate, you need to merge it with the key file into a single PKCS#12 file (which I will reference as ${PFXFILE})

To do so, openssl wants us to use a PEM certificate (typically using a .pem extension). Unfortunately the file provided by Apple is in the wrong format (it is a Certificate, Version=3 file with a .cer extension), so we have to convert it first:

1openssl x509 -inform der -in "${CERTFILE}" -out "${CERTFILE%.cer}.pem"

The PKCS#12 file needs to be protected by a password (otherwise macOS will refuse to import the bundle later). We can create a rather secure password with openssl, but make sure to remember it, as we will need it when importing the certificate on the signing machine.

1PFXPASSWORD=$(openssl rand -hex 64)

Finally, we can combine the ${KEYFILE} with the PEM-file ${CERTFILE%.cer}.pem into the ${PFXFILE} using ${PFXPASSWORD}.

1openssl pkcs12 \
2   -export \
3   -password "pass:${PFXPASSWORD}" \
4   -in "${CERTFILE%.cer}.pem" \
5   -inkey "${KEYFILE}" \
6   -out "${PFXFILE}"

Scripts

Of course I don't want to type in all these commands every time I create a new certificate (request). So here's two scripts that help me.

Creating a Certificate Signing Request

This creates a new Private Key (if there isn't one already) and a CSR (with the current date in the filename).

Invoke it with a certificate name (like ./mkcert apple) and it will create a file apple.key with the private key and a file apple_20220202.csr (or whatever date is today) with the CSR.

 1#!/bin/sh
 2
 3CERT=${1}
 4KEYFILE="${CERT}.key"
 5DATE=$(date +%Y%m%d)
 6CSRFILE="${CERT}_${DATE}.csr"
 7CONFFILE=req.conf
 8
 9test -e ${CONFFILE} || CONFFILE=""
10if [ "x${CERT}" = "x" ]; then
11cat >/dev/stderr <<EOF
12usage: $0 <cert>
13
14creates a private key file '<cert>.key' (unless this file already exists),
15and a certificate signing request file '<cert>_${DATE}.csr'
16
17EOF
18exit 1
19fi
20
21encrypt=yes
22if [ "x${encrypt}" = "xno" ]; then
23  encryptflags="-nodesi -utf8"
24else
25  encryptflags="-utf8"
26fi
27
28if [ ! -e "${KEYFILE}" ]; then
29  openssl genrsa -out "${KEYFILE}" 2048  || exit 1
30fi
31openssl req ${CONFFILE:+-config ${CONFFILE}} -new -sha256 -key "${KEYFILE}" -out "${CSRFILE}" ${encryptflags}
32openssl req -text -noout -verify -in "${CSRFILE}"
33
34echo "================================================================"
35echo "        private key: ${KEYFILE}"
36echo "certificate request: ${CSRFILE}"

If you (like me) are tired of typing out all those answers, you can create a file named req.conf that contains all the answers like so:

 1[req]
 2distinguished_name = req_distinguished_name
 3prompt = no
 4[req_distinguished_name]
 5C = AT
 6ST = Styria
 7L = Graz
 8O = University of Music and Performing Arts Graz
 9OU = Institute of Electronic Music and Acoustics
10CN = IOhannes m zmölnig
11emailAddress = email@example.com

Creating a bundle with the private key and the public certificate

Similarly, once you've downloaded your certificate, here's a script that creates a bundle with both private key and public certificate merged. It will convert the certificate file if necessary and generate a password for you (which will be printed out, so you can later use it)

 1#!/bin/sh
 2
 3usage() {
 4cat <<EOF
 5usage: $0 <keyfile> <certfile>
 6
 7    merges <keyfile> and <certfile> into a single PKCS#12 bundle
 8    the name is the same as <certfile> but with the ".pfx" extension
 9EOF
10exit
11}
12
13KEYFILE=$1
14CERTFILE=$2
15
16if [ ! -f "${KEYFILE}" ]; then usage; fi
17if [ ! -f "${CERTFILE}" ]; then usage; fi
18
19PFXFILE="${CERTFILE%.*}.pfx"
20password=$(openssl rand -hex 64)
21
22if [ "${CERTFILE}" != "${CERTFILE%.cer}" ] ; then
23openssl x509 -inform der -in "${CERTFILE}" -out "${CERTFILE%.cer}.pem"
24CERTFILE="${CERTFILE%.cer}.pem"
25fi
26openssl pkcs12 \
27	-export \
28	-password "pass:${password}" \
29	-in "${CERTFILE}" \
30	-inkey "${KEYFILE}" \
31	-out "${PFXFILE}"
32
33echo "================================================================"
34echo "certificate: ${PFXFILE}"
35echo "   password: ${password}"

Workflow

Importing the certificate

Importing a certificate bundle (such as created above), is easy on the GUI, a bit less so on the cmdline.

In order to be able to use a certificate for signing, we need to establish a full Chain of Trust. At least my macOS 10.14 Mojave system lacked a number of intermediate certificates, so my Apple-issued Developer certificate was not accepted 🤔.

I found it simplest to create a new keychain on the fly, that contains my certificate bundle and the intermediate certificates downloaded from Apple.

In order to be able to use the keychain on a headless system, it must be unlocked (so you don't get a popup asking you for your password whenever you want to use it). For this, you create the keychain with a password (${keychainpass}), which is then used for unlocking...

 1# download intermediate certificates from Apple
 2curl https://developer.apple.com/certificationauthority/AppleWWDRCA.cer > /tmp/AppleWWDRCA.cer
 3curl https://www.apple.com/certificateauthority/AppleWWDRCAG3.cer       > /tmp/AppleWWDRCAG3.cer
 4# create a new keychain
 5security create-keychain -p "${keychainpass}" build.keychain
 6security default-keychain -s build.keychain
 7security unlock-keychain -p "${keychainpass}" build.keychain
 8# import the intermediate certificates into the keychain
 9security import /tmp/AppleWWDRCA.cer -k build.keychain
10security import /tmp/AppleWWDRCAG3.cer -k build.keychain

The certificate bundle we created before must be available on the machine used for signing, in the following script I use the ${PFXFILE} (and the associated ${PFXPASSWORD}) in the same way as above.

The following adds the certificate bundle to the new keychain, and allows the /usr/bin/codesign binary to access it.

1security import "${PFXFILE}" -k build.keychain -P "${PFXPASSWORD}" -T /usr/bin/codesign
2security set-key-partition-list -S apple-tool:,apple:,codesign: -s -k "${keychainpass}" build.keychain

Importing in the CI

The above snippets can be run almost directly on the CI. The keychain password (${keychainpass}) need not be secure, as it is only used within a single job. I typically set it to ${CI_JOB_TOKEN} (which contains another password-like token, automatically generated by GitLab-CI for each job), and fallback to $(date +%s)

Instead of the variables $PFXFILE and $PFXPASSWORD I use ${MACOS_CERTIFICATE_PFX} resp ${MACOS_CERTIFICATE_PWD}.

These variables are set via the CI/CD Project Settings (e.g. https://gitlab.example.com/GROUP/PROJECT/-/settings/ci_cd):

The password in MACOS_CERTIFICATE_PWD is masked (in the project settings), which makes GitLab-CI protect it from accidentally printout.

The MACOS_CERTIFICATE_PFX variable holds the base64-encoded certificate bundle, as generated with this:

1cat "${PFXFILE}" | base64

This gives nice copyable content which can be pasted into the web-interface. Due to the size, I'm using a File variable. Before using it, we have to decode it into binary form:

1cat "${MACOS_CERTIFICATE_PFX}" | base64 -D >/tmp/sign.pfx

Unfortunately, I haven't found a way to mask the File variable (either because it is multi-line, or due to its size).

Signing software

For signing binaries, we invoke codesign and pass it the file. We also have to provide an identity that identifies our certificate. The safest way to specify the identity is via a fingerprint that can be obtained with security find-identity -v. This will output all known identities (and more than just the fingerprint), so we have to massage the output a bit .

1sign_id=$(security find-identity -v | head -1 | awk '{print $2}')
2find . -type f -exec codesign --verbose --force --sign "${sign_id}" --timestamp --strict {} +

The exact invocation of codesign depends on the artifact you want to sign. E.g. you might need to add --entitlements EEE (to give specific permissions to you application).

The --timestamp flag is generally needed if you also want to notarize (as it is required for Developer ID signed code). For "main executables" you will need to enable the hardened runtime by also passing --options runtime.

Signing in the CI

It's really quite straightforward.

The only issue I find is, that it's not really possible to find an invocation of codesign for generic pipelines that are supposed to run on multiple projects producing different things (Applications, libraries, ...).

Notarizing software

To notarize software, you upload it to an Apple cloud service, which does some checks on the binaries, and if all goes well issues you a "notarization ticket".

This ticket (which is signed by Apple) can then be embedded with your deliverable. If the users attempt to run the binaries, the macOS gatekeeper verifies the ticket and allows the software to run (or not). This verification only needs to be performed once on each macOS system. If you do not embed the ticket, the gatekeeper will perform an online verification (using hashes of the binaries), so in this case your users need to be online when they first run the software.

Binaries submitted for notarization must be collected in a container. Such a container can either be an Installer-Package (.pkg), a disk image (.dmg) or simply a zip-file (${ZIPFILE}) containing all the signed files you want to notarize.

Your software needs to be associated with a "bundle ID" (${BUNDLE_ID} that helps identifying it. In theory the "bundle ID" is rather arbitrary (with some restrictions on which characters are allowed), but it should be unique. The common scheme is to base it on a domain you control and reverse that (similar to JAVA package names), e.g. com.example.myapp.

Finally, for the actual upload, you must authorize with your Apple ID (${APPLE_ID}) and the corresponding password token (${APPLE_PASS}).

1xcrun altool --notarize-app \
2      --primary-bundle-id "${BUNDLE_ID}" \
3      --username "${APPLE_ID}" --password "${APPLE_PASS}" \
4      --file "${ZIPFILE}" \
5      --verbose --output-format xml

This will return as soon as the upload succeeded. The notarization itself takes a while (typically a few minutes).

Once the process is complete, you get an email notification from Apple telling you that notarization was successful (or not). This notification contains the ${BUNDLE_ID}, so you know which software this is about (in case you notarize multiple software packages in parallel).

Embedding the notarization

As soon as the notarization is completed successfully, you can embed the notarization into the container if the container supports it (.pkg and .dmg).

This is as simple as running

1xcrun stapler staple "${DMGFILE}"

Unfortunately you cannot staple into standalone binaries or ZIP-files.

Notarizing in the CI

Submitting a package for notarization is of course trivial. Use CI-Variables for your ${APPLE_ID} resp. your ${APPLE_PASS} (and be sure to mask the values, esp. the password token).

I use the repository URL, to generate a default bundle ID (e.g. turning https://gitlab.example.com/GROUP/PROJECT/ into com.example.gitlab.GROUP.PROJECT):

1BUNDLE_ID=$(echo $(echo ${CI_SERVER_HOST} | tr '.' $'\n' | (tac 2>/dev/null || tail -r) | paste -s -d '.' -).${CI_PROJECT_NAMESPACE}.${CI_PROJECT_NAME} | sed -e 's/[^a-zA-Z0-9-]/./g' | tr 'A-Z' 'a-z')

The biggest challenge is stapling the notarization ticket in the container, as we have to wait until the notarization succeeds.

For this we have to poll the notarization server, and check the status. Each notarization request is assigned a "RequestUUID", which is returned when we submit the package. The --output-format xml flag we passed to xcrun altool will output the information in a parsable format to stdout, so we only need to redirect the stdout to a file e.g. notarize_request.plist, which we can then parse (note that defaults read requires an absolute path to read from a file):

1requestuuid=$(defaults read $(pwd)/notarize_request.plist | grep RequestUUID | sed -e 's/[^"]*"//' -e 's/".*//')

With this UUID we can query the current status of the notarization:

1xcrun altool --notarization-info ${requestuuid} \
2      --username "${APPLE_ID}" --password "${APPLE_PASS}" \
3      --verbose --output-format xml \
4| tee notarization-info.plist

Querying the notarization info, will return a =Status=, which can be (among other things):

So we only need to repeatedly query the status, until we get a definitive code (or hit a timeout).

Once the notarization process has finished (either successful or not), the notarization-info will also contain a link to a log file (formatted as JSON), that contains some details:

1curl $(defaults read $(pwd)/notarization-info.plist notarization-info | egrep '^ *LogFileURL *=' | sed -e 's|.*"\(.*\)";|\1|')

GitLab-CI configuration

Putting it all together for the iem-ci

CI-Variables

There are a couple of required variables that must be set for the pipelines. They contain sensitive information and must not be stored in the repository itself. Instead store them as (masked) variables in the project settings.

 1variables:
 2  MACOS_CERTIFICATE_PFX:
 3    value: ""
 4    description: "PKCS#12 certificate bundle with a Developer certificate"
 5  MACOS_CERTIFICATE_PWD:
 6    value: ""
 7    description: "the password for the MACOS_CERTIFICATE_PFX certificate"
 8  APPLE_ID:
 9    value: ""
10    description: "the Apple-ID for submitting the notarization request"
11  APPLE_PWD:
12    value: ""
13    description: "the password token for the APPLE_IP"

There are also some optional variables:

 1variables:
 2  CODESIGNFLAGS:
 3    value: "--timestamp --strict --force"
 4    description: "base flags for signing"
 5  APP:
 6    value: ""
 7    description: "Application to sign/notarize (defaults to Pd*.app)"
 8  PKG:
 9    value: ""
10    description: "Container into which to wrap the Application (defaults to ${APP}.dmg); must be .zip or .dmg"
11  BUNDLE_ID:
12    value: ""
13    description: "Bundle-ID for the notarization request (default: calculated from the repository URL)"
14  NOTARIZE_TIMEOUT:
15    value: ""
16    description: "maximum time (in seconds) to wait for the notarization to finish"

Setting up the keychain

 1.script:keychain:macos:
 2  script: &script_keychain_macos
 3    # get certificate chain
 4    - curl https://developer.apple.com/certificationauthority/AppleWWDRCA.cer >/tmp/AppleWWDRCA.cer
 5    - curl https://www.apple.com/certificateauthority/AppleWWDRCAG3.cer       >/tmp/AppleWWDRCAG3.cer
 6    - if test -e "${MACOS_CERTIFICATE_PFX}" && test -n "${MACOS_CERTIFICATE_PWD}"; then cat "${MACOS_CERTIFICATE_PFX}" | base64 -D >/tmp/sign.pfx; fi
 7    - test -e /tmp/sign.pfx || exit 77
 8    - shasum /tmp/sign.pfx
 9    - keychainpass=${keychainpass:-${CI_BUILD_TOKEN:-secret$(date +%s)}}
10    - security create-keychain -p "${keychainpass}" build.keychain
11    - security default-keychain -s build.keychain
12    - security unlock-keychain -p "${keychainpass}" build.keychain
13    - security import /tmp/AppleWWDRCA.cer   -k build.keychain
14    - security import /tmp/AppleWWDRCAG3.cer -k build.keychain
15    - security import /tmp/sign.pfx -k build.keychain -P "${MACOS_CERTIFICATE_PWD}" -T /usr/bin/codesign
16    - security set-key-partition-list -S "apple-tool:,apple:,codesign:" -s -k "${keychainpass}" build.keychain >/dev/null
17    - security find-identity -v
18    - sign_id=$(security find-identity -v | head -1 | awk '{print $2}')

Signing

 1.script:codesign:macos:
 2  script: &script_codesign_macos
 3    - APP=${APP:-$(ls -d Pd*.app | tail)}
 4    - echo "CODESIGNFLAGS ${CODESIGNFLAGS}"
 5    - codesignflags=${CODESIGNFLAGS:---timestamp --force --entitlements mac/stuff/pd.entitlements}
 6    - codesignflags="${codesignflags} --sign ${sign_id}"
 7    - echo "codesignflags ${codesignflags}"
 8    - codesign ${codesignflags} ${APP}/Contents/Frameworks/Tcl.framework/Versions/Current
 9    - codesign ${codesignflags} ${APP}/Contents/Frameworks/Tk.framework/Versions/Current
10    - find ${APP}/Contents/Resources/bin -type f                                        -exec codesign ${codesignflags} --strict --options runtime {} ";"
11    - find ${APP}/Contents/Resources     -type f "(" -name "*.d_*" -o -name "*.pd_" ")" -exec codesign ${codesignflags} --strict {} ";"
12    - codesign ${codesignflags} --deep --strict --options=runtime ${APP}

Notarizing

 1.script:notarize:macos:
 2  script: &script_notarize_macos
 3    # setup some vars
 4    - APP=${APP:-$(ls -d Pd*.app | tail)}
 5    - APP=${APP%/}
 6    - PKG=${PKG:-${APP%.app}.dmg}
 7    - IEM_CI_PROJECT_NAME=${IEM_CI_PROJECT_NAME:-${APP%.app}}
 8    - BUNDLE_ID=${BUNDLE_ID:-$(echo $(echo ${CI_SERVER_HOST} | tr '.' $'\n' | (tac 2>/dev/null || tail -r) | paste -s -d '.' -).${CI_PROJECT_NAMESPACE}.${CI_PROJECT_NAME} | sed -e 's/[^a-zA-Z0-9-]/./g' | tr 'A-Z' 'a-z')}
 9    # try to switch to the latest and greatest XCode, so we get 'altool'
10    - sudo xcode-select --switch $(for d in /Applications/Xcode*.app/Contents/Info.plist ; do echo $(defaults read ${d%.plist} CFBundleShortVersionString) ${d%/Contents/Info.plist}; done | sort -t. -k1,1n -k2,2n -k3,3n | tail -1 | sed -e 's/^[^ ]* //' ) || true
11    # check if altool exists
12    - xcrun --find altool
13    # stuff everything into a disk image
14    - test "${PKG}" = "${PKG%.dmg}" || hdiutil create -volname "${IEM_CI_PROJECT_NAME}" -format UDZO -srcfolder "${APP}" "${PKG}"
15    # or a ZIP-file , if you really want
16    - test "${PKG}" = "${PKG%.zip}" || zip -r -y "${PKG}" "${IEM_CI_PKGLIBDIR:-${IEM_CI_PROJECT_NAME}}"
17    # and upload to apple...
18    - test -z "${APPLE_ID}" || test -z "${APPLE_PWD}" || (xcrun altool --notarize-app --primary-bundle-id "${BUNDLE_ID}" --username "${APPLE_ID}" --password "${APPLE_PWD}" --file "${PKG}" --verbose --output-format xml > notarize.plist && defaults read $(pwd)/notarize.plist notarization-upload)
19    # read back the UUID of the notarization request
20    - test -z "${APPLE_ID}" || test -z "${APPLE_PWD}" || notarize_uuid=$(defaults read $(pwd)/notarize.plist notarization-upload | grep RequestUUID | sed -e 's|.*"\(.*\)";|\1|')

Waiting for the notarization to finish

 1.script:notarizewait:macos:
 2  script: &script_notarizewait_macos
 3    # setup some vars
 4    - NOTARIZE_TIMEOUT=${NOTARIZE_TIMEOUT:-300}
 5    - end=0
 6    - logfile=""
 7    - test -z "${notarize_uuid}" || test 0 -ge ${NOTARIZE_TIMEOUT} || end=$(($(date +%s) + ${NOTARIZE_TIMEOUT}))
 8    # wait until either
 9    # - the current date exceeds the timeout
10    # - the Status is no longer 'in progress'
11    - |
12      while [ ${end} -gt $(date +%s) ]; do
13        sleep 10;
14        date;
15        xcrun altool -u "${APPLE_ID}" -p "${APPLE_PWD}" --output-format xml --notarization-info "${notarize_uuid}" > notarization-info.plist;
16        defaults read $(pwd)/notarization-info.plist notarization-info | tee /dev/stderr | egrep "Status.*in progress" >/dev/null && continue || break;
17      done      
18    # check whether there's a logfile to report
19    - test ! -e notarization-info.plist || logfile=$(defaults read $(pwd)/notarization-info.plist notarization-info | egrep '^ *LogFileURL *=' | sed -e 's|.*"\(.*\)";|\1|')
20    - test -z "${logfile}" || curl "${logfile}" | tee notarization.log.json

The final job

 1.notarize:macos:
 2  tags:
 3     - osx
 4  stage: deploy
 5  script:
 6  - *script_keychain_macos
 7  - *script_codesign_macos
 8  - *script_notarize_macos
 9  - *script_notarizewait_macos
10  - xcrun stapler staple ${APP} || true
11  - mkdir -p "${APP}/Contents/_CodeSignature/"
12  - cp notari* "${APP}/Contents/_CodeSignature/"

Further reading

• How to automatically sign macOS apps using GitHub Actions
• Signing a Mac Product For Distribution