EV Code Signing in Azure Pipelines

2020-10-29 Azure, Azure Key Vault, continuous integration, devops, code signing, Nexus

For basic code-signing on Azure, their secret files is very easy to use and described in many places. Unfortunately, when you want to sign with EV certificates, secret files won’t work and one must use the Azure Key Vault. This makes the process much more complicated. In this blog post, meant mainly as a memory dump to which I can return later when I forget everything, I will describe how we finally managed to do this.

Why EV-certificates?

I am working on a WebAuthN soft-authenticator (backed by smart cards). The authenticator is split into two parts — a part that does the bulk of the work and a driver part, which creates a virtual HID device to communicate with the rest of the system. On windows, the driver part is built using the virtual hid device framework. Unfortunately this framework is only supported for kernel mode drivers. To be able to install these, they must be submitted to Microsoft for signing (how to achieve this is a long, sad and frustrating story, a story for another day) and Microsoft requires the submissions to be signed by EV-certificates. So that is why we need EV-certificates.

Why Azure Key Vault?

EV-certificates come with higher security guarantees than normal certificates. First, the CA issuing them does more thorough checks to ensure that you are really who you claim to be when you ask for a certificate. Second, you don’t actually get the certificate as a simple p12 file. The requirements for EV-certificates state that they must be stored securely — in particular, you aren’t supposed actually be able to access the private key. Instead what you typically get is a secure USB token which stores the private key and you can use it to sign code, but there is no way for you to extract the key to, e.g., store it in azure secrets . And a hardware usb token is useless for azure pipelines. Luckily, there is another option for getting an EV-certificates — Azure Key Vault. Azure Key Vault can function as a sort of secure token which has all the certifications which allow it to be used for EV-certificate storage. And we can ask the CA to have our certificate delivered into AKV instead of on a hardware token. Well, not quite (at the moment anyway). What you need to do instead is to generate the certificate in AKV, generate a CSR (certificate signing request), ask the CA to sign this CSR, and then upload the signed CSR back to AKV.

Obtaining an EV-certificate stored in Azure Key Vault

This can be done from the web interface, but for automation purposes, I’ll give the instructions using the azure cli command line client for azure. First you need to create a policy file describing the certificate you want to create:

{
  "issuerParameters": {
    "certificateTransparency": null,
    "certificateType": "DigiCert",
    "name": "Unknown"
  },
  "keyProperties": {
    "curve": null,
    "exportable": false,
    "keySize": 2048,
    "keyType": "RSA",
    "reuseKey": true
  },
  "lifetimeActions": [
    {
      "action": {
        "actionType": "emailContacts"
      },
      "trigger": {
        "daysBeforeExpiry": 90,
        "lifetimePercentage": null
      }
    }
  ],
  "secretProperties": {
    "contentType": "application/x-pkcs12"
  },
  "x509CertificateProperties": {
    "ekus": [
      "1.3.6.1.5.5.7.3.3"
    ],
    "keyUsage": [
      "digitalSignature",
      "keyCertSign"
    ],
    "subject": "C=SE, L=Hägersten-Liljeholmen, O=Technology Nexus Secured Business Solutions AB, CN=Technology Nexus Secured Business Solutions AB",
    "subjectAlternativeNames": {
      "emails": [
        "support@nexusgroup.com"
      ]
    },
    "validityInMonths": 24
  }
}

The keyProperties key defines the parameters of your RSA keypair and how its stored. If you signed up for a HSM-backed azure key vault, you can set keytype to RSA-HSM for even more security. The lifetimeActions key instructs AKV to send you an email three months before the certificate will expire. The x509CertificateProperties contains en extended key usage oid 1.3.6.1.5.5.7.3.3 indicating that the certificate can be used for code signing. Once you have prepared this file and stored it in, say cert-policy.json you can use the following command to generate a new certificate in azure key vault:

$ az keyvault certificate create --vault-name $KEY_VAULT_NAME -p @cer
t-policy.json --name $CERT_NAME

replacing $KEY_VAULT_NAME and $CERT_NAME as appropriate. This will create the certificate in the key vault, but the certificate will be in a disabled state. To enable the certificate, you need to generate and download a CSR, have it signed by a CA, and then upload the signed CSR back to the key vault. This is done as follows:

$ az keyvault certificate pending show --name $CERT_NAME --vault-name $KEY_VAULT_NAME | jq -r .csr | sed -e'1,1s/^/-----BEGIN CERTIFICATE REQUEST-----/g' | sed -e'$,$s/$/-----END CERTIFICATE REQUEST-----/g'

The first command in the pipeline just downloads info in JSON-format about the newly created certificate. The second element in the pipeline uses the jq tool to extract the csr key, which holds the certificate signing request, from the JSON. The last element wraps this in a -----BEGIN/END CERTIFICATE REQUEST----- block so that it is usable by, e.g., openssl.

Now you need to get your CA to sign this CSR. How that is done will depend in your CA and is not described here. For testing purposes you can use a dummy CA as described, e.g., in the pki tutorial or in the How to setup your own CA with OpenSSL gist.

Once you have your CSR signed by a CA, stored in, say, signed-csr.crt which will look something like this:

Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number: 4660 (0x1234)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=SE, ST=Some-State, L=H\xC3\x83\xC2\xA4gersten-Liljeholmen, O=Technology Nexus Secured Business Solutions AB, OU=Prague Office, CN=Technology Nexus Secured Business Solutions AB/emailAddress=jonathan.verner@nexusgroup.com
        Validity
            Not Before: Oct 26 13:51:33 2020 GMT
            Not After : Oct 26 13:51:33 2021 GMT
        Subject: C=SE, L=H\xC3\xA4gersten-Liljeholmen, O=Technology Nexus Secured Business Solutions AB, CN=Technology Nexus Secured Business Solutions AB
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:d3:b2:6c:2e:83:8f:c2:bf:b9:52:46:d5:b5:26:
                    73:d3:ab:c9:88:16:7a:8a:9a:8c:6b:02:5c:ee:16:
                    91:3a:f9:f7:57:ef:c1:dd:0a:1b:4d:27:5f:2a:32:
                    d5:88:c1:ae:66:fc:8a:07:38:2e:5a:62:48:15:02:
                    94:3c:59:6a:a7:7e:a7:0d:ad:18:cf:65:5b:15:1b:
                    4d:81:4a:71:1d:b5:f6:8c:a5:45:9e:c3:5b:2b:9d:
                    f5:82:80:55:5c:0e:19:4f:14:4e:f8:27:f4:54:ef:
                    08:3e:66:56:9b:95:79:c0:09:10:d9:04:d3:e2:30:
                    bb:42:e5:47:5e:b3:19:64:dc:0e:c0:19:a5:64:e0:
                    0d:8b:55:d7:51:53:6b:e2:57:1c:a5:1a:72:fe:8d:
                    f5:25:30:de:80:1b:63:c1:b4:df:2f:31:f7:e0:eb:
                    a2:07:1f:c3:55:f8:12:58:ce:5e:f9:21:7c:89:3f:
                    95:92:d1:e3:d4:96:79:05:bc:31:1d:2b:da:88:15:
                    37:4c:e8:b6:f4:71:ac:6e:78:14:c4:1b:75:3f:ed:
                    b4:27:d9:59:02:bf:c7:65:25:bc:08:eb:78:bf:94:
                    54:76:13:e5:18:77:79:80:f6:3a:a3:d2:06:c1:08:
                    fe:5f:fd:d8:b8:cc:d5:58:b1:a1:e2:14:91:ec:cb:
                    6a:c9
                Exponent: 65537 (0x10001)
    Signature Algorithm: sha256WithRSAEncryption
         98:1b:93:2c:68:69:a1:88:39:70:81:c4:79:55:94:72:f9:3e:
         45:b3:53:d4:69:b0:77:6e:28:ae:b9:ae:00:90:eb:a5:87:8b:
         ea:d8:f3:f5:7b:17:23:a1:d9:b2:fe:a8:ad:27:b7:cd:b0:92:
         33:68:6c:80:cd:06:fe:d5:93:4e:3c:0b:39:92:54:c6:ce:6a:
         77:be:64:06:4a:f6:21:db:b9:96:05:7c:85:16:a6:6b:d3:26:
         63:c6:28:2e:9b:43:20:ce:63:09:c8:bc:ac:09:93:c4:6d:b7:
         dc:27:0f:ef:67:eb:7f:cd:59:5f:9d:53:2c:8f:36:f2:70:1e:
         81:e8:29:c6:27:71:16:e3:6b:db:d6:c0:45:dd:4b:e7:b4:2e:
         c8:cb:f2:87:a8:a3:20:4e:3f:22:60:ba:de:80:e8:3d:08:1b:
         18:88:e8:bb:c7:33:42:8f:6d:57:5e:ef:9d:71:52:c2:70:c6:
         5f:c6:4c:91:68:52:09:c2:94:81:a2:4f:97:dd:a7:89:a8:09:
         1b:d3:e8:95:6b:c1:43:cd:f1:fc:b3:ef:a3:89:d8:c7:09:b6:
         8d:94:d2:da:cb:15:58:d7:af:dc:9e:e9:17:37:e9:c5:2f:5a:
         17:a5:64:50:8a:74:84:f1:1d:14:f1:ab:51:f2:05:94:67:61:
         61:64:e7:c4
-----BEGIN CERTIFICATE-----
MIIEFTCCAv0CAhI0MA0GCSqGSIb3DQEBCwUAMIH+MQswCQYDVQQGEwJTRTETMBEG
A1UECAwKU29tZS1TdGF0ZTEhMB8GA1UEBwwYSMODwqRnZXJzdGVuLUxpbGplaG9s
bWVuMTcwNQYDVQQKDC5UZWNobm9sb2d5IE5leHVzIFNlY3VyZWQgQnVzaW5lc3Mg
U29sdXRpb25zIEFCMRYwFAYDVQQLDA1QcmFndWUgT2ZmaWNlMTcwNQYDVQQDDC5U
ZWNobm9sb2d5IE5leHVzIFNlY3VyZWQgQnVzaW5lc3MgU29sdXRpb25zIEFCMS0w
KwYJKoZIhvcNAQkBFh5qb25hdGhhbi52ZXJuZXJAbmV4dXNncm91cC5jb20wHhcN
MjAxMDI2MTM1MTMzWhcNMjExMDI2MTM1MTMzWjCBoDELMAkGA1UEBhMCU0UxHzAd
BgNVBAcMFkjDpGdlcnN0ZW4tTGlsamVob2xtZW4xNzA1BgNVBAoTLlRlY2hub2xv
Z3kgTmV4dXMgU2VjdXJlZCBCdXNpbmVzcyBTb2x1dGlvbnMgQUIxNzA1BgNVBAMT
LlRlY2hub2xvZ3kgTmV4dXMgU2VjdXJlZCBCdXNpbmVzcyBTb2x1dGlvbnMgQUIw
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDTsmwug4/Cv7lSRtW1JnPT
q8mIFnqKmoxrAlzuFpE6+fdX78HdChtNJ18qMtWIwa5m/IoHOC5aYkgVApQ8WWqn
fqcNrRjPZVsVG02BSnEdtfaMpUWew1srnfWCgFVcDhlPFE74J/RU7wg+ZlablXnA
CRDZBNPiMLtC5Udesxlk3A7AGaVk4A2LVddRU2viVxylGnL+jfUlMN6AG2PBtN8v
Mffg66IHH8NV+BJYzl75IXyJP5WS0ePUlnkFvDEdK9qIFTdM6Lb0caxueBTEG3U/
7bQn2VkCv8dlJbwI63i/lFR2E+UYd3mA9jqj0gbBCP5f/di4zNVYsaHiFJHsy2rJ
AgMBAAEwDQYJKoZIhvcNAQELBQADggEBAJgbkyxoaaGIOXCBxHlVlHL5PkWzU9Rp
sHduKK65rgCQ66WHi+rY8/V7FyOh2bL+qK0nt82wkjNobIDNBv7Vk048CzmSVMbO
ane+ZAZK9iHbuZYFfIUWpmvTJmPGKC6bQyDOYwnIvKwJk8Rtt9wnD+9n63/NWV+d
UyyPNvJwHoHoKcYncRbja9vWwEXdS+e0LsjL8oeooyBOPyJgut6A6D0IGxiI6LvH
M0KPbVde751xUsJwxl/GTJFoUgnClIGiT5fdp4moCRvT6JVrwUPN8fyz76OJ2McJ
to2U0trLFVjXr9ye6Rc36cUvWhelZFCKdITxHRTxq1HyBZRnYWFk58Q=
-----END CERTIFICATE-----

you need to upload it back to azure:

az keyvault certificate pending merge --name $CERT_NAME --vault-name $KEY_VAULT_NAME -f signed-csr.crt

The certificate should now be ready to be used. Now comes the fun part: actually using the certificate to sign software.

Signing with a Key Vault certificate in azure pipelines

Signing software on Windows is usually done using SignTool.exe which comes with Visual Studio (there is also an open source alternative, osslcigncode which can be used on other platforms). However, this tool only works with local certificates. Luckily, vcjones did Microsofts homework for them and created a tool called AzureSignTool to fix exactly this problem. This tool can be installed by running the following command:

dotnet tool install --global azuresigntool

Unfortunately, at the time this blog post was written, this installs a basically unusable version which has abysmal error handling — running it produces no output what-so-ever and if an error occurs, you’re not told and are on your own. So you need to build the tool from source.

Another hurdle is that the tool needs to authenticate to the Key Vault. The easiest way to do this is to create a service connection for your pipeline and use managed identity. Unfortunately, the –azure-key-vault-managed-identity command line option to AzureSignTool doesn’t seem to work and one needs to use an access token. The following yaml snippet and powershell script shows how to do this:

- task: AzureCLI@2
      displayName: Sign Artifacts
      inputs:
        azureSubscription: SERVICE_CONNECTION_NAME
        scriptType: ps
        scriptPath: sign.ps1

The key here is to use the AzureCLI task, which logs you in and allows you to get the access token. The work is done by the sign.ps1 script:

# Set up some env variables (certificate name, keyvault url, signed file description, timestamp server)
$CERTIFICATE_NAME="..."
$KEYVAULT_URL="https://KEY_VAULT_NAME.vault.azure.net"
$DESCRIPTION="File content description"
$TIMESTAMP_SERVER="http://timestamp.verisign.com/scripts/timstamp.dll"

# Get an access token (note that by running through the AzureCLI, we are logged in to azure and
# can actually get the token)
$ACCESS_TOKEN=(az account get-access-token --resource https://vault.azure.net | jq -r .accessToken)

# Do the actual signing
AzureSignTool.exe sign --verbose -kva $ACCESS_TOKEN -d "$DESCRIPTION" -kvc "$CERTIFICATE_NAME" -kvu "$KEYVAULT_URL" -fd sha256 -t "$TIMESTAMP_SERVER" file-to-sign.exe