Azure API Management Self-Hosted Gateway - Token Cycling

The self-hosted gateway is a feature of the API Management to support the management of hybrid and multi-cloud environments. More clearly, this feature enables organizations to manage their APIs hosted either on Azure cloud, on-premises, or on any other cloud, from a single API Management service in Azure. This is done through deploying a containerized version of the API Management Gateway to the same environment where the APIs are hosted. This containerized version of the gateway can be deployed in Docker, or in a Kubernetes cluster.

Connectivity to Azure:

The Self-Hosted Gateway requires a connection to Azure in order to:

Report its status through sending a heartbeat every minute.
Check regularly for configuration updates and apply them.

However, in order for the self-hosted gateway to be able to download the configuration data from the API Management endpoint; it needs a valid access token.

Token Cycling / Token Rotation:

The access token given for the self-hosted gateway is only valid for 30 days. Therefore, this token needs to be regenerated and assigned to the self-hosted gateway either manually or through automation before it expires in order for the self-hosted gateway to keep accessing and getting the configuration updates.

In order to automatically rotate API management self-hosted gateway tokens, we can have Kubernetes CronJob that runs on a regular basis and regenerate the token and assign it to the self-hosted gateway.

The following section describes a project called “apim-shg-rotate” which is a Docker container image that can be deployed in a Kubernetes CronJob to automatically rotate API management self-hosted gateway tokens regularly. ]1[

1. Requirements:

A Kubernetes cluster, and a self-hosted gateway into that cluster.
A service principal with one of the following roles:
- API Management Service Contributor (built-in role). This role gives access to far more than what we actually need here.
- API Management Self-Hosted Gateway Token Operator (custom role). This role gives only two permissions, to list the self-hosted gateways and to generate new tokens. ]2[ There are specific steps that should be followed in order to create a custom role.

2. Deployment in the cluster:

The script should be deployed in the same namespace as the API management self-hosted gateway.

The following is required for the deployment:

Kubernetes service principal secret, client ID, client secret, and tenant ID.
RBAC which is a .yaml file for Role, Role Binding, and Service Account.
CronJob

]1[ https://github.com/phealy/apim-shg-rotate
]2[ https://docs.microsoft.com/en-us/azure/role-based-access-control/custom-roles

The following steps describe in detail the deployment process:

Create the following “updateToken.sh” script file.
PS: Double click the below text snippet to open the full script file.

#! /bin/bash
set -eo pipefail

# updateToken.sh
#
# A shell script to use the Azure CLI and kubectl in order to rotate a token on a
# self-hosted application gateway. Designed to be run in a container using a Kubernetes
# CronJob.
#
# Based on an idea from https://github.com/phealy/apim-shg-rotate
# but rewritten.

function usage {
cat <<-EOF

Generates a new token for an APIM self-hosted gateway, updates the Kubernetes secret, then performs a rolling restart.
Arguments may be specified as parameters or as environment variables.

Arguments:

-s, –subscription-id, SUBSCRIPTION_ID environment variable
The GUID of the scription containing the APIM instance.

-r, –resource-group, RESOURCE_GROUP environment variable
The name of the resource group containing the APIM instance.

-a, –apim-instance, APIM_INSTANCE environment variable
The name of the APIM instance

-g, –apim-gateway, APIM_GATEWAY environment variable
The name of the self-hosted gateway

-n, –namespace, NAMESPACE environment variable
The namespace containing the token to rotate

-t, –token-secret, TOKEN_SECRET environment variable
The name of the Kubernetes secret object containing the token to rotate

-k, –token-key, TOKEN_KEY
Switch the key used to generate the token. Can be set to one of the following values:
„last“ – default, the value from the last-key-used annotation on the token will be used

if no annotation is present, defaults to primary
„rotate“ – rotate from the last used key to the other key: primary -> secondary or secondary -> primary
„primary“ – generate the new token from the primary key
„secondary“ – generate the new token from the secondary key

-o, –k8s-object, K8S_OBJECT
The name of the Kubernetes object to restart after updating the secret, using kubectl rollout restart. Should be specified in the form „type/name“, e.g. „deployment/apim-gateway“ or „statefulset/apim-gateway“

–client-id, SP_ID
The client ID of the service principal to use, if the login method is „sp“

–client-secret, SP_SECRET
The client secret of the service principal to use, if the login method is „sp“. This can be either a secret or the path to a certificate.

–tenant, TENANT
The tenant ID of the service principal to use, if the login method is „sp“.

–debug
Enable debug logging (set -x)

EOF
exit 1
}

# For SP login, we make a temporary directory. Clean it up on exit.
function cleanup {
[[ ! -z „${MY_AZURE_CONFIG_DIR}“ ]] && {
rm -rf „${MY_AZURE_CONFIG_DIR}“
}
}
trap cleanup EXIT

PARSED_ARGUMENTS=$(getopt -a -n „$(basename $0)“ -o s:r:a:g:n:t:k:o:l:h –long subscription-id:,resource-group:,apim-instance:,apim-gateway:,namespace:,token-secret:,token-key:,k8s-object:,client-id:,client-secret:,tenant:,debug,help — „$@“)
VALID_ARGUMENTS=$?
if [ „$VALID_ARGUMENTS“ != „0“ ]; then
usage
fi

eval set — „$PARSED_ARGUMENTS“
while :
do

case „$1“ in
–debug) set -x; shift;;
-s | –subscription-id) SUBSCRIPTION_ID=“${2}“; shift 2;;
-r | –resource-group) RESOURCE_GROUP=“${2}“; shift 2;;
-a | –apim-instance) APIM_INSTANCE=“${2}“; shift 2;;
-g | –apim-gateway) APIM_GATEWAY=“${2}“; shift 2;;
-n | –namespace) NAMESPACE=“${2}“; shift 2;;
-t | –token-secret) TOKEN_SECRET=“${2}“; shift 2;;
-k | –token-key) TOKEN_KEY=“${2}“; shift 2;;
-o | –k8s-object) K8S_OBJECT=“${2}“; shift 2;;
–client-id) SP_ID=“${2}“; shift 2;;
–client-secret) SP_SECRET=“${2}“; shift 2;;
–tenant) TENANT=“${2}“; shift 2;;
-h | –help) usage;;
–) shift; break ;;
*) echo „ERROR: didn’t parse an argument properly: ${1} ${2}“; usage;;
esac
done

# Set defaults
TOKEN_KEY=“${TOKEN_KEY:-last}“
MASKED_SP_SECRET=“${SP_SECRET//?/*}“

# Error out if variables aren’t set
for var in SUBSCRIPTION_ID RESOURCE_GROUP APIM_INSTANCE APIM_GATEWAY NAMESPACE TOKEN_SECRET; do
if [[ -z „${!var}“ ]]; then
FAIL=1
echo -e „ERROR: Parameter ${var} is undefined. Please specify the parameter as either an environment variable or a command argument.“
fi
done
if [[ „${FAIL:-0}“ == „1“ ]]; then
echo -e „\nRun $(basename $0) –help for usage information.“
exit 1
fi

cat << EOF

APIM Self-hosted Gateway Secret Rotation
Date: $(date)
—————————————-

Parameters:
Subscription ID: $SUBSCRIPTION_ID
Resource group:   $RESOURCE_GROUP
APIM Instance:    $APIM_INSTANCE
APIM Gateway:     $APIM_GATEWAY
K8s Namespace:    $NAMESPACE
K8s Token Secret: $TOKEN_SECRET
APIM Key Source: $TOKEN_KEY
K8S Object:       ${K8S_OBJECT:-not provided}
Client ID:        ${SP_ID:-not provided}
Client Secret:    ${MASKED_SP_SECRET:-not provided}
Client Tenant:    ${TENANT:-not provided}

—————————————-

EOF

for var in SP_ID SP_SECRET TENANT; do
if [[ -z „${!var}“ ]]; then
FAIL=1
echo -e „ERROR: When using service principal login type, ${var} must be defined!“
fi
done
if [[ „${FAIL:-0}“ == „1“ ]]; then
echo -e „\nRun $(basename $0) –help for usage information.“
exit 1
fi
MY_AZURE_CONFIG_DIR=$(mktemp -d)
AZURE_CONFIG_DIR=“${MY_AZURE_CONFIG_DIR}“
echo -n „Checking for service principal access…“
# Try a service principal
az login –service-principal -u „$SP_ID“ -p „$SP_SECRET“ –tenant $TENANT >/dev/null
if [[ $(az account list –refresh –query „length([?id==’$SUBSCRIPTION_ID‘])“ 2>/dev/null) == 1 ]]; then
echo „done (logged in via service principal).“
else
echo -e „\n\nERROR: Failed to access the subscription via az cli, either via already logged in credentials or identity.“
exit 1
fi

echo -n „Validating APIM instance is present and correct…“
APIM_RESOURCE_ID=$(az apim show –subscription $SUBSCRIPTION_ID –resource-group $RESOURCE_GROUP –name $APIM_INSTANCE –query „id“ –only-show-errors -o tsv 2>&1) || {
echo -e „\n\nERROR: Unable to find $APIM_INSTANCE in resource group $RESOURCE_GROUP in subscription $SUBSCRIPTION_ID.“
echo -e „\nCommand output:\n${APIM_RESOURCE_ID}“
exit 1
}
echo „done.“

echo -n „Validating APIM gateway instance is present and correct…“
GATEWAY_RESOURCE_ID=“${APIM_RESOURCE_ID}/gateways/${APIM_GATEWAY}“
OUTPUT=$(az rest –method GET –uri „https://management.azure.com${GATEWAY_RESOURCE_ID}“ –uri-parameters „api-version=2021-08-01“ 2>&1) || {
echo -e „\n\nERROR: Unable to query APIM self-hosted gateway instance properties.“
echo -e „\nAPI output:\n${OUTPUT}“
exit 1
}
echo „done.“

echo -n „Validating Kubernetes secret is present and correct…“
LAST_USED_KEY_ANNOTATION=$(kubectl –namespace $NAMESPACE get secret $TOKEN_SECRET -o jsonpath='{.metadata.annotations.last-used-key}‘ 2>&1) || {
echo -e „\n\nERROR: unable to retrieve Kubernetes secret ${TOKEN_SECRET}.“
echo -e „\nkubectl output:\n${LAST_USED_KEY_ANNOTATION}“
exit 1
}
echo „done (last used key: \“${LAST_USED_KEY_ANNOTATION:-unset}\“).“
echo -n „Determining which key to use to generate the token…“
case „${TOKEN_KEY}“ in
last)
TOKEN_KEY=“${LAST_USED_KEY_ANNOTATION:-primary}“
;;
primary)
TOKEN_KEY=“primary“
;;
secondary)
TOKEN_KEY=“secondary“
;;
rotate)
case „${LAST_USED_KEY_ANNOTATION:-secondary}“ in
primary) TOKEN_KEY=“secondary“;;
secondary) TOKEN_KEY=“primary“;;
esac
;;
*)
echo -e „\n\nERROR: Unrecognized argument to -k/–token-key/TOKEN_KEY.“
usage
;;
esac
echo „done – token will be generated from the ${TOKEN_KEY} key.“

echo -n „Generating new token for gateway…“
GENERATE_TOKEN_URL=“https://management.azure.com${GATEWAY_RESOURCE_ID}/generateToken?api-version=2021-08-01″
TOKEN_EXPIRATION_DATE=“$(date -Iseconds -d“@$(($(date +%s)+1592000))“)“ # Date +30 days; have to use this format for busybox date
TOKEN=$(az rest –method POST –uri $GENERATE_TOKEN_URL –body „{ \“expiry\“: \“${TOKEN_EXPIRATION_DATE}\“, \“keyType\“: \“${TOKEN_KEY}\“ }“ –query value -o tsv) || {
echo -e „\n\nERROR: unable to generate new token for gateway.“
echo -e „\nAPI call output:\n${TOKEN}“
exit 1
}
echo „done.“

echo -n „Updating Kubernetes secret…“
OUTPUT=$(kubectl –namespace $NAMESPACE create secret generic ${TOKEN_SECRET} –from-literal value=“GatewayKey ${TOKEN}“ –dry-run=client -o yaml 2>&1 | kubectl –namespace $NAMESPACE apply -f – 2>&1) || {
echo -e „\n\nERROR: Unable to update token secret.“
echo -e „\nkubectl output:\n${OUTPUT}“
exit 1
}
echo „${OUTPUT}“

echo -n „Annotating token secret with \“last-used-key: ${TOKEN_KEY}\“…“
OUTPUT=$(kubectl –namespace $NAMESPACE annotate –overwrite=true secret $TOKEN_SECRET last-used-key=“${TOKEN_KEY}“ 2>&1) || {
echo -e „\n\nERROR: Failed to annotate token secret.“
echo -e „\nkubectl output:\n${OUTPUT}“
exit 1
}
echo „${OUTPUT}“

if [[ ! -z „${K8S_OBJECT}“ ]]; then
echo -n „Performing a rolling restart of the self-hosted gateway…“
OUTPUT=$(kubectl –namespace $NAMESPACE rollout restart $K8S_OBJECT 2>&1) || {
echo -e „\n\nERROR: Failed to restart $K8S_OBJECT.“
echo -e „\nkubectl output:\n${OUTPUT}“
exit 1
}
echo „${OUTPUT}“
fi

echo

echo „Token rotation complete.“
echo „A new token was generated based on the ${TOKEN_KEY} APIM key and will expire on ${TOKEN_EXPIRATION_DATE}.“

2. Create a Docker image.
PS: Use Dockerfile to create the image as follow:

FROM mcr.microsoft.com/azure-cli:latest
WORKDIR /root

RUN /usr/local/bin/az aks install-cli

COPY updateToken.sh /root

ENTRYPOINT „/root/updateToken.sh“

3. Push the image into a Docker Hub Repository.

4. Create the rbac.yaml file as follows:
—
apiVersion: v1
kind: ServiceAccount
metadata:
name: apim-shg-rotate
—
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: apim-shg-rotate
rules:
– apiGroups: [„*“]
resources: [„deployments“]
verbs: [„get“, „patch“]
– apiGroups: [„*“]
resources: [„secrets“]
verbs: [„get“, „update“, „patch“]
– apiGroups: [„*“]
resources: [„statefulsets“]
verbs: [„get“, „patch“]
—
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: apim-shg-rotate
subjects:
– kind: ServiceAccount
name: apim-shg-rotate
roleRef:
kind: Role
name: apim-shg-rotate
apiGroup: rbac.authorization.k8s.io
5. Create the cronjob.yaml file as follows:
PS: Adapt the variables inside this cronjob.yaml file according to your information. Variables that need to be adapted are marked with #adapt.
—
apiVersion: batch/v1
kind: CronJob
metadata:
name: apim-shg-rotate
spec:
concurrencyPolicy: Forbid
schedule: „0 9 1,15 * *“     #adapt
jobTemplate:
spec:
activeDeadlineSeconds: 600
completions: 1
parallelism: 1
template:
metadata:
labels:
aadpodidbinding: apim-shg-rotate
spec:
containers:
– name: update-token
image: qbqalex/rotate4shg:latest     #adapt
imagePullPolicy: Always
env:
– name: SUBSCRIPTION_ID
value: SUBSCRIPTION_ID_Value     #adapt
– name: RESOURCE_GROUP
value: Resource_Group_Name       #adapt
– name: APIM_INSTANCE
value: APIM_INSTANCE_Name   #adapt
– name: APIM_GATEWAY
value: APIM_GATEWAY_Name         #adapt
– name: NAMESPACE
value: APIM_SHG_NAMESPACE        #adapt
– name: TOKEN_SECRET
value: APIM_TOKEN_Secret         #adapt
– name: TOKEN_KEY
value: APIM_TOKEN_KEY            #adapt
– name: K8S_OBJECT
value: APIM_SHG_OBJECT           #adapt
– name: SP_ID
value: Service_Principal_ID      #adapt
– name: SP_SECRET
value: Service_Principal_Secret  #adapt
– name: TENANT

value: Tenant_ID #adapt

restartPolicy: Never

serviceAccountName: apim-shg-rotate

automountServiceAccountToken: true
6. Finally, apply the two .yaml files created in the previous two steps as follows:

kubectl apply -f rbac.yalm
kubectl apply -f cronjob.yalm

Now the CronJob will run on a regular basis, depending on the “schedule” chosen in the cronjob.yaml file.

For example, if the schedule was chosen as follow 0 0 */25 * * : this means that CronJob will regenerate the token at 00:00 on every 25^th day of the month, i.e., the token will be regenerated one time each month before it expires, and it will be re-assigned to the self-hosted gateway.

CONNECTING

ENDPOINTS

TECHNOLOGIES

PROCESSES

APPLICATIONS

DATA

DEVELOPERS

Azure API Management Self-Hosted Gateway – Token Cycling