Workload Identity Registration

Both the management cluster and the managed cluster are EKS clusters in the same AWS account.

Variables

$ export ACCOUNT_ID=<aws-account-id>
$ export REGION=us-east-1
$ export MGMT_CLUSTER=sveltos-mgmt
$ export MANAGED_CLUSTER=sveltos-managed
$ export SERVICE_ACCOUNTS="access-manager addon-controller event-manager hc-manager sc-manager techsupport-controller mcp-server"
# Agentless mode (agent.managementCluster: true)? drift-detection-manager and sveltos-agent-manager need access too:
# $ export SERVICE_ACCOUNTS="${SERVICE_ACCOUNTS} drift-detection-manager sveltos-agent-manager"

Step 1 — Create clusters as an IAM user (not root)

EKS grants cluster admin access to the IAM entity that creates the cluster. Root-created clusters cannot be accessed by IAM users without extra configuration.

$ eksctl create cluster --name ${MGMT_CLUSTER} --region ${REGION} --nodes 2
$ eksctl create cluster --name ${MANAGED_CLUSTER} --region ${REGION} --nodes 1

Step 2 — Install Sveltos on the management cluster

$ aws eks update-kubeconfig --name ${MGMT_CLUSTER} --region ${REGION}
$ kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/main/manifest/manifest.yaml
$ kubectl get pods -n projectsveltos

Step 3 — Associate the OIDC provider

eksctl does not register the OIDC provider automatically.

$ eksctl utils associate-iam-oidc-provider \
  --cluster ${MGMT_CLUSTER} --region ${REGION} --approve

$ export OIDC_ID=$(aws eks describe-cluster --name ${MGMT_CLUSTER} --region ${REGION} \
  --query "cluster.identity.oidc.issuer" --output text | awk -F'/' '{print $NF}')

Step 4 — Create an IAM role trusted by every Sveltos service account

One IAM role is shared by all the service accounts in SERVICE_ACCOUNTS (all in namespace projectsveltos). The trust policy's sub condition lists one entry per service account.

$ SUBS=""
$ for sa in ${SERVICE_ACCOUNTS}; do
    SUBS="${SUBS}\"system:serviceaccount:projectsveltos:${sa}\","
  done
$ SUBS="[${SUBS%,}]"

$ cat > /tmp/trust-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/oidc.eks.${REGION}.amazonaws.com/id/${OIDC_ID}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.${REGION}.amazonaws.com/id/${OIDC_ID}:aud": "sts.amazonaws.com"
        },
        "ForAnyValue:StringEquals": {
          "oidc.eks.${REGION}.amazonaws.com/id/${OIDC_ID}:sub": ${SUBS}
        }
      }
    }
  ]
}
EOF

$ aws iam create-role \
  --role-name sveltos-wi \
  --assume-role-policy-document file:///tmp/trust-policy.json

Step 5 — Annotate every service account

$ for sa in ${SERVICE_ACCOUNTS}; do
    kubectl annotate serviceaccount ${sa} -n projectsveltos \
      eks.amazonaws.com/role-arn=arn:aws:iam::${ACCOUNT_ID}:role/sveltos-wi --overwrite
  done

$ kubectl rollout restart deployment -n projectsveltos \
  access-manager addon-controller event-manager hc-manager sc-manager techsupport-controller mcp-server

drift-detection-manager and sveltos-agent-manager (agentless mode) are not static Deployments — Sveltos (re)creates their pods on demand, so no rollout restart is needed for them; the annotation is picked up the next time a pod is created.

Verify IRSA environment variables are injected, e.g. for sc-manager:

$ kubectl describe pod -n projectsveltos -l app=sc-manager | grep "AWS_ROLE_ARN\|AWS_WEB_IDENTITY_TOKEN_FILE"

Step 6 — Grant the IAM role access to the managed cluster

$ aws eks create-access-entry \
  --cluster-name ${MANAGED_CLUSTER} \
  --principal-arn arn:aws:iam::${ACCOUNT_ID}:role/sveltos-wi \
  --region ${REGION}

$ aws eks associate-access-policy \
  --cluster-name ${MANAGED_CLUSTER} \
  --principal-arn arn:aws:iam::${ACCOUNT_ID}:role/sveltos-wi \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \
  --access-scope type=cluster \
  --region ${REGION}

Step 7 — Get the managed cluster endpoint and CA certificate

$ export ENDPOINT=$(aws eks describe-cluster --name ${MANAGED_CLUSTER} --region ${REGION} \
  --query "cluster.endpoint" --output text)

$ aws eks describe-cluster --name ${MANAGED_CLUSTER} --region ${REGION} \
  --query "cluster.certificateAuthority.data" --output text \
  | base64 --decode > /tmp/managed-ca.crt

Step 8 — Register the managed cluster

$ sveltosctl register cluster-eks \
  --namespace=projectsveltos \
  --cluster=eks-managed \
  --endpoint=${ENDPOINT} \
  --eks-cluster-name=${MANAGED_CLUSTER} \
  --ca-file=/tmp/managed-ca.crt

Step 9 — Verify

$ kubectl get sveltoscluster eks-managed -n projectsveltos

READY should become true within a few seconds.

Troubleshooting

"the server has asked for the client to provide credentials": Verify the access entry and policy association:

$ aws eks list-associated-access-policies \
  --cluster-name ${MANAGED_CLUSTER} \
  --principal-arn arn:aws:iam::${ACCOUNT_ID}:role/sveltos-wi \
  --region ${REGION}

OIDC provider missing: aws iam list-open-id-connect-providers returns an empty list. Re-run step 3.

Cluster created as root: Only the creating IAM entity has access by default. Delete and recreate the cluster as your IAM user, or add the IAM user via an EKS access entry using root credentials.

AWS_REGION not set: --region is optional and falls back to the AWS_REGION environment variable injected by IRSA. Set it explicitly in the flag if you see a region error.

Both the management cluster and the managed cluster are GKE clusters in the same GCP project.

Variables

$ export PROJECT=<project-id>
$ export MGMT_CLUSTER=cluster-mgmt
$ export MANAGED_CLUSTER=cluster-managed
$ export ZONE=us-central1-a
$ export SERVICE_ACCOUNTS="access-manager addon-controller event-manager hc-manager sc-manager techsupport-controller mcp-server"
# Agentless mode (agent.managementCluster: true)? drift-detection-manager and sveltos-agent-manager need access too:
# $ export SERVICE_ACCOUNTS="${SERVICE_ACCOUNTS} drift-detection-manager sveltos-agent-manager"

Step 1 — Create clusters

$ gcloud container clusters create ${MGMT_CLUSTER} \
  --zone=${ZONE} --project=${PROJECT}

$ gcloud container clusters create ${MANAGED_CLUSTER} \
  --zone=${ZONE} --project=${PROJECT}

Step 2 — Enable Workload Identity on the management cluster

$ gcloud container clusters update ${MGMT_CLUSTER} \
  --workload-pool=${PROJECT}.svc.id.goog \
  --zone=${ZONE} --project=${PROJECT}

Step 3 — Enable GKE_METADATA on the management node pool

Without this, pods cannot use Workload Identity even when the cluster has it enabled.

$ gcloud container node-pools update default-pool \
  --cluster=${MGMT_CLUSTER} \
  --zone=${ZONE} --project=${PROJECT} \
  --workload-metadata=GKE_METADATA

Wait for node rotation to complete:

$ gcloud container node-pools describe default-pool \
  --cluster=${MGMT_CLUSTER} --zone=${ZONE} --project=${PROJECT} \
  --format='value(config.workloadMetadataConfig.mode)'
# should print: GKE_METADATA

Step 4 — Install Sveltos on the management cluster

$ gcloud container clusters get-credentials ${MGMT_CLUSTER} \
  --zone=${ZONE} --project=${PROJECT}

$ kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/main/manifest/manifest.yaml

Step 5 — Create a Google Service Account for Sveltos

$ gcloud iam service-accounts create sveltos-wi --project=${PROJECT}

Step 6 — Link every Kubernetes service account to the GSA

$ for sa in ${SERVICE_ACCOUNTS}; do
    gcloud iam service-accounts add-iam-policy-binding \
      sveltos-wi@${PROJECT}.iam.gserviceaccount.com \
      --role=roles/iam.workloadIdentityUser \
      --member="serviceAccount:${PROJECT}.svc.id.goog[projectsveltos/${sa}]" \
      --project=${PROJECT}

    kubectl annotate serviceaccount ${sa} \
      -n projectsveltos \
      iam.gke.io/gcp-service-account=sveltos-wi@${PROJECT}.iam.gserviceaccount.com --overwrite
  done

$ kubectl rollout restart deployment -n projectsveltos \
  access-manager addon-controller event-manager hc-manager sc-manager techsupport-controller mcp-server

drift-detection-manager and sveltos-agent-manager (agentless mode) are not static Deployments — Sveltos (re)creates their pods on demand, so no rollout restart is needed for them; the annotation is picked up the next time a pod is created.

Step 7 — Grant the GSA access to the managed cluster

$ gcloud projects add-iam-policy-binding ${PROJECT} \
  --member=serviceAccount:sveltos-wi@${PROJECT}.iam.gserviceaccount.com \
  --role=roles/container.admin

Step 8 — Get the managed cluster endpoint and CA certificate

$ export ENDPOINT=$(gcloud container clusters describe ${MANAGED_CLUSTER} \
  --zone=${ZONE} --project=${PROJECT} \
  --format='value(endpoint)')

$ gcloud container clusters describe ${MANAGED_CLUSTER} \
  --zone=${ZONE} --project=${PROJECT} \
  --format='value(masterAuth.clusterCaCertificate)' \
  | base64 --decode > /tmp/ca.crt

Step 9 — Register the managed cluster

$ sveltosctl register cluster-gke \
  --namespace=projectsveltos \
  --cluster=gke-managed \
  --endpoint=https://${ENDPOINT} \
  --project-id=${PROJECT} \
  --gke-cluster-name=${MANAGED_CLUSTER} \
  --location=${ZONE} \
  --ca-file=/tmp/ca.crt

Step 10 — Verify

$ kubectl get sveltoscluster gke-managed -n projectsveltos

READY should become true within a few seconds.

Troubleshooting

"the server has asked for the client to provide credentials": The node pool is not using GKE_METADATA mode. Run step 3 and wait for node rotation to complete, then restart the sc-manager pod.

PROJECT is empty: Shell variables are lost between sessions. Re-export them before running any gcloud commands.

Unlike the cloud examples above, the management and managed clusters don't need to be related in any way. All that's required is that the managed cluster's kube-apiserver trusts Dex as an OIDC issuer, and that Sveltos holds a client_id/client_secret Dex will accept.

Step 1 — Enable the client credentials grant on Dex

The client credentials grant (RFC 6749 §4.4) is what lets Sveltos exchange a client_id/client_secret directly for a token, without a user login. It must be turned on explicitly:

$ export DEX_CLIENT_CREDENTIAL_GRANT_ENABLED_BY_DEFAULT=true

# dex config.yaml
oauth2:
  grantTypes:
    - "client_credentials"
staticClients:
  - id: sveltos
    secret: <a-strong-secret>
    name: Sveltos

Step 2 — Trust Dex from the managed cluster's kube-apiserver

Add these flags to the managed cluster's kube-apiserver. The exact mechanism depends on how the cluster is provisioned: kubeadm's ClusterConfiguration, a Cluster API KubeadmControlPlane/ClusterClass patch, or your distribution's equivalent.

--oidc-issuer-url=https://<dex-host>:<port>/dex
--oidc-client-id=sveltos
--oidc-username-claim=aud
--oidc-username-prefix=-
--oidc-ca-file=/etc/kubernetes/pki/dex-ca.crt   # only if Dex's cert isn't already trusted

--oidc-username-claim=aud reads the token's audience claim, which the client credentials grant sets to the client_id verbatim (sveltos here). Combined with --oidc-username-prefix=- (no prefix), the resulting Kubernetes username is exactly sveltos. That's simpler to reason about than the sub claim, which Dex encodes internally rather than leaving as the plain client_id.

Step 3 — Grant that identity the permissions Sveltos needs

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: sveltos-oidc-workload-identity
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: User
    name: sveltos
    apiGroup: rbac.authorization.k8s.io

Step 4 — Register the cluster

See the OIDC tab under Programmatic Registration for the SveltosCluster and Secrets to apply.

Step 5 — Verify

$ kubectl get sveltoscluster <name> -n <namespace>

READY should become true within a few seconds.

Troubleshooting

x509: certificate signed by unknown authority in sc-manager/addon-controller logs: the token exchange with Dex's own token endpoint is failing TLS verification. This is a different trust boundary from Step 2: set oidc.caSecretRef in the SveltosCluster to a Secret containing Dex's own CA (see Programmatic Registration).

... is forbidden: User "sveltos" cannot get resource ...: the token is being accepted, but no RBAC grants that identity anything. Revisit Step 3.

The setup is very similar to the one with DEX. We will provide instructions on how to allow Sveltos to register Kubernetes clusters using Keycloak as a Workload Identity. The full guide is available here.

Step 1 — Realm, Client ID and Configuration Mappers

Realm and Client ID

1. Create a new Realm with the name sveltos-realm
2. Create a Client ID test-env-auth
3. Under Capability config: Enable Client authentication and enable Service account roles in the Authentication flow section
4. Save the configuration

Mapper Configuration -> Hardcoded claim

1. Set Name to test-env-user
2. Set Token Claim Name to test-env-user
3. Set Claim value to sveltos
4. Set Claim JSON Type to string
5. Enable Add to access token
6. Save the configuration

Mapper Configuration -> Audience

1. Set Name to test-env-audience
2. Set Included Custom Audience to test-env-auth
3. Disable Add to ID token
4. Enable Add to access token
5. Save the configuration

Step 2 — Update Managed Cluster Kubernetes API Server Details

Add these flags to the managed cluster's kube-apiserver. The exact mechanism depends on how the cluster is provisioned: kubeadm's ClusterConfiguration, a Cluster API KubeadmControlPlane/ClusterClass patch, or your distribution's equivalent.

--oidc-issuer-url=https://<Your Keycloak Domain>/realms/sveltos-realm
--oidc-client-id=test-env-auth
--oidc-username-claim=test-env-user
--oidc-username-prefix=-
--oidc-ca-file=/etc/kubernetes/pki/keycloak-ca.crt # Only if Keycloak's cert is not trusted

The oidc-username-claim=test-env-user value resolves to sveltos. Remember, this comes from the Hardcoded claim created in a previous step. The oidc-username-prefix=- field means no prefix is added. The Kubernetes username is sveltos.

Step 3 — Grant Indentity the Permissions Sveltos Requires

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: sveltos-oidc-workload-identity
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: User
    name: sveltos
    apiGroup: rbac.authorization.k8s.io

Step 4 — Register the cluster

See the OIDC tab under Programmatic Registration for the SveltosCluster and Secrets to apply. The cluster can be registered in a different namespace instead of projectsveltos. Choose the namespace of your preference.

Step 5 — Verification

$ kubectl get sveltoscluster <name> -n <namespace>

READY should become true within a few seconds.

Troubleshooting

x509: certificate signed by unknown authority in sc-manager/addon-controller logs: The token exchange with Dex's own token endpoint is failing TLS verification. This is a different trust boundary from Step 2: set oidc.caSecretRef in the SveltosCluster to a Secret containing Dex's own CA (see Programmatic Registration).

... is forbidden: User "sveltos" cannot get resource ...: the token is being accepted, but no RBAC grants that identity anything. Revisit Step 3.