Sveltos - Kubernetes Add-on Controller | Manage Kubernetes Add-ons with Ease

ClusterProfile policyRefs Reference

The ClusterProfile spec.policyRefs is a list of Secrets/ConfigMaps. Both Secrets and ConfigMaps data fields can be a list of key-value pairs. Any key is acceptable, and the value can be multiple objects in YAML or JSON format¹.

Example: Create a Secret

To create a Kubernetes Secret that contains the Calico YAMLs and make it usable with Sveltos, utilise the below commands.

$ wget https://raw.githubusercontent.com/projectcalico/calico/master/manifests/calico.yaml

$ kubectl create secret generic calico --from-file=calico.yaml --type=addons.projectsveltos.io/cluster-profile

The commands will download the calico.yaml manifest file and afterwards create a Kubernetes secret of type generic by defining the file downloaded in the previous command plus defining the needed type=addons.projectsveltos.io/cluster-profile.

Example: Create a ConfigMap

The YAML definition below exemplifies a ConfigMap that holds multiple resources². When a ClusterProfile instance references the ConfigMap, a Namespace and a Deployment instance are automatically deployed in any managed cluster that adheres to the ClusterProfile clusterSelector.

cat > nginx_cm.yaml <<EOF
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx
  namespace: default
data:
  namespace.yaml: |
    kind: Namespace
    apiVersion: v1
    metadata:
      name: nginx
  deployment.yaml: |
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: nginx-deployment
      namespace: nginx
    spec:
      replicas: 2 # number of pods to run
      selector:
        matchLabels:
          app: nginx
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx:latest # public image from Docker Hub
            ports:
            - containerPort: 80
EOF

Once the required Kubernetes resources are created/deployed, the below example represents a ClusterProfile resource that references the ConfigMap and the Secret created above.

cat > clusterprofile_deploy_nginx.yaml <<EOF
---
apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: deploy-resources
spec:
  clusterSelector:
    matchLabels:
      env: fv
  policyRefs:
  - name: nginx
    namespace: default
    kind: ConfigMap
  - name: calico
    namespace: default
    kind: Secret
EOF

When a ClusterProfile references a ConfigMap or a Secret, the kind and name fields are required, while the namespace field is optional. Specifying a namespace uniquely identifies the resource using the tuple namespace, name, and kind, and that resource will be used for all matching clusters.

If you leave the namespace field empty, Sveltos will search for the ConfigMap or the Secret with the provided name within the namespace of each matching cluster.

Example: Understand Namespace Definition

---
apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: deploy-resources
spec:
  clusterSelector:
    matchLabels:
      env: fv
  policyRefs:
  - name: nginx
    kind: ConfigMap

Consider the provided ClusterProfile, when we have two workload clusters matching. One in the foo namespace and another in the bar namespace. Sveltos will search for the ConfigMap nginx in the foo namespace for the Cluster in the foo namespace and for a ConfigMap ngix in the bar namespace for the Cluster in the bar namespace.

More ClusterProfile examples can be found here.

Bypassing Automatic Namespace Creation

By default, when Sveltos deploys resources defined in policyRefs or kustomizationRefs, it automatically checks if the target namespace exists in the managed cluster. If the namespace is missing, Sveltos attempts to create it.

While convenient, this requires Sveltos to have cluster-wide get and create permissions for Namespaces. In multi-tenant or restricted RBAC environments, namespaces are often pre-provisioned by cluster administrators, and Sveltos may not have the permissions required to manage namespaces at the cluster level.

To handle these scenarios, you can use the skipNamespaceCreation field. When set to true, Sveltos bypasses the namespace check/creation logic and attempts to deploy resources directly into the existing namespace.

apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: deploy-to-preprovisioned-ns
spec:
  clusterSelector:
    matchLabels:
      env: production
  policyRefs:
    - name: nginx
      namespace: default
      kind: ConfigMap
      # If true, Sveltos won't try to create the target namespace
      # even if it is defined within the ConfigMap resources.
      skipNamespaceCreation: true

Example: Template-based Referencing for ConfigMaps and Secrets

We can express ConfigMap and Secret names as templates. This allows us to generate them dynamically based on the available cluster information, simplifying management and reducing repetition.

Available cluster information

• cluster namespace: .Cluster.metadata.namespace
• cluster name: .Cluster.metadata.name
• cluster type: .Cluster.kind

Consider two SveltosCluster instances in the civo namespace.

$ kubectl get sveltoscluster -n civo --show-labels
NAME             READY   VERSION        LABELS
pre-production   true    v1.29.2+k3s1   env=civo,projectsveltos.io/k8s-version=v1.29.2
production       true    v1.28.7+k3s1   env=civo,projectsveltos.io/k8s-version=v1.28.7

Two ConfigMaps named nginx-pre-production and nginx-production exist in the same namespace.

$ kubectl get configmap -n civo
NAME                   DATA   AGE
nginx-pre-production   2      4m59s
nginx-production       2      4m41s

The only difference between the ConfigMaps is the replicas setting: 1 for pre-production and 3 for production.

The below points are included in the ClusterProfile.

1. Matches both SveltosClusters
2. Dynamic ConfigMap Selection:
   • For the pre-production cluster, the profile should use the nginx-pre-production ConfigMap.
   • For the production cluster, the profile should use the nginx-production ConfigMap.

---
apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: deploy-nginx
spec:
  clusterSelector:
    matchLabels:
      env: civo
  policyRefs:
  - name: nginx-{{ .Cluster.metadata.name }}
    kind: ConfigMap

The demonstrated approach provides a flexible and centralized way to reference ConfigMaps and Secrets based on the availanle cluster information.

Template

Define the content for resources in your PolicyRefs using templates. During deployment, Sveltos will automatically populate these templates with relevant information from your cluster and other resources in the management cluster. See the template section template section for details.

Remember to adapt the provided resources to your specific repository structure, cluster configuration, and desired templating logic.

Remote URL Sources

Instead of storing YAML in a ConfigMap or Secret, a PolicyRef entry can point directly at a remote source. This removes the ~1 MB size limit imposed by ConfigMaps. Two source types are supported:

Sveltos fetches the content on every reconciliation and redeploys only when the content hash changes. A configurable interval controls how often Sveltos re-fetches (default: 5 minutes).

HTTP/HTTPS

Reference upstream manifests hosted on any HTTP/HTTPS endpoint. For example, operator releases hosted on GitHub.

spec:
  policyRefs:
  - deploymentType: Remote
    remoteURL:
      url: https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
      interval: 1h0m0s

OCI Registry

Reference YAML manifests stored as OCI artifacts in any OCI-compliant registry. Sveltos pulls the artifact layers and extracts all .yaml, .yml, and .json files found inside:

spec:
  policyRefs:
  - deploymentType: Remote
    remoteURL:
      url: oci://ghcr.io/my-org/my-manifests:v1.0.0
      interval: 5m0s

The artifact can be produced with any OCI-compatible tool, for example with [ORAS}(https://github.com/oras-project/oras).

$ oras push ghcr.io/my-org/my-manifests:v1.0.0 manifests/

Authentication

Private sources are supported via a secretRef. If a namespace is omitted, Sveltos defaults to the matching cluster's namespace. Both name and namespace are treated as Go templates, so they can reference cluster fields. An example can be found below.

secretRef:
  name: registry-auth-{{ .Cluster.metadata.name }}
  namespace: "{{ .Cluster.metadata.namespace }}"

The Secret must use type addons.projectsveltos.io/cluster-profile and can contain the following keys:

$ kubectl create secret generic registry-auth \
    --namespace=projectsveltos \
    --from-literal=username=<username> \
    --from-literal=password=<token-or-password> \
    --type=addons.projectsveltos.io/cluster-profile

spec:
  policyRefs:
  - deploymentType: Remote
    remoteURL:
      url: oci://ghcr.io/my-org/private-manifests:v1.0.0
      secretRef:
        name: registry-auth
        namespace: projectsveltos

The same secretRef syntax applies to private HTTP endpoints.

spec:
  policyRefs:
  - deploymentType: Remote
    remoteURL:
      url: https://private-server.example.com/manifests/app.yaml
      secretRef:
        name: registry-auth
        namespace: projectsveltos

Template rendering

Set template: true to treat the fetched content as a Go template. Sveltos instantiates it using management-cluster data, the same way it handles the projectsveltos.io/template annotation on a ConfigMap.

spec:
  policyRefs:
  - deploymentType: Remote
    remoteURL:
      url: oci://ghcr.io/my-org/my-manifests:v1.0.0
      template: true

Subresources

Sveltos can update specific subresources of a resource. This is achieved by leveraging the projectsveltos.io/subresources annotation. When the annotation is present on a resource referenced in the PolicyRefs section, Sveltos updates the designated subresources alongside the main resource. Subresources are specified as a comma-separated list within the annotation value.

For example, to instruct Sveltos to update the status subresource of a Service, we can create a ConfigMap with the following structure and reference this ConfigMap from a ClusterProfile/Profile.

---
apiVersion: v1
data:
  service.yaml: |
    apiVersion: v1
    kind: Service
    metadata:
      name: sveltos-subresource
      namespace: default
    spec:
      selector:
        app: foo
      ports:
      - name: my-port
        port: 443
        protocol: TCP
        targetPort: 1032
      type: LoadBalancer
    status:
      loadBalancer:
        ingress:
        - ip: 1.1.1.1
kind: ConfigMap
metadata:
  annotations:
    projectsveltos.io/subresources: status
  name: load-balancer-service
  namespace: default