Skip to content

Kubernetes Cluster Classification - Project Sveltos

Classifier - Automatically Manage Cluster Labels and Add-Ons

Sveltos provides users with the power to decide which add-ons should get deployed to which clusters programmatically by the use of a ClusterSelector. Sometimes the versions of the required and needed add-ons depend on the cluster's runtime state. This is where the Sveltos Classifier comes into play.

With the Classifier, Sveltos can be configured to automatically update the cluster labels based on the cluster runtime state. As the runtime state changes, the cluster labels are automatically updated. This makes sure the right ClusterProfile instances match the chosen clusters. Then, it allows automatic upgrades of the Kubernetes add-ons.

Once the Classifier is deployed in the management cluster, it is distributed to each cluster. A Sveltos service runs in every managed cluster to monitor the cluster's runtime state. As soon as a match is detected, the information is transmitted back to the management cluster, and the cluster labels are appropriately updated by Sveltos.

Sveltos can track each cluster's runtime status by combining the Classifier with the ClusterProfile resource. When the cluster's state changes, it updates the cluster labels and manages the deployment and upgrade of Kubernetes add-ons as needed.

Classifier in action

Use Case: Upgrade Helm Charts when Kubernetes Cluster is Upgraded

Suppose we are managing several Kubernetes clusters with different versions, and we want to deploy the points below.

  1. OPA Gatekeeper version 3.10.0 in any Kubernetes cluster whose version is >= v1.25.0
  2. OPA Gatekeeper version 3.9.0 in any Kubernetes cluster whose version is >= v1.24.0 && < v1.25.0

Management Cluster

ClusterProfile

---
apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: deploy-gatekeeper-3-10
spec:
  clusterSelector:
    matchLabels:
      gatekeeper: v3-10
  syncMode: Continuous
  helmCharts:
  - repositoryURL: https://open-policy-agent.github.io/gatekeeper/charts
    repositoryName: gatekeeper
    chartName: gatekeeper/gatekeeper
    chartVersion:  3.10.0
    releaseName: gatekeeper
    releaseNamespace: gatekeeper
    helmChartAction: Install

---
apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: deploy-gatekeeper-3-9
spec:
  clusterSelector:
    matchLabels:
      gatekeeper: v3-9
  syncMode: Continuous
  helmCharts:
  - repositoryURL: https://open-policy-agent.github.io/gatekeeper/charts
    repositoryName: gatekeeper
    chartName: gatekeeper/gatekeeper
    chartVersion:  3.9.0
    releaseName: gatekeeper
    releaseNamespace: gatekeeper
    helmChartAction: Install

Classifier

---
apiVersion: lib.projectsveltos.io/v1beta1
kind: Classifier
metadata:
  name: deploy-gatekeeper-3-10
spec:
  classifierLabels:
  - key: gatekeeper
    value: v3-10
  kubernetesVersionConstraints:
  - comparison: GreaterThanOrEqualTo
    version: 1.25.0

---
apiVersion: lib.projectsveltos.io/v1beta1
kind: Classifier
metadata:
  name: deploy-gatekeeper-3-9
spec:
  classifierLabels:
  - key: gatekeeper
    value: v3-9
  kubernetesVersionConstraints:
  - comparison: GreaterThanOrEqualTo
    version: 1.24.0
  - comparison: LessThan
    version: 1.25.0

Based on the configuration above, we achieved the below outcomes.

  1. Any cluster with a Kubernetes version v1.24.x will get the label gatekeeper:v3.9 added, and then the Gatekeeper v3.9.0 Helm chart will be deployed
  2. Any cluster with a Kubernetes version v1.25.x will get the label gatekeeper:v3.10 added, and then the Gatekeeper v3.10.0 Helm chart will be deployed
  3. As soon as a cluster is upgraded from Kubernetes v1.24.x to v1.25.x, the Gatekeeper Helm chart will be automatically upgraded from 3.9.0 to 3.10.0

More Resources

To read more about the Classifier configuration, with examples using the resources and the Lua language, have a look at the section.

More Examples

  1. Classify clusters based on Kubernetes version classifier.yaml
  2. Classify clusters based on the number of namespaces classifier.yaml
  3. Classify clusters based on Kubernetes version and resources classifier.yaml

Use Case: Metrics-based Classifier

There are cases when end-users want to deploy add-ons and applications in Kubernetes clusters that are not heavily loaded. That means, based on the CPU, memory, or any other metric, controlling what will get deployed and where.

We can achieve this functionality by letting the deployedResourceConstraint reference resources in the management cluster. The available feature request is located here.

Example

apiVersion: lib.projectsveltos.io/v1beta1
kind: Classifier
metadata:
  name: low-load
spec:
  classifierLabels:
  - key: load # Label applied to matching clusters
    value: low
  deployedResourceConstraint:
    resourceSelectors:
    - group: metrics.keptn.sh
      version: v1alpha1 # Adjust based on the Keptn version
      kind: KeptnMetric
      namespace: sveltos-metrics # KeptnMetric runs in the mgmt cluster
      name: keptnmetric-cpu-{{ .cluster }}
      evaluate: | # Use Lua for conditional checks
        function evaluate()
          local hs = { matching = false, message = "" }
          local v = tonumber(obj.status.value)
          if v and v < 0.5 then
            hs.matching = true          -- cluster is “low load”
          else
            hs.message = "CPU util " .. tostring(v)
          end
          return hs
        end
  # <--- new field proposed in #375
  sourceClusterScope: Local

How does the proposed resource work? The KeptnMetric.status.value already carries the live metric. The Lua code used will evaluate the block checks and the threshold directly. No additional controller is required. When the defined metric goes over the defined limit, the label is either added or removed, and any ClusterProfile will react automatically.

Classifier CRD - Deep Dive

Classifier CRD is the CRD used to instructs Sveltos on how to classify a cluster.

Classifier Labels

The field classifierLabels contains all the labels (key/value pair) which will be added automatically to any cluster matching a Classifier instance.

Kubernetes Version constraints

The field kubernetesVersionConstraints can be used to classify a cluster based on its current Kubernetes version.

Resource Constraints

The field deployedResourceConstraint can be used to classify a cluster based on current deployed resources. Resources are identified by Group/Version/Kind and can be filtered based on their namespace, labels and some fields. It supports Lua script as well.

Following Classifier matches any cluster with a service with the label sveltos:fv.

---
apiVersion: lib.projectsveltos.io/v1beta1
kind: Classifier
metadata:
  name: sveltos-service
spec:
  classifierLabels:
  - key: sveltos-service
    value: present
  deployedResourceConstraint:
    resourceSelectors:
    - group: ""
      version: v1
      kind: Service
      labelFilters:
      - key: sveltos
        operation: Equal
        value: fv

Following Classifier, matches any cluster with a ClusterIssuer using acme-staging-v02.api.letsencrypt.org.

---
apiVersion: lib.projectsveltos.io/v1beta1
kind: Classifier
metadata:
  name: acme-staging-v02
spec:
  classifierLabels:
  - key: issuer
    value: acme-staging-v02
  deployedResourceConstraint:
    resourceSelectors:
    - group: "cert-manager.io"
      version: v1
      kind: ClusterIssuer
      evaluate: |
        function evaluate()
          hs = {}
          hs.matching = false
          hs.message = ""
          if obj.spec.acme ~= nil then
            if string.find(obj.spec.acme.server, "acme-staging-v02.api.letsencrypt.org", 1, true) then
              hs.matching = true
            end
          end
          return hs
        end

A Classifier can also look at the resources of different kinds altogether.

AggregatedClassification is optional and can be used to specify a Lua function that will be used to further detect whether the subset of the resources selected using the ResourceSelectors field is a match for this Classifier. The function will receive the array of resources selected by ResourceSelectors. If this field is not specified, a cluster is a match for the Classifier instance, if all ResourceSelectors returns at least one match. This field allows to perform more complex evaluation on the resources, looking at all resources together. This can be useful for more sophisticated tasks, such as identifying resources that are related to each other or that have similar properties. The Lua function must return a struct with:

  • "matching" field: boolean indicating whether the cluster is a match
  • "message" field: (optional) message

Note

Keep in mind the CEL language can be used as a way to express logic.

Classifier Controller Configuration

  1. concurrent-reconciles: By default, Sveltos manager reconcilers run with a parallelism set to 10. This arg can be used to change the level of parallelism
  2. worker-number: Number of workers performing long-running task. By default, this is set to 20. If the number of Classifier instances is in the hundreds, please consider increasing this
  3. report-mode: By default, the Classifier controller running in the management cluster periodically collects ClassifierReport instances from each managed cluster. Setting report-mode to "1" will change this and have each Classifier Agent send back ClassifierReport to the management cluster. When setting report-mode to 1, control-plane-endpoint must be set as well. When in this mode, Sveltos automatically creates a ServiceAccount in the management cluster for Classifier Agent. Only permissions granted for this ServiceAccount are the update of ClassifierReports
  4. control-plane-endpoint: The management cluster controlplane endpoint. Format <ip>:<port>. This must be reachable from each managed cluster