Skip to content

Notifications - Projectsveltos

Introduction to Notifications

Sveltos uses ClusterProfiles/Profiles to automatically track matching clusters and deploy specified add-ons (like Helm charts or Kubernetes resources). It can then assess the cluster health (ensuring all add-ons are ready) and send notifications. These notifications allow external tools to trigger further workflows, like CI/CD pipelines, only once the cluster is confirmed healthy and stable.

ClusterHealthCheck

ClusterHealthCheck is the CRD that can be used to:

  1. Define the cluster health checks;
  2. Instruct Sveltos when and how to send notifications

Cluster Selection

The clusterSelector field is a Kubernetes label selector. Sveltos uses it to detect all the clusters to assess health and send out notifications.

LivenessChecks

The livenessCheck field is a list of cluster liveness checks to be evaluated.

The supported types are:

  1. Addons: Addons type instructs Sveltos to evaluate state of add-ond deployment in such a cluster;
  2. HealthCheck: HealthCheck type allows to define a custom health check for any Kubernetes type.

Notifications

The notifications fields is a list of all notifications to be sent when the liveness check state changes.

The supported types are:

  1. Slack Slack
  2. Webex Webex
  3. Teams Teams
  4. Discord Discord
  5. Telegram Telegram
  6. SMTP SMTP
  7. Kubernetes Kubernetes events (reason=ClusterHealthCheck)

HealthCheck CRD

The HealthCheck resource defines a custom health assessment by first selecting Kubernetes resources and then applying custom evaluation logic to determine their collective health.

  • resourceSelectors
    • Purpose: Resource Selection
    • Details: An array of ResourceSelector objects. These define the Kubernetes resources to monitor by specifying their Group, Version, Kind, Namespace, and Name.
  • resourceSelectors[*].LabelFilters
    • Purpose: Filtering by Label
    • Details: Filters the selected resources using standard label operations: Equal, Different, Has, or DoesNotHave.
  • resourceSelectors[*].Evaluate
    • Purpose: Lua Pre-Filter (Optional)
    • Details: An optional Lua script used to additionally filter resources before the main health check is performed.
  • resourceSelectors[*].EvaluateCEL
    • Purpose: CEL Pre-Filter (Optional)
    • Details: An optional list of Common Expression Language (CEL) rules used to additionally filter resources.
  • evaluateHealth
    • Purpose: Custom Health Evaluation
    • Details: A mandatory Lua script that performs the core health check logic on all the final, filtered resources.
  • flapping.consecutiveEvaluations
    • Purpose: Flapping mitigation (Optional)
    • Details: The number of consecutive evaluations a resource must be found in the same non-Healthy status before it is reported. Omit flapping entirely to report every non-Healthy resource immediately, exactly as before this field existed. See Example: Filtering Out Flapping Resources below.

The Spec.evaluateHealth field must contain a Lua script with a function named evaluate().

The healthcheck-manager/examples/healthchecks directory collects ready-to-apply HealthCheck definitions for well-known CRDs (Velero Backup, Kyverno PolicyReport, cert-manager Certificate, Job, StatefulSet rollout, CloudNativePG Cluster, Contour HTTPProxy, Knative Service). Each one is covered by a unit test, so the scripts are exactly what's tested, not a copy that can drift out of sync.

Input Access: The function accesses all Kubernetes resources selected by resourceSelectors using the global Lua variable: resources.

Required Output: It must return an array of tables (structured instances), with the following required and optional fields for each evaluated resource:

  • resource
    • Type: Object
    • Description: The specific Kubernetes resource that was evaluated.
  • healthStatus
    • Type: String
    • Description: The assessment of the resource's health. Must be one of: Healthy, Progressing, Degraded, or Suspended.
  • message
    • Type: String
    • Description: Optional, an informative message providing context for the status.
  • reEvaluate
    • Type: Boolean
    • Description: Optional. If set to true, the health check will be automatically re-evaluated in 10 seconds.
  • ignore
    • Type: Boolean
    • Description: Optional. If set to true, Sveltos will ignore this resource's result during the overall health calculation.

Example: ConfigMap HealthCheck

In the follwoing example1, we are creating an HealthCheck that watches all the ConfigMap Kubernetes resources.

hs is the health status object we will return to Sveltos. It must contain a status attribute which indicates whether the resource is Healthy, Progressing, Degraded or Suspended. By default,the status is set to Healthy and the hs.ignore is set to true, as we do not want to mess with the status of other, non-OPA ConfigMaps. Optionally, the health status object may also contain a message.

In this example, we want to identify if the ConfigMap is an OPA policy or another kind of ConfigMap. If it is a OPA policy, we retrieve the value of the openpolicyagent.org/policy-status annotation. The annotation is set to {"status":"ok"} if the policy loaded successfully. If errors occurred during loading (e.g., the policy contained a syntax error) the cause will be reported in the annotation. Depending on the value of the annotation, we set the status and message attributes appropriately.

At the end, we return the hs object to Sveltos.

Example - HealthCheck Definition

---
apiVersion: lib.projectsveltos.io/v1beta1
kind: HealthCheck
metadata:
  name: opa-configmaps
spec:
  resourceSelectors:
  - group: ""
    version: v1
    kind: ConfigMap
  evaluateHealth: |
    function evaluate()
      statuses = {}

      status = "Healthy"
      message = ""

      local opa_annotation = "openpolicyagent.org/policy-status"

      for _,resource in ipairs(resources) do
        if resource.metadata.annotations ~= nil then
          if resource.metadata.annotations[opa_annotation] ~= nil then
            if obj.metadata.annotations[opa_annotation] == '{"status":"ok"}' then
              status = "Healthy"
              message = "Policy loaded successfully"
            else
              status = "Degraded"
              message = obj.metadata.annotations[opa_annotation]
            end
            table.insert(statuses, {resource=resource, status = status, message = message})
          end
        end
      end
      local hs = {}
      if #statuses > 0 then
        hs.resources = statuses
      end
      return hs
    end

The below ClusterHealthCheck resources, will send a Webex message as notification if a ConfigMap with an incorrect OPA policy is detected.

---
apiVersion: lib.projectsveltos.io/v1beta1
kind: ClusterHealthCheck
metadata:
  name: hc
spec:
  clusterSelector:
    matchLabels:
      env: fv
  livenessChecks:
  - name: deployment
    type: HealthCheck
    livenessSourceRef:
      kind: HealthCheck
      apiVersion: lib.projectsveltos.io/v1beta1
      name: opa-configmaps
  notifications:
  - name: webex
    type: Webex
    notificationRef:
      apiVersion: v1
      kind: Secret
      name: webex
      namespace: default

Example: Filtering Out Flapping Resources

By default, a HealthCheck re-evaluates a resource only when Sveltos detects a change to it. This is efficient, but it also means a resource that flips briefly into a bad state (a Pod restarting once, a Job failing and immediately retrying) can trigger a Degraded report for something that was never really a problem.

The flapping field tells Sveltos to hold a resource in a pending state until it has been observed in the same non-Healthy status for a set number of consecutive evaluations, instead of reporting it right away. Recovery is always immediate: as soon as a resource is evaluated Healthy again, any pending count for it is cleared.

While a resource has not yet crossed the threshold, Sveltos keeps re-evaluating that HealthCheck, even without a new change on the resource, so the pending count keeps advancing until it either crosses the threshold or the resource recovers. In this scenario, HealthCheck instances are re-evaluated roughly every 10 seconds on average.

Take the classic CrashLoopBackOff Pod: without flapping, a script has to hand-roll its own debounce logic, parsing lastTransitionTime off a condition and comparing it to os.time(). With flapping, the script only needs to report the instantaneous truth and Sveltos takes care of the rest:

Example - HealthCheck Definition with Flapping

---
apiVersion: lib.projectsveltos.io/v1beta1
kind: HealthCheck
metadata:
  name: pod-crashloopbackoff
spec:
  collectResources: true
  resourceSelectors:
  - group: ""
    version: v1
    kind: Pod
  flapping:
    consecutiveEvaluations: 6
  evaluateHealth: |
    function evaluate()
      local statuses = {}

      for _, pod in ipairs(resources) do
        local hasError = false

        if pod.status and pod.status.containerStatuses then
          for _, container in ipairs(pod.status.containerStatuses) do
            if container.state and container.state.waiting then
              local reason = container.state.waiting.reason
              if reason == "CrashLoopBackOff" or reason == "BackOff" then
                hasError = true
              end
            end
          end
        end

        if hasError then
          table.insert(statuses, {resource = pod, status = "Degraded",
            message = "Container is in CrashLoopBackOff"})
        end
      end

      local hs = {}
      if #statuses > 0 then
        hs.resources = statuses
      end
      return hs
    end

With HealthCheck instances re-evaluated roughly every 10 seconds on average, consecutiveEvaluations: 6 means a Pod has to be observed crash-looping for about 60 seconds before it is reported Degraded. A single restart, or two restarts a few seconds apart, never reaches the threshold and is never reported.

Notifications and multi-tenancy

If the below label is set on the HealthCheck instance created by the tenant admin

projectsveltos.io/admin-name: <admin>

Sveltos will ensure the tenant admin can define notifications only by looking at the resources it has been authorized to by platform admin.

Sveltos suggests using the below Kyverno ClusterPolicy, which takes care of adding proper labels to each HealthCheck at creation time.

---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-labels
  annotations:
    policies.kyverno.io/title: Add Labels
    policies.kyverno.io/description: >-
      Adds projectsveltos.io/admin-name label on each HealthCheck
      created by tenant admin. It assumes each tenant admin is
      represented in the management cluster by a ServiceAccount.
spec:
  background: false
  rules:
  - exclude:
      any:
      - clusterRoles:
        - cluster-admin
    match:
      all:
      - resources:
          kinds:
          - HealthCheck
    mutate:
      patchStrategicMerge:
        metadata:
          labels:
            +(projectsveltos.io/serviceaccount-name): '{{serviceAccountName}}'
            +(projectsveltos.io/serviceaccount-namespace): '{{serviceAccountNamespace}}'
    name: add-labels
  validationFailureAction: enforce

  1. Credit for this example to https://blog.cubieserver.de/2022/argocd-health-checks-for-opa-rules/ ↩