This section describes how to define alerting rules for Kubernetes logs.

Note

WizTelemetry Events Alerting receives logs from the Kubernetes cluster by default. To receive other logs, you need to modify the configuration of the WizTelemetry Data Pipeline or create a new Secret for configuration. Please refer to the Vector official documentation for configuration.

Prerequisites

  • You need to have the platform-admin role in the KubeSphere platform.

  • The WizTelemetry Events Alerting extension needs to be installed and enabled on the KubeSphere platform.

  • Log alerting has been enabled in the extension configuration of WizTelemetry Events Alerting, i.e., loggingAlerting.enabled is set to true.

Steps

  1. On any cluster node, create a YAML file to define log alerting rules for that cluster. For more information on parameter configuration, please refer to Parameter Description.

    • Keyword-based Alerting

      For example, the following configuration means an alert is triggered when the test log contains [ERROR] and the pod name contains pod_name.

      apiVersion: logging.whizard.io/v1alpha1
kind: ClusterRuleGroup
metadata:
  name: logging-rules
spec:
  type: logs
  rules:
    - name: log-test
      expr:
        kind: rule
        condition: log contains "[ERROR]" and kubernetes.pod_name contains "pod_name"
      desc: 'test logging'
      enable: true
      alerts:
        severity: warning

    • Sliding Window Alerting

      The configuration is basically the same as keyword-based alerting, the only difference is the need to add the slidingWindow configuration.

      For example, the following configuration represents a 20s window, triggered every 10s. An alert is triggered when the logs in the window contain [INFO] and the cluster contains "gcp-prod-cluster" and the count is greater than or equal to 20.

      apiVersion: logging.whizard.io/v1alpha1
kind: ClusterRuleGroup
metadata:
  name: logging-rules
spec:
  type: logs
  rules:
    - name: TestLoggingRuleWindow-02
      expr:
        kind: rule
        condition: log contains "[INFO]" and cluster contains "gcp-prod-cluster"
      desc: stream test 2
      enable: true
      alerts:
        severity: info
        annotations:
          summary: for test
          summaryCn: Test TestLoggingRuleWindow-02
        message:  Alert test TestLoggingRuleWindow-02
      slidingWindow:
        windowSize: 20s
        slidingInterval: 10s
        count: 20

  2. Execute the following command to create the CRD (Custom Resource Definition).

    kubectl apply -f xxx.yaml  # Note: Replace with your YAML file name

  3. After successful execution, search for ClusterRuleGroup on the Custom Resource Definitions page of the cluster.

  4. click ClusterRuleGroup to enter its details page, and view, edit, or delete alerting rules on the right-hand page.