用 Kubernetes 资源创建 Grafana Dashboard

作者 |阳明
来源 | k8s技术圈（ID：kube100）

我们在使用 Grafana Dashboard 来展示我们的监控图表的时候，很多时候我们都是去找别人已经做好的 Dashboard 拿过来改一改，但是这样也造成了很多使用 Grafana 的人员压根不知道如何去自定义一个 Dashboard，虽然这并不是很困难。这里我们介绍一个比较新颖（骚）的工具：DARK(https://github.com/K-Phoen/dark)，通过 Kubernetes 的资源对象来定义 Grafana Dashboard，实现原理也很简单，也就是通过 CRD 来定义 Dashboard，然后通过和 Grafana 的 API Token 进行交互实现 Dashboard 的 CRUD。

下面我们来看下如何使用 DARK 定义 Grafana Dashboard。首先 Clone 项目代码：

$ git clone https://github.com/K-Phoen/dark.git

然后安装 CRD 资源：

$ kubectl apply -f k8s/crd.yaml

然后通过 Secret 对象创建 Grafana 的 API KEYS，在 Grafana 主界面中，选择左侧的配置菜单 -> API Keys 创建 API Keys，选择 Editor 的角色：

创建完成后会弹出一个对话框显示对应的 APIKeys，使用这个 KEY 来创建一个对应的 Secret 对象：

$ kubectl create secret generic dark-tokens --from-literal=grafana=<替换成APIKEY>

然后修改 k8s/cluster-role.yaml 文件，如下所示：

apiVersion: v1
kind: ServiceAccount
metadata:
  name: dark
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: dashboards-viewer
rules:
- apiGroups: ["k8s.kevingomez.fr"]
  resources: ["grafanadashboards"]
  verbs: ["get", "watch", "list"]
- apiGroups: [""]
  resources: ["events"]
  verbs: ["create", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: dashboards-viewer-cluster
subjects:
  - kind: ServiceAccount
    name: dark
    namespace: default
roleRef:
  kind: ClusterRole
  name: dashboards-viewer
  apiGroup: rbac.authorization.k8s.io

然后创建上面的资源对象：

$ kubectl apply -f k8s/cluster-role.yaml

修改 k8s/deployment.yaml 文件，将 GRAFANA_HOST 环境变量修改成自己的 Grafana 的地址，由于我这里 Grafana 也安装在 Kubernetes 集群中的，所以直接用 DNS 形式配置，然后加上上面创建的 dark 这个 ServiceAccount：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dark
  labels:
    app: dark
spec:
  selector:
    matchLabels:
      app: dark
  template:
    metadata:
      labels:
        app: dark
    spec:
      volumes:
      - name: dark-tokens
        secret:
          secretName: dark-tokens
      serviceAccountName: dark
      containers:
      - name: dark
        image: kphoen/dark:latest
        env:
        - name: GRAFANA_HOST
          value: http://grafana.kube-mon:3000
        - name: GRAFANA_TOKEN
          valueFrom:
            secretKeyRef:
              key: grafana
              name: dark-tokens

修改完成后直接创建上面的 Controller：

$ kubectl apply -f k8s/deployment.yaml
$ kubectl get pods -l app=dark
NAME                    READY   STATUS    RESTARTS   AGE
dark-6bd956b8d6-755p2   1/1     Running   0          36m

现在 Controller 定义好过后，实际上我们就可以去通过 CRD 对象来定义 Grafana Dashboard 了，如下所示定义了一个 GrafanaDashboard 对象，在对象中我们完全就可以根据自己的需求去定义内容了，比如定义 annotations、 variables、 graph、 table 都可以，当然最重要的还是数据源要正确，以及查询语句：（example-dashboards.yaml）

apiVersion: k8s.kevingomez.fr/v1
kind: GrafanaDashboard
metadata:
  name: example-dashboard
folder: "Test folder"
spec:
  title: Awesome dashboard
  editable: true
  shared_crosshair: true
  tags: [generated, yaml]
  auto_refresh: 10s
  tags_annotations:
    - name: Deployments
      datasource: "Prometheus"
      color: "#5794F2"
      tags: ["deploy", "production"]
  variables:
    - interval:
        name: interval
        label: Interval
        values: ["30s", "1m", "5m", "10m", "30m", "1h", "6h", "12h"]
    - query:
        name: status
        label: HTTP status
        datasource: Prometheus
        request: "label_values(prometheus_http_requests_total, code)"
    - const:
        name: percentile
        label: Percentile
        default: 80
        values_map:
          50th: "50"
          75th: "75"
          80th: "80"
          85th: "85"
          90th: "90"
          95th: "95"
          99th: "99"
    - custom:
        name: vX
        default: v2
        values_map:
          v1: v1
          v2: v2
  rows:
    - name: Prometheus
      panels:
        - graph:
            title: HTTP Rate
            height: 400px
            datasource: Prometheus
            targets:
              - prometheus:
                  query: "rate(promhttp_metric_handler_requests_total[$interval])"
                  legend: "{{handler}} - {{ code }}"
        - graph:
            title: Heap allocations
            height: 400px
            datasource: Prometheus
            targets:
              - prometheus:
                  query: "go_memstats_heap_alloc_bytes"
                  legend: "{{job}}"
                  ref: A
        - table:
            title: Threads
            datasource: Prometheus
            targets:
              - prometheus:
                  query: "go_threads"
            hidden_columns: ["Time"]
            time_series_aggregations:
              - label: AVG
                type: avg
              - label: Current
                type: current
        - single_stat:
            title: Heap Allocations
            datasource: Prometheus
            targets:
              - prometheus:
                  query: 'go_memstats_heap_alloc_bytes{job="prometheus"}'
            unit: bytes
            thresholds: ["26000000", "28000000"]
            color: ["value"]
    - name: "Some text, because it might be useful"
      panels:
        - text:
            title: Some awesome text?
            markdown: "Markdown syntax help: [commonmark.org/help](https://commonmark.org/help/)\n${percentile}"
        - text:
            title: Some awesome html?
            html: "Some <b>awesome</b> html?"

同样直接创建上面的示例文件：

$ kubectl apply -f example-dashboards.yaml
$ kubectl get dashboards
NAME                AGE
example-dashboard   35m
$ kubectl logs -f dark-6bd956b8d6-755p2
W0327 11:10:24.356194       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0327 11:10:24.360886       1 controller.go:87] Setting up event handlers
I0327 11:10:24.362305       1 controller.go:118] Starting dark-controller
I0327 11:10:24.362341       1 controller.go:121] Waiting for informer caches to sync
I0327 11:10:24.462733       1 controller.go:126] Starting workers
I0327 11:10:24.462820       1 controller.go:132] Started workers
I0327 11:13:22.641706       1 controller.go:197] Successfully synced 'default/example-dashboard'
I0327 11:13:22.643061       1 event.go:278] Event(v1.ObjectReference{Kind:"GrafanaDashboard", Namespace:"default", Name:"example-dashboard", UID:"efc6f96f-c7fc-40b5-8b8f-831a95b0a042", APIVersion:"k8s.kevingomez.fr/v1", ResourceVersion:"48490732", FieldPath:""}): type: 'Normal' reason: 'Synced' GrafanaDashboard synced successfully

在 Controller 中也可以看到对应的日志信息，资源对象创建成功以后，现在去 Grafana 页面上查看可以看到已经新增了一个 Test folder 的文件夹以及 Awesome dashboard：

查看 Dashboard 就可以看到和上面 CRD 中定义的各种图表信息了：

这样我们就使用 Kubernetes 资源对象去定义了 Grafana Dashboard 了，这种方式比直接在页面上去手动配置显然要更优雅，也符合 Everything AS Code 的思想🤯。