在前一陣子 EDB 曾經在 K8s/Openshift 平台還沒變成壓倒性主流容器平台方案時，就已經提供 Docker Container 的資料庫解決方案，當時也做了一點點練習（筆記在這）。
不過當時得努力的搞清楚最原生的 K8s YAML 設置，其實很複雜。。。
隨著 K8s 整體的推進，出現了 Helm 與 Operator 這類「套件管理」的工具，加上 K8s 強化 Stateful Container 的支援，資料庫陸陸續續也可以輕易登陸 K8s 平台。
EDB 團隊基於這些變化，加上原 2ndQ 團隊的一同合作，便推出這個成熟的 K8s 解決方案。
這邊就要來初步體驗一下～
- 原生版 PostgreSQL 與企業版 EDB Postgres Advanced Server 資料庫
- 基於 K8s 調度機制的資料庫高可用功能
- 支援 S3 協定的資料庫全備份、WAL 備份
- 自動設置資料傳輸安全性
- 滾動式小版本更新支援、K8s 底層維護支援
- K8s 節點磁碟壓測工具 cnp-bench
這邊就使用好心的 K8s 測試環境 KinD 作簡易的三節點主從 PGSQL 資料庫練習，並且偷偷潛入容器裡面喵一下。
以下會進行的步驟大致如下
- 操作預備好的 K8s in Docker 三節點環境
- 安裝 CNP Operator：有這個才能佈署 EDB 資料庫解決方案
- 設置 1 Primary/2 Standby 的 PGSQL 資料庫
- 偷偷溜進去 Container 觀察一下
- 觸發簡單的高可用切換
以上的第一步就是沿用上一篇筆記環境～
現在要安裝 CNP Operator：這需要利用 kubectl apply 指令佈署 Operator，利用 -f 指定 YAML 檔案。這邊直接參照以下頁面操作。
EDB Docs - Installation and upgrades
EDB Docs - Installation, Configuration and Deployment Demo
[goodgame@k8sindocker ~]$ curl https://get.enterprisedb.io/cnp/postgresql-operator-1.7.1.yaml -o postgresql-operator-1.7.1.yaml
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 218k 100 218k 0 0 221k 0 --:--:-- --:--:-- --:--:-- 220k
[goodgame@k8sindocker ~]$ kubectl apply -f postgresql-operator-1.7.1.yaml
namespace/postgresql-operator-system created
customresourcedefinition.apiextensions.k8s.io/backups.postgresql.k8s.enterprisedb.io created
customresourcedefinition.apiextensions.k8s.io/clusters.postgresql.k8s.enterprisedb.io created
customresourcedefinition.apiextensions.k8s.io/scheduledbackups.postgresql.k8s.enterprisedb.io created
Warning: admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
mutatingwebhookconfiguration.admissionregistration.k8s.io/postgresql-operator-mutating-webhook-configuration created
serviceaccount/postgresql-operator-manager created
clusterrole.rbac.authorization.k8s.io/postgresql-operator-manager created
clusterrolebinding.rbac.authorization.k8s.io/postgresql-operator-manager-rolebinding created
service/postgresql-operator-webhook-service created
deployment.apps/postgresql-operator-controller-manager created
Warning: admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
validatingwebhookconfiguration.admissionregistration.k8s.io/postgresql-operator-validating-webhook-configuration created
[goodgame@k8sindocker ~]$
[goodgame@k8sindocker ~]$ kubectl get deploy -n postgresql-operator-system postgresql-operator-controller-manager NAME READY UP-TO-DATE AVAILABLE AGE postgresql-operator-controller-manager 1/1 1 1 72s [goodgame@k8sindocker ~]$ [goodgame@k8sindocker ~]$ kubectl get pods -n postgresql-operator-system NAME READY STATUS RESTARTS AGE postgresql-operator-controller-manager-684bb9c549-5rjbh 1/1 Running 0 83s [goodgame@k8sindocker ~]$
CNP Operator 安裝完成之後，就可以佈署 DB 叢集了。相關的叢集範例可以參照 Configuration Samples - Cloud Native PostgreSQL（EDB Docs - Configuration Samples）。基本上都是高可用配置的範本。單節點不需要用 Operator。
資料庫初始化選項可以參考 EDB Docs - Bootstrap，這邊直接選用下面這個範本
https://docs.enterprisedb.io/cloud-native-postgresql/1.7.1/samples/cluster-example.yaml
執行後就一邊觀察 pod 狀態。這邊顯示一開始跟完成後的狀況：可以撘配 watch 觀察會更清楚～
[goodgame@k8sindocker ~]$ curl https://docs.enterprisedb.io/cloud-native-postgresql/1.7.1/samples/cluster-example.yaml
# Example of PostgreSQL cluster
apiVersion: postgresql.k8s.enterprisedb.io/v1
kind: Cluster
metadata:
name: cluster-example
spec:
instances: 3
# Require 1Gi of space
storage:
size: 1Gi
[goodgame@k8sindocker ~]$
[goodgame@k8sindocker ~]$ curl https://docs.enterprisedb.io/cloud-native-postgresql/1.7.1/samples/cluster-example.yaml -o cluster-example.yaml
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 141 100 141 0 0 173 0 --:--:-- --:--:-- --:--:-- 173
[goodgame@k8sindocker ~]$
[goodgame@k8sindocker ~]$ kubectl apply -f cluster-example.yaml
cluster.postgresql.k8s.enterprisedb.io/cluster-example created
[goodgame@k8sindocker ~]$
[goodgame@k8sindocker ~]$ # 馬上觀察 [goodgame@k8sindocker ~]$ kubectl get pods NAME READY STATUS RESTARTS AGE cluster-example-1-initdb-nvhp8 0/1 PodInitializing 0 19s [goodgame@k8sindocker ~]$
[goodgame@k8sindocker ~]$ # 中間過程 [goodgame@k8sindocker ~]$ kubectl get pods NAME READY STATUS RESTARTS AGE cluster-example-1 1/1 Running 0 2m6s cluster-example-1-initdb-nvhp8 0/1 Completed 0 3m37s cluster-example-2 1/1 Running 0 8s cluster-example-2-join-8xn92 0/1 Completed 0 2m4s cluster-example-3-join-lnq24 0/1 Pending 0 5s [goodgame@k8sindocker ~]$
[goodgame@k8sindocker ~]$ #叢集完成 [goodgame@k8sindocker ~]$ kubectl get pods NAME READY STATUS RESTARTS AGE cluster-example-1 1/1 Running 0 2m22s cluster-example-2 1/1 Running 0 24s cluster-example-3 1/1 Running 0 12s [goodgame@k8sindocker ~]$
等到完成後，就可以用 kubectl 原生指令查看資料庫叢集資訊
[goodgame@k8sindocker ~]$ kubectl get cluster cluster-example -o yaml
apiVersion: postgresql.k8s.enterprisedb.io/v1
kind: Cluster
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"postgresql.k8s.enterprisedb.io/v1","kind":"Cluster","metadata":{"annotations":{},"name":"cluster-example","namespace":"default"},"spec":{"instances":3,"storage":{"size":"1Gi"}}}
creationTimestamp: "2021-09-10T10:30:29Z"
generation: 1
name: cluster-example
namespace: default
resourceVersion: "8495"
uid: 465XXX96-9e2c-4O8f-8OO0-7b6XXXX28a2e
spec:
affinity:
podAntiAffinityType: preferred
topologyKey: ""
bootstrap:
initdb:
database: app
owner: app
imageName: quay.io/enterprisedb/postgresql:13.3
instances: 3
postgresql:
parameters:
log_destination: csvlog
log_directory: /controller/log
log_filename: postgres
log_rotation_age: "0"
log_rotation_size: "0"
log_truncate_on_rotation: "false"
logging_collector: "on"
max_parallel_workers: "32"
max_replication_slots: "32"
max_worker_processes: "32"
shared_preload_libraries: ""
wal_keep_size: 512MB
resources: {}
storage:
size: 1Gi
status:
certificates:
clientCASecret: cluster-example-ca
expirations:
cluster-example-ca: 2021-12-09 10:25:29 +0000 UTC
cluster-example-replication: 2021-12-09 10:25:29 +0000 UTC
cluster-example-server: 2021-12-09 10:25:29 +0000 UTC
replicationTLSSecret: cluster-example-replication
serverAltDNSNames:
- cluster-example-rw
- cluster-example-rw.default
- cluster-example-rw.default.svc
- cluster-example-r
- cluster-example-r.default
- cluster-example-r.default.svc
- cluster-example-ro
- cluster-example-ro.default
- cluster-example-ro.default.svc
serverCASecret: cluster-example-ca
serverTLSSecret: cluster-example-server
configMapResourceVersion: {}
currentPrimary: cluster-example-1
healthyPVC:
- cluster-example-1
- cluster-example-2
- cluster-example-3
instances: 3
instancesStatus:
healthy:
- cluster-example-1
- cluster-example-2
- cluster-example-3
latestGeneratedNode: 3
licenseStatus:
isImplicit: true
isTrial: true
licenseExpiration: "2021-10-10T10:30:29Z"
licenseStatus: Implicit trial license
repositoryAccess: false
valid: true
phase: Cluster in healthy state
pvcCount: 3
readService: cluster-example-r
readyInstances: 3
secretsResourceVersion:
applicationSecretVersion: "6616"
clientCaSecretVersion: "6612"
replicationSecretVersion: "6614"
serverCaSecretVersion: "6612"
serverSecretVersion: "6613"
superuserSecretVersion: "6615"
targetPrimary: cluster-example-1
writeService: cluster-example-rw
[goodgame@k8sindocker ~]$
觀察一下 service
[goodgame@k8sindocker ~]$ kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE cluster-example-any ClusterIP 10.96.194.146 <none> 5432/TCP 5h50m cluster-example-r ClusterIP 10.96.173.199 <none> 5432/TCP 5h50m cluster-example-ro ClusterIP 10.96.238.83 <none> 5432/TCP 5h50m cluster-example-rw ClusterIP 10.96.24.26 <none> 5432/TCP 5h50m kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 6h32m [goodgame@k8sindocker ~]$
以上就完成資料庫叢集的設置了。
不過，為了接觸這組服務，EDB K8s operator 有附一個管理 CNP 資料庫叢集的 kubectl 外掛 kubectl cnp，這邊就把這裝起來。
[goodgame@k8sindocker ~]$ curl -sSfL https://github.com/EnterpriseDB/kubectl-cnp/raw/main/install.sh -o install.sh [goodgame@k8sindocker ~]$ [goodgame@k8sindocker ~]$ chmod +x install.sh [goodgame@k8sindocker ~]$ sudo ./install.sh -b /usr/local/bin EnterpriseDB/kubectl-cnp info checking GitHub for latest tag EnterpriseDB/kubectl-cnp info found version: 1.7.1 for v1.7.1/linux/x86_64 EnterpriseDB/kubectl-cnp info installed /usr/local/bin/kubectl-cnp [goodgame@k8sindocker ~]$
查看資料庫叢集狀態
[goodgame@k8sindocker ~]$ kubectl cnp status cluster-example Cluster in healthy state Name: cluster-example Namespace: default PostgreSQL Image: quay.io/enterprisedb/postgresql:13.3 Primary instance: cluster-example-1 Instances: 3 Ready instances: 3 Instances status Pod name Current LSN Received LSN Replay LSN System ID Primary Replicating Replay paused Pending restart Status -------- ----------- ------------ ---------- --------- ------- ----------- ------------- --------------- ------ cluster-example-1 0/5000060 7006250831794221074 ✓ ✗ ✗ ✗ OK cluster-example-2 0/5000060 0/5000060 7006250831794221074 ✗ ✓ ✗ ✗ OK cluster-example-3 0/5000060 0/5000060 7006250831794221074 ✗ ✓ ✗ ✗ OK [goodgame@k8sindocker ~]$
來看一下這個 plugin 提供的功能。
[goodgame@k8sindocker ~]$ kubectl cnp --help
A plugin to manage your Cloud Native PostgreSQL clusters
Usage:
kubectl-cnp [command]
Available Commands:
certificate Create a client certificate to connect to PostgreSQL using TLS and Certificate authentication
completion generate the autocompletion script for the specified shell
help Help about any command
promote Promote the pod named [cluster]-[node] or [node] to primary
reload Reload the cluster
restart Restart the cluster
status Get the status of a PostgreSQL cluster
version Prints version, commit sha and date of the build
Flags:
--as string Username to impersonate for the operation
--as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.
--cache-dir string Default cache directory (default "/home/goodgame/.kube/cache")
--certificate-authority string Path to a cert file for the certificate authority
--client-certificate string Path to a client certificate file for TLS
--client-key string Path to a client key file for TLS
--cluster string The name of the kubeconfig cluster to use
--context string The name of the kubeconfig context to use
-h, --help help for kubectl-cnp
--insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure
--kubeconfig string Path to the kubeconfig file to use for CLI requests.
-n, --namespace string If present, the namespace scope for this CLI request
--request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0")
-s, --server string The address and port of the Kubernetes API server
--tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used
--token string Bearer token for authentication to the API server
--user string The name of the kubeconfig user to use
Use "kubectl-cnp [command] --help" for more information about a command.
[goodgame@k8sindocker ~]$
以上從 K8s 觀察資料庫同步資訊之後，然後再來是想辦法進去資料庫裡面，檢驗一下同步。進入 Container 的方法在這 Get a Shell to a Running Container | Kubernetes。
不過值得一提的是，除了資料目錄（PGDATA）之外，Container 裡面是唯讀的 OS 環境。
[goodgame@k8sindocker ~]$ kubectl exec --stdin --tty cluster-example-1 -- /bin/bash
Defaulted container "postgres" out of: postgres, bootstrap-controller (init)
bash-4.4$ hostname
cluster-example-1
bash-4.4$
bash-4.4$ psql
psql (13.3)
Type "help" for help.
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+-----------+---------+-------+-----------------------
app | app | SQL_ASCII | C | C |
postgres | postgres | SQL_ASCII | C | C |
template0 | postgres | SQL_ASCII | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | SQL_ASCII | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
postgres=# select * from pg_stat_replication ;
pid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | backend_xmin | state | sent_lsn | write_lsn | flush_lsn | replay_lsn | write_lag | flush_lag | replay_lag | sync_priority | sync_state | reply_time
-----+----------+-------------------+-------------------+-------------+-----------------+-------------+-------------------------------+--------------+-----------+-----------+-----------+-----------+------------+-----------+-----------+------------+---------------+------------+-------------------------------
201 | 16386 | streaming_replica | cluster-example-2 | 10.244.1.5 | | 49320 | 2021-09-10 10:34:02.023046+00 | | streaming | 0/8000000 | 0/8000000 | 0/8000000 | 0/8000000 | | | | 0 | async | 2021-09-10 13:11:19.97608+00
343 | 16386 | streaming_replica | cluster-example-3 | 10.244.1.8 | | 44016 | 2021-09-10 10:34:14.565879+00 | | streaming | 0/8000000 | 0/8000000 | 0/8000000 | 0/8000000 | | | | 0 | async | 2021-09-10 13:11:20.267473+00
(2 rows)
postgres=#
postgres=# select * from pg_hba_file_rules ;
line_number | type | database | user_name | address | netmask | auth_method | options | error
-------------+---------+---------------+---------------------+---------+---------+-------------+------------------------+-------
2 | local | {all} | {all} | | | peer | {map=local} |
5 | hostssl | {postgres} | {streaming_replica} | all | | cert | {clientcert=verify-ca} |
6 | hostssl | {replication} | {streaming_replica} | all | | cert | {clientcert=verify-ca} |
9 | host | {all} | {all} | all | | md5 | |
(4 rows)
postgres=# \q
bash-4.4$ ls /var/lib/pgsql/13/data/
bash-4.4$ ls /var/lib/postgresql/data/pgdata/
PG_VERSION pg_commit_ts pg_notify pg_subtrans postgresql.conf
base pg_dynshmem pg_replslot pg_tblspc postmaster.opts
current_logfiles pg_hba.conf pg_serial pg_twophase postmaster.pid
custom.conf pg_ident.conf pg_snapshots pg_wal
global pg_logical pg_stat pg_xact
log pg_multixact pg_stat_tmp postgresql.auto.conf
bash-4.4$
bash-4.4$ cat /var/lib/postgresql/data/pgdata/custom.conf
archive_command = '/controller/manager wal-archive %p'
archive_mode = 'on'
archive_timeout = '5min'
cluster_name = 'cluster-example'
full_page_writes = 'on'
hot_standby = 'true'
listen_addresses = '*'
log_destination = 'csvlog'
log_directory = '/controller/log'
log_filename = 'postgres'
log_rotation_age = '0'
log_rotation_size = '0'
log_truncate_on_rotation = 'false'
logging_collector = 'on'
max_parallel_workers = '32'
max_replication_slots = '32'
max_worker_processes = '32'
port = '5432'
shared_preload_libraries = ''
ssl = 'on'
ssl_ca_file = '/controller/certificates/client-ca.crt'
ssl_cert_file = '/controller/certificates/server.crt'
ssl_key_file = '/controller/certificates/server.key'
unix_socket_directories = '/controller/run'
wal_keep_size = '512MB'
wal_level = 'logical'
wal_log_hints = 'on'
cnp.config_sha256 = '058ae20f8aa96974535cbf449X1X4XcXabaXb74e6Xe0XXcb81a85Xe77d34197f'
bash-4.4$
[goodgame@k8sindocker ~]$ kubectl exec --stdin --tty cluster-example-2 -- /bin/bash
Defaulted container "postgres" out of: postgres, bootstrap-controller (init)
bash-4.4$ hostname
cluster-example-2
bash-4.4$ psql -c 'select pg_is_in_recovery();' -c 'show restore_command ;'
pg_is_in_recovery
-------------------
t
(1 row)
restore_command
---------------------------------------
/controller/manager wal-restore %f %p
(1 row)
bash-4.4$
最後，作一個簡易高可用切換測試：這邊進入 primary DB 的 container 執行 pg_ctl stop 指令後，在回到外面觀察 CNP 叢集狀態
[brandon_hsu@k8sindocker ~]$ kubectl exec --stdin --tty cluster-example-1 -- /bin/bash Defaulted container "postgres" out of: postgres, bootstrap-controller (init) bash-4.4$ pg_ctl stop waiting for server to shut down....command terminated with exit code 137 [brandon_hsu@k8sindocker ~]$
[brandon_hsu@k8sindocker ~]$ kubectl cnp status cluster-example Cluster in healthy state Name: cluster-example Namespace: default PostgreSQL Image: quay.io/enterprisedb/postgresql:13.3 Primary instance: cluster-example-2 Instances: 3 Ready instances: 3 Instances status Pod name Current LSN Received LSN Replay LSN System ID Primary Replicating Replay paused Pending restart Status -------- ----------- ------------ ---------- --------- ------- ----------- ------------- --------------- ------ cluster-example-1 0/8004958 0/8004958 7006250831794221074 ✗ ✓ ✗ ✗ OK cluster-example-2 0/8004958 7006250831794221074 ✓ ✗ ✗ ✗ OK cluster-example-3 0/8004958 0/8004958 7006250831794221074 ✗ ✓ ✗ ✗ OK [brandon_hsu@k8sindocker ~]$
以上觀察到，Primary 切換到第二個 pod，而第一個 pod 也自動轉換為 standby DB 復活了～
以上簡單測試就到這結束～
參考資料
都附在上面惹～
沒有留言:
張貼留言