F5 BIG-IP CIS 與 NGINX IC 強化 OpenShift 入口控制能力 (Part.2)
測試環境 OpenShift 4.8 / F5 CIS 2.7.1 / NGINX IC 2.1.0
Introduce
本文將介紹如何在 OpenShift 環境中搭配使用 F5 相關的解決方案,其中包括 F5 BIG-IP CIS 、F5 IPAM 和 NGINX IC 的部屬和應用。此外,還需要注意與 OpenShift 整合所需的相關事項,可以參考之前分享的『F5 BIG-IP CIS 與 NGINX IC 強化 OpenShift 入口控制能力 (Part.1)』內容。至於 OpenShift 集群的安裝細節,可以參考紅帽官方文件。
Architecture
F5 BIG-IP CIS
- 負載流量至 NGINX IC Pod (Host Base)
- 動態服務發現 NGINX IC Pod,並自動更新相關設定
- 使用 F5 IPAM 實現自動獲取 VIP
- 自動發布服務至 F5 BIG-IP (DNS Record)
- 套用 WAF 安全策略
NGINX IC
- 負載流量至應用程式 Pod (Path Base)
- 訪問請求頻率限制
Config F5 BIG-IP
VXLAN Tunnel
-
建立 VXALN 隧道,注意 VNI 需使用 4097 與 OpenShift 溝通
-
設定 VXLAN 隧道 Over Layer Self IP,請確保網段範圍能夠覆蓋 OpenShift 集群的網路,以便讓 F5 BIG-IP 可以直接路由到 OpenShift
Create Partition
- 建立一個 Partition 供 F5 CIS 使用,注意名稱不要使用大寫
Install AS3
-
下載 AS3 RPM 安裝檔 (F5 Networks GitHub)
-
到 F5 BIG-IP 安裝 AS3
Deploy F5 BIG-IP Container Ingress Services (CIS)
Create CIS Login Account
替換成自己的 BIG-IP 設備管理帳號和密碼
oc create secret generic bigip-login --namespace kube-system --from-literal=username=<BIG-IP管理帳號> --from-literal=password=<BIG-IP管理密碼>
Create Service Account / Cluster Role / Cluster Role Binding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: bigip-ctlr-clusterrole
rules:
- apiGroups: ["", "extensions", "networking.k8s.io"]
resources: ["nodes", "services", "endpoints", "namespaces", "ingresses", "pods", "ingressclasses", "policies"]
verbs: ["get", "list", "watch"]
- apiGroups: ["", "extensions", "networking.k8s.io"]
resources: ["configmaps", "events", "ingresses/status", "services/status"]
verbs: ["get", "list", "watch", "update", "create", "patch"]
- apiGroups: ["cis.f5.com"]
resources: ["virtualservers","virtualservers/status", "tlsprofiles", "transportservers", "transportservers/status", "ingresslinks", "ingresslinks/status", "externaldnses", "policies"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["fic.f5.com"]
resources: ["ipams", "ipams/status"]
verbs: ["get", "list", "watch", "update", "create", "patch", "delete"]
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: ["get", "list", "watch", "update", "create", "patch"]
- apiGroups: ["", "extensions"]
resources: ["secrets"]
verbs: ["get", "list", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: bigip-ctlr-clusterrole-binding
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: bigip-ctlr-clusterrole
subjects:
- apiGroup: ""
kind: ServiceAccount
name: bigip-ctlr
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: bigip-ctlr
namespace: kube-system
注意需設定 bigip-ctlr service account 集群管理員權限
[root@bastion01 cis]# oc create -f sa_clusterrole.yaml
clusterrole.rbac.authorization.k8s.io/bigip-ctlr-clusterrole created
clusterrolebinding.rbac.authorization.k8s.io/bigip-ctlr-clusterrole-binding created
serviceaccount/bigip-ctlr created
[root@bastion01 cis]# oc adm policy add-cluster-role-to-user cluster-admin -z bigip-ctlr -n kube-system
clusterrole.rbac.authorization.k8s.io/cluster-admin added: "bigip-ctlr"
Create Custom Resource Definitions
[root@bastion01 cis]# oc create -f customresourcedefinitions.yml
customresourcedefinition.apiextensions.k8s.io/virtualservers.cis.f5.com created
customresourcedefinition.apiextensions.k8s.io/tlsprofiles.cis.f5.com created
customresourcedefinition.apiextensions.k8s.io/transportservers.cis.f5.com created
customresourcedefinition.apiextensions.k8s.io/externaldnses.cis.f5.com created
customresourcedefinition.apiextensions.k8s.io/ingresslinks.cis.f5.com created
customresourcedefinition.apiextensions.k8s.io/policies.cis.f5.com created
Create Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: k8s-bigip-ctlr-deployment
namespace: kube-system
spec:
# DO NOT INCREASE REPLICA COUNT
replicas: 1
selector:
matchLabels:
app: k8s-bigip-ctlr-deployment
template:
metadata:
labels:
app: k8s-bigip-ctlr-deployment
spec:
# Name of the Service Account bound to a Cluster Role with the required
# permissions
containers:
- name: k8s-bigip-ctlr
image: "f5networks/k8s-bigip-ctlr:2.8.0"
env:
- name: BIGIP_USERNAME
valueFrom:
secretKeyRef:
# Replace with the name of the Secret containing your login
# credentials
name: bigip-login
key: username
- name: BIGIP_PASSWORD
valueFrom:
secretKeyRef:
# Replace with the name of the Secret containing your login
# credentials
name: bigip-login
key: password
command: ["/app/bin/k8s-bigip-ctlr"]
args: [
# See the k8s-bigip-ctlr documentation for information about
# all config options
# https://clouddocs.f5.com/containers/latest/
"--bigip-username=$(BIGIP_USERNAME)",
"--bigip-password=$(BIGIP_PASSWORD)",
"--bigip-url=192.168.102.98", ### F5 管理 IP
"--bigip-partition=ocp", ### CIS 使用的 Partition 名稱
"--pool-member-type=cluster",
"--openshift-sdn-name=/Common/ocp-vxlan-tunnel", ### F5 BIG-IP VXLAN Tunnel 名稱
"--log-level=DEBUG",
"--insecure=true",
"--custom-resource-mode=true", ### 啟用 CRD 設定
"--as3-validation=true",
"--log-as3-response=true",
"--ipam=true", ### 啟用 IPAM 支援
"--namespace-label=cis=true", ### 指定監聽的 Namespace Label
"--disable-teems=true",
]
serviceAccountName: bigip-ctlr
[root@bastion01 cis]# oc create -f deployment.yaml
deployment.apps/k8s-bigip-ctlr-deployment created
Deploy F5 IPAM
Create Service Account / Cluster Role / Cluster Role Binding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: ipam-ctlr-clusterrole
rules:
- apiGroups: ["fic.f5.com"]
resources: ["ipams","ipams/status"]
verbs: ["get", "list", "watch", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: ipam-ctlr-clusterrole-binding
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ipam-ctlr-clusterrole
subjects:
- apiGroup: ""
kind: ServiceAccount
name: ipam-ctlr
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: ipam-ctlr
namespace: kube-system
[root@bastion01 ipam]# oc create -f sa_clusterrole.yaml
clusterrole.rbac.authorization.k8s.io/ipam-ctlr-clusterrole created
clusterrolebinding.rbac.authorization.k8s.io/ipam-ctlr-clusterrole-binding created
serviceaccount/ipam-ctlr created
Create Volume
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: local-pv
spec:
capacity:
storage: 1Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
storageClassName: local-storage
local:
path: /tmp/cis_ipam
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- worker01
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-local
namespace: kube-system
spec:
storageClassName: local-storage
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 0.1Gi
[root@bastion01 ipam]# oc create -f pv.yaml
storageclass.storage.k8s.io/local-storage created
persistentvolume/local-pv created
persistentvolumeclaim/pvc-local created
Create Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
name: f5-ipam-controller
name: f5-ipam-controller
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app: f5-ipam-controller
template:
metadata:
labels:
app: f5-ipam-controller
spec:
containers:
- args:
- --orchestration=openshift
- --ip-range='{"test":"172.16.90.180-172.16.90.185"}' ### 要派發的 IP 及其名稱
- --log-level=DEBUG
command:
- /app/bin/f5-ipam-controller
image: f5networks/f5-ipam-controller:0.1.6
imagePullPolicy: IfNotPresent
name: f5-ipam-controller
terminationMessagePath: /dev/termination-log
volumeMounts:
- mountPath: /app/ipamdb
name: local-pv
securityContext:
fsGroup: 1200
runAsGroup: 1200
runAsUser: 1200
serviceAccount: ipam-ctlr
serviceAccountName: ipam-ctlr
volumes:
- name: local-pv
persistentVolumeClaim:
claimName: pvc-local
[root@bastion01 ipam]# oc create -f deployment.yaml
deployment.apps/f5-ipam-controller created
Deploy NGINX IC
Create Namespace / Service Account
apiVersion: v1
kind: Namespace
metadata:
name: nginx-ingress
labels:
cis: "true" ### 標記讓 F5 BIG-IP CIS 知道需要監聽這個 Namespace
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress
namespace: nginx-ingress
[root@bastion01 nginx]# oc create -f ns-and-sa.yaml
namespace/nginx-ingress created
serviceaccount/nginx-ingress created
Create Cluster Role / Cluster Role Binding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: nginx-ingress
rules:
- apiGroups:
- ""
resources:
- services
- endpoints
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
- update
- create
- apiGroups:
- ""
resources:
- pods
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- list
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- list
- watch
- get
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- k8s.nginx.org
resources:
- virtualservers
- virtualserverroutes
- globalconfigurations
- transportservers
- policies
verbs:
- list
- watch
- get
- apiGroups:
- k8s.nginx.org
resources:
- virtualservers/status
- virtualserverroutes/status
- policies/status
- transportservers/status
verbs:
- update
- apiGroups:
- networking.k8s.io
resources:
- ingressclasses
verbs:
- get
- apiGroups:
- cis.f5.com
resources:
- ingresslinks
verbs:
- list
- watch
- get
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nginx-ingress
subjects:
- kind: ServiceAccount
name: nginx-ingress
namespace: nginx-ingress
roleRef:
kind: ClusterRole
name: nginx-ingress
apiGroup: rbac.authorization.k8s.io
[root@bastion01 nginx]# oc create -f ns-and-sa.yaml
namespace/nginx-ingress created
serviceaccount/nginx-ingress created
Create Ingress Class
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
name: nginx
annotations:
ingressclass.kubernetes.io/is-default-class: "true"
spec:
controller: nginx.org/ingress-controller
[root@bastion01 nginx]# oc create -f ingress-class.yaml
ingressclass.networking.k8s.io/nginx created
Create Default Server Certificate
apiVersion: v1
kind: Secret
metadata:
name: default-server-secret
namespace: nginx-ingress
type: kubernetes.io/tls
data:
tls.crt: ### 你的憑證
tls.key: ### 你的金鑰
[root@bastion01 nginx]# oc create -f default-server-secret.yaml
secret/default-server-secret created
Create Service
apiVersion: v1
kind: Service
metadata:
name: nginx-ingress
namespace: nginx-ingress
spec:
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
- port: 443
targetPort: 443
protocol: TCP
name: https
selector:
app: nginx-ingress
[root@bastion01 nginx]# oc create -f nginx-kic-svc.yaml
service/nginx-ingress created
Create Config Map
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-config
namespace: nginx-ingress
data:
lb-method: "round_robin"
[root@bastion01 nginx]# oc create -f nginx-config.yaml
configmap/nginx-config created
Create Custom Resource Definitions
[root@bastion01 nginx]# oc create -f crd/
customresourcedefinition.apiextensions.k8s.io/aplogconfs.appprotect.f5.com created
customresourcedefinition.apiextensions.k8s.io/appolicies.appprotect.f5.com created
customresourcedefinition.apiextensions.k8s.io/apusersigs.appprotect.f5.com created
customresourcedefinition.apiextensions.k8s.io/globalconfigurations.k8s.nginx.org created
customresourcedefinition.apiextensions.k8s.io/policies.k8s.nginx.org created
customresourcedefinition.apiextensions.k8s.io/transportservers.k8s.nginx.org created
customresourcedefinition.apiextensions.k8s.io/virtualserverroutes.k8s.nginx.org created
customresourcedefinition.apiextensions.k8s.io/virtualservers.k8s.nginx.org created
Create Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-ingress
namespace: nginx-ingress
spec:
replicas: 1
selector:
matchLabels:
app: nginx-ingress
template:
metadata:
labels:
app: nginx-ingress
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9113"
spec:
serviceAccountName: nginx-ingress
containers:
- image: nginx-plus:2.1.0
imagePullPolicy: IfNotPresent
name: nginx-plus-ingress
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
- name: readiness-port
containerPort: 8081
- name: dashboard
containerPort: 9000
- name: prometheus
containerPort: 9113
# readinessProbe:
# httpGet:
# path: /nginx-ready
# port: readiness-port
# periodSeconds: 1
securityContext:
allowPrivilegeEscalation: true
runAsUser: 101 #nginx
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
args:
- -nginx-plus
- -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
- -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret
- -nginx-status-port=9000
- -nginx-status-allow-cidrs=0.0.0.0/0
- -enable-snippets
#- -enable-app-protect
#- -v=3 # Enables extensive logging. Useful for troubleshooting.
#- -report-ingress-status
#- -external-service=nginx-ingress
- -enable-prometheus-metrics
- -enable-latency-metrics
#- -global-configuration=$(POD_NAMESPACE)/nginx-configuration
#- -ingresslink=nginx-ingress
#- -report-ingress-status
- -enable-preview-policies
[root@bastion01 nginx]# oc create -f nginx-plus-ingress.yaml
deployment.apps/nginx-ingress created
Expose Application
Application : Coffee & Tea
[root@bastion01 app]# oc get pods,svc
NAME READY STATUS RESTARTS AGE
pod/coffee-9759f989b-6dpcj 1/1 Running 0 7d5h
pod/coffee-9759f989b-8jd9t 1/1 Running 0 7d5h
pod/coffee-9759f989b-cdrxr 1/1 Running 0 7d5h
pod/tea-7464cc487b-7cthd 1/1 Running 0 7d5h
pod/tea-7464cc487b-vr5vz 1/1 Running 0 7d5h
pod/tea-7464cc487b-z8z2d 1/1 Running 0 7d5h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/coffee-svc ClusterIP 172.30.141.25 <none> 80/TCP 26d
service/tea-svc ClusterIP 172.30.90.69 <none> 80/TCP 26d
F5 CIS CR (Cluster Operator)
TLS Profile: 定義要使用的 SSL Profile
apiVersion: cis.f5.com/v1
kind: TLSProfile
metadata:
name: reencrypt-tls
namespace: nginx-ingress
labels:
f5cr: "true"
spec:
tls:
termination: reencrypt
clientSSL: /Common/clientssl ### Client SSL Profile
serverSSL: /Common/serverssl ### Server SSL Profile
reference: bigip
hosts:
- cafe.example.com
[root@bastion01 cr]# oc create -f tls.yaml
tlsprofile.cis.f5.com/reencrypt-tls created
Policy: 定義要使用的 WAF 策略和設定 Persistence
apiVersion: cis.f5.com/v1
kind: Policy
metadata:
labels:
f5cr: "true"
name: cafe-policy
namespace: nginx-ingress
spec:
l7Policies:
waf: /Common/cafe_sp ### 預先建立好的 WAF Policy
profiles:
persistenceProfile: none ### 不做 Persistence
logProfiles:
- /Common/Log all requests ### WAF Log Profile
[root@bastion01 cr]# oc create -f policy_f5.yaml
policy.cis.f5.com/cafe-policy created
Virtual Server: 基於主機名稱的方式來發布服務
apiVersion: "cis.f5.com/v1"
kind: VirtualServer
metadata:
name: cafe-vs
namespace: nginx-ingress
labels:
f5cr: "true"
spec:
host: cafe.example.com
virtualServerHTTPSPort: 443
virtualServerName: cafe_demo_vs
ipamLabel: test ### IPAM Pool
tlsProfileName: reencrypt-tls ### TLS Profile
policyName: cafe-policy ### Policy
snat: auto
allowVlans: ["/Common/vlan_90"]
pools:
- monitor:
interval: 15
recv: ""
send: "GET /coffee HTTP/1.1\r\nHost: cafe.example.com\r\n"
timeout: 46
type: https
service: nginx-ingress
servicePort: 443
[root@bastion01 cr]# oc create -f vs_f5.yaml
virtualserver.cis.f5.com/cafe-vs created
ExternalDNS: 將服務的 DNS 紀錄新增至 F5 BIG-IP DNS 智能解析 (GSLB)
在上一步驟中建立 Virtual Server 時,已經配置了對後端進行健康狀態監控的設定(使用 HTTP Monitor)。因此,不需要進一步設定額外的 DNS 監控,只需使用預設的 BIG-IP 監控(透過 I-Query)來獲取 Virtual Server 的狀態即可。
apiVersion: "cis.f5.com/v1"
kind: ExternalDNS
metadata:
name: cafe-edns
namespace: nginx-ingress
labels:
f5cr: "true"
spec:
domainName: cafe.example.com
dnsRecordType: A
loadBalanceMethod: round-robin
pools:
- dnsRecordType: A
loadBalanceMethod: round-robin
dataServerName: /Common/demo-f5
[root@bastion01 cr]# oc create -f edns.yaml
externaldns.cis.f5.com/cafe-edns created
NGINX IC CR (Application Developer)
Policy: 定義請求速率限制策略
除了 Rate Limit 策略之外,Policy 還可以定義 WAF 等相關安全策略 (APP Protect)。
apiVersion: k8s.nginx.org/v1
kind: Policy
metadata:
name: rate-limit
spec:
rateLimit:
rate: 5r/s ### 限制每秒五次請求
zoneSize: 1M
key: ${request_uri} ### 針對每個 URI 做計數
[root@bastion01 nginx]# oc create -f policy_nginx.yaml
policy.k8s.nginx.org/rate-limit created
Virtual Server: 基於路徑的方式來發布服務
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
name: cafe-vs
spec:
host: cafe.example.com
tls:
secret: cafe-secret
policies:
- name: rate-limit ### Policy
upstreams:
- name: tea
service: tea-svc
port: 80
healthCheck:
enable: true
path: /tea
interval: 20s
jitter: 3s
fails: 5
passes: 2
connect-timeout: 30s
read-timeout: 20s
- name: coffee
service: coffee-svc
port: 80
healthCheck:
enable: true
path: /coffee
interval: 10s
jitter: 3s
fails: 3
passes: 2
connect-timeout: 30s
read-timeout: 20s
routes:
- path: /tea
action:
pass: tea
- path: /coffee
action:
pass: coffee
- path: /milk
action:
return:
code: 200
type: text/html
body: "Welcome to Nginx Plus KIC Workshop!!"
[root@bastion01 app]# oc create -f vs_nginx.yaml
virtualserver.k8s.nginx.org/cafe-vs created
NGINX IC Configuration
Virtual Server 以及後端 Pod
Rate Limit Zone
BIG-IP Configuration
透過 F5 IPAM 可以取得可用的 IP 地址,並且可以自動完成 Virtual Server(VS)及 Pool 的設定
自動套用預先定義好的 WAF 策略
自動註冊服務 DNS 紀錄
Testing
Application
存取應用路徑 /coffee
Security
嘗試輸入疑似違規參數,被 F5 WAF 阻擋
查看 F5 WAF Log,被判定為 XSS 攻擊
Rate limit
若每秒存取次數超過 5 次,將觸發請求限制門檻,NGINX 會回覆 503 狀態碼,表示暫時無法存取
查看 NGINX IC Pod 日誌
[root@bastion01 ~]# oc logs -f -n nginx-ingress nginx-ingress-674ddcc9dc-pvxmd | grep limiting
2022/04/05 07:55:16 [error] 29#29: *83496 limiting requests, excess: 0.105 by zone “pol_rl_default_rate-limit_default_cafe-vs”, client: 10.142.2.60, server: cafe.example.com, request: “GET /coffee HTTP/1.1”, host: “cafe.example.com”
查看 NGINX IC 儀表板,紅色區塊表示被限制的請求