測試環境 OpenShift 4.8 / F5 CIS 2.7.1 / NGINX IC 2.1.0


Introduce

本文將介紹如何在 OpenShift 環境中搭配使用 F5 相關的解決方案,其中包括 F5 BIG-IP CIS 、F5 IPAM 和 NGINX IC 的部屬和應用。此外,還需要注意與 OpenShift 整合所需的相關事項,可以參考之前分享的『F5 BIG-IP CIS 與 NGINX IC 強化 OpenShift 入口控制能力 (Part.1)』內容。至於 OpenShift 集群的安裝細節,可以參考紅帽官方文件


Architecture

F5 BIG-IP CIS
  • 負載流量至 NGINX IC Pod (Host Base)
  • 動態服務發現 NGINX IC Pod,並自動更新相關設定
  • 使用 F5 IPAM 實現自動獲取 VIP
  • 自動發布服務至 F5 BIG-IP (DNS Record)
  • 套用 WAF 安全策略
NGINX IC
  • 負載流量至應用程式 Pod (Path Base)
  • 訪問請求頻率限制


Config F5 BIG-IP

VXLAN Tunnel
  1. 建立 VXALN 隧道,注意 VNI 需使用 4097 與 OpenShift 溝通

  2. 設定 VXLAN 隧道 Over Layer Self IP,請確保網段範圍能夠覆蓋 OpenShift 集群的網路,以便讓 F5 BIG-IP 可以直接路由到 OpenShift

Create Partition
  1. 建立一個 Partition 供 F5 CIS 使用,注意名稱不要使用大寫
Install AS3
  1. 下載 AS3 RPM 安裝檔 (F5 Networks GitHub)

  2. 到 F5 BIG-IP 安裝 AS3


Deploy F5 BIG-IP Container Ingress Services (CIS)

Create CIS Login Account

替換成自己的 BIG-IP 設備管理帳號和密碼

oc create secret generic bigip-login --namespace kube-system --from-literal=username=<BIG-IP管理帳號> --from-literal=password=<BIG-IP管理密碼>
Create Service Account / Cluster Role / Cluster Role Binding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: bigip-ctlr-clusterrole
rules:
  - apiGroups: ["", "extensions", "networking.k8s.io"]
    resources: ["nodes", "services", "endpoints", "namespaces", "ingresses", "pods", "ingressclasses", "policies"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["", "extensions", "networking.k8s.io"]
    resources: ["configmaps", "events", "ingresses/status", "services/status"]
    verbs: ["get", "list", "watch", "update", "create", "patch"]
  - apiGroups: ["cis.f5.com"]
    resources: ["virtualservers","virtualservers/status", "tlsprofiles", "transportservers", "transportservers/status", "ingresslinks", "ingresslinks/status", "externaldnses", "policies"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: ["fic.f5.com"]
    resources: ["ipams", "ipams/status"]
    verbs: ["get", "list", "watch", "update", "create", "patch", "delete"]
  - apiGroups: ["apiextensions.k8s.io"]
    resources: ["customresourcedefinitions"]
    verbs: ["get", "list", "watch", "update", "create", "patch"]
  - apiGroups: ["", "extensions"]
    resources: ["secrets"]
    verbs: ["get", "list", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: bigip-ctlr-clusterrole-binding
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: bigip-ctlr-clusterrole
subjects:
  - apiGroup: ""
    kind: ServiceAccount
    name: bigip-ctlr
    namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bigip-ctlr
  namespace: kube-system
注意需設定 bigip-ctlr service account 集群管理員權限
[root@bastion01 cis]# oc create -f sa_clusterrole.yaml
clusterrole.rbac.authorization.k8s.io/bigip-ctlr-clusterrole created
clusterrolebinding.rbac.authorization.k8s.io/bigip-ctlr-clusterrole-binding created
serviceaccount/bigip-ctlr created
[root@bastion01 cis]# oc adm policy add-cluster-role-to-user cluster-admin -z bigip-ctlr -n kube-system
clusterrole.rbac.authorization.k8s.io/cluster-admin added: "bigip-ctlr"
Create Custom Resource Definitions

CRD 定義檔請參考

[root@bastion01 cis]# oc create -f customresourcedefinitions.yml
customresourcedefinition.apiextensions.k8s.io/virtualservers.cis.f5.com created
customresourcedefinition.apiextensions.k8s.io/tlsprofiles.cis.f5.com created
customresourcedefinition.apiextensions.k8s.io/transportservers.cis.f5.com created
customresourcedefinition.apiextensions.k8s.io/externaldnses.cis.f5.com created
customresourcedefinition.apiextensions.k8s.io/ingresslinks.cis.f5.com created
customresourcedefinition.apiextensions.k8s.io/policies.cis.f5.com created
Create Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-bigip-ctlr-deployment
  namespace: kube-system
spec:
  # DO NOT INCREASE REPLICA COUNT
  replicas: 1
  selector:
    matchLabels:
      app: k8s-bigip-ctlr-deployment
  template:
    metadata:
      labels:
        app: k8s-bigip-ctlr-deployment
    spec:
      # Name of the Service Account bound to a Cluster Role with the required
      # permissions
      containers:
        - name: k8s-bigip-ctlr
          image: "f5networks/k8s-bigip-ctlr:2.8.0"
          env:
            - name: BIGIP_USERNAME
              valueFrom:
                secretKeyRef:
                  # Replace with the name of the Secret containing your login
                  # credentials
                  name: bigip-login
                  key: username
            - name: BIGIP_PASSWORD
              valueFrom:
                secretKeyRef:
                  # Replace with the name of the Secret containing your login
                  # credentials
                  name: bigip-login
                  key: password
          command: ["/app/bin/k8s-bigip-ctlr"]
          args: [
            # See the k8s-bigip-ctlr documentation for information about
            # all config options
            # https://clouddocs.f5.com/containers/latest/
              "--bigip-username=$(BIGIP_USERNAME)",
              "--bigip-password=$(BIGIP_PASSWORD)",
              "--bigip-url=192.168.102.98",                          ### F5 管理 IP
              "--bigip-partition=ocp",                               ### CIS 使用的 Partition 名稱
              "--pool-member-type=cluster",
              "--openshift-sdn-name=/Common/ocp-vxlan-tunnel",       ### F5 BIG-IP VXLAN Tunnel 名稱
              "--log-level=DEBUG",
              "--insecure=true",
              "--custom-resource-mode=true",                         ### 啟用 CRD 設定
              "--as3-validation=true",
              "--log-as3-response=true",
              "--ipam=true",                                         ### 啟用 IPAM 支援
              "--namespace-label=cis=true",                          ### 指定監聽的 Namespace Label
              "--disable-teems=true",
          ]
      serviceAccountName: bigip-ctlr
[root@bastion01 cis]# oc create -f deployment.yaml
deployment.apps/k8s-bigip-ctlr-deployment created

Deploy F5 IPAM

Create Service Account / Cluster Role / Cluster Role Binding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ipam-ctlr-clusterrole
rules:
  - apiGroups: ["fic.f5.com"]
    resources: ["ipams","ipams/status"]
    verbs: ["get", "list", "watch", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: ipam-ctlr-clusterrole-binding
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ipam-ctlr-clusterrole
subjects:
  - apiGroup: ""
    kind: ServiceAccount
    name: ipam-ctlr
    namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ipam-ctlr
  namespace: kube-system
[root@bastion01 ipam]# oc create -f sa_clusterrole.yaml
clusterrole.rbac.authorization.k8s.io/ipam-ctlr-clusterrole created
clusterrolebinding.rbac.authorization.k8s.io/ipam-ctlr-clusterrole-binding created
serviceaccount/ipam-ctlr created
Create Volume
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-pv
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  storageClassName: local-storage
  local:
    path: /tmp/cis_ipam
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - worker01
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-local
  namespace: kube-system
spec:
  storageClassName: local-storage
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 0.1Gi
[root@bastion01 ipam]# oc create -f pv.yaml
storageclass.storage.k8s.io/local-storage created
persistentvolume/local-pv created
persistentvolumeclaim/pvc-local created
Create Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    name: f5-ipam-controller
  name: f5-ipam-controller
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: f5-ipam-controller
  template:
    metadata:
      labels:
        app: f5-ipam-controller
    spec:
      containers:
      - args:
        - --orchestration=openshift
        - --ip-range='{"test":"172.16.90.180-172.16.90.185"}'     ### 要派發的 IP 及其名稱
        - --log-level=DEBUG
        command:
        - /app/bin/f5-ipam-controller
        image: f5networks/f5-ipam-controller:0.1.6
        imagePullPolicy: IfNotPresent
        name: f5-ipam-controller
        terminationMessagePath: /dev/termination-log
        volumeMounts:
        - mountPath: /app/ipamdb
          name: local-pv
      securityContext:
        fsGroup: 1200
        runAsGroup: 1200
        runAsUser: 1200
      serviceAccount: ipam-ctlr
      serviceAccountName: ipam-ctlr
      volumes:
      - name: local-pv
        persistentVolumeClaim:
          claimName: pvc-local
[root@bastion01 ipam]# oc create -f deployment.yaml
deployment.apps/f5-ipam-controller created

Deploy NGINX IC

Create Namespace / Service Account
apiVersion: v1
kind: Namespace
metadata:
  name: nginx-ingress
  labels:
    cis: "true"     ### 標記讓 F5 BIG-IP CIS 知道需要監聽這個 Namespace
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress
  namespace: nginx-ingress
[root@bastion01 nginx]# oc create -f ns-and-sa.yaml
namespace/nginx-ingress created
serviceaccount/nginx-ingress created
Create Cluster Role / Cluster Role Binding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: nginx-ingress
rules:
- apiGroups:
  - ""
  resources:
  - services
  - endpoints
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - get
  - list
  - watch
  - update
  - create
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
  - patch
  - list
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs:
  - list
  - watch
  - get
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses/status
  verbs:
  - update
- apiGroups:
  - k8s.nginx.org
  resources:
  - virtualservers
  - virtualserverroutes
  - globalconfigurations
  - transportservers
  - policies
  verbs:
  - list
  - watch
  - get
- apiGroups:
  - k8s.nginx.org
  resources:
  - virtualservers/status
  - virtualserverroutes/status
  - policies/status
  - transportservers/status
  verbs:
  - update
- apiGroups:
  - networking.k8s.io
  resources:
  - ingressclasses
  verbs:
  - get
- apiGroups:
    - cis.f5.com
  resources:
    - ingresslinks
  verbs:
    - list
    - watch
    - get
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nginx-ingress
subjects:
- kind: ServiceAccount
  name: nginx-ingress
  namespace: nginx-ingress
roleRef:
  kind: ClusterRole
  name: nginx-ingress
  apiGroup: rbac.authorization.k8s.io
[root@bastion01 nginx]# oc create -f ns-and-sa.yaml
namespace/nginx-ingress created
serviceaccount/nginx-ingress created
Create Ingress Class
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx
  annotations:
    ingressclass.kubernetes.io/is-default-class: "true"
spec:
  controller: nginx.org/ingress-controller
[root@bastion01 nginx]# oc create -f ingress-class.yaml
ingressclass.networking.k8s.io/nginx created
Create Default Server Certificate
apiVersion: v1
kind: Secret
metadata:
  name: default-server-secret
  namespace: nginx-ingress
type: kubernetes.io/tls
data:
  tls.crt:     ### 你的憑證
  tls.key:     ### 你的金鑰
[root@bastion01 nginx]# oc create -f default-server-secret.yaml
secret/default-server-secret created
Create Service
apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress
  namespace: nginx-ingress
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  - port: 443
    targetPort: 443
    protocol: TCP
    name: https
  selector:
    app: nginx-ingress
[root@bastion01 nginx]# oc create -f nginx-kic-svc.yaml
service/nginx-ingress created
Create Config Map
kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-config
  namespace: nginx-ingress
data:
  lb-method: "round_robin"
[root@bastion01 nginx]# oc create -f nginx-config.yaml
configmap/nginx-config created
Create Custom Resource Definitions

CRD 定義檔請參考

[root@bastion01 nginx]# oc create -f crd/
customresourcedefinition.apiextensions.k8s.io/aplogconfs.appprotect.f5.com created
customresourcedefinition.apiextensions.k8s.io/appolicies.appprotect.f5.com created
customresourcedefinition.apiextensions.k8s.io/apusersigs.appprotect.f5.com created
customresourcedefinition.apiextensions.k8s.io/globalconfigurations.k8s.nginx.org created
customresourcedefinition.apiextensions.k8s.io/policies.k8s.nginx.org created
customresourcedefinition.apiextensions.k8s.io/transportservers.k8s.nginx.org created
customresourcedefinition.apiextensions.k8s.io/virtualserverroutes.k8s.nginx.org created
customresourcedefinition.apiextensions.k8s.io/virtualservers.k8s.nginx.org created
Create Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-ingress
  namespace: nginx-ingress
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-ingress
  template:
    metadata:
      labels:
        app: nginx-ingress
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9113"
    spec:
      serviceAccountName: nginx-ingress
      containers:
      - image: nginx-plus:2.1.0
        imagePullPolicy: IfNotPresent
        name: nginx-plus-ingress
        ports:
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443
        - name: readiness-port
          containerPort: 8081
        - name: dashboard
          containerPort: 9000
        - name: prometheus
          containerPort: 9113
        # readinessProbe:
        #   httpGet:
        #     path: /nginx-ready
        #     port: readiness-port
        #   periodSeconds: 1
        securityContext:
          allowPrivilegeEscalation: true
          runAsUser: 101 #nginx
          capabilities:
            drop:
            - ALL
            add:
            - NET_BIND_SERVICE
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        args:
          - -nginx-plus
          - -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
          - -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret
          - -nginx-status-port=9000
          - -nginx-status-allow-cidrs=0.0.0.0/0
          - -enable-snippets
         #- -enable-app-protect
         #- -v=3 # Enables extensive logging. Useful for troubleshooting.
         #- -report-ingress-status
         #- -external-service=nginx-ingress
          - -enable-prometheus-metrics
          - -enable-latency-metrics
         #- -global-configuration=$(POD_NAMESPACE)/nginx-configuration
         #- -ingresslink=nginx-ingress
         #- -report-ingress-status
          - -enable-preview-policies
[root@bastion01 nginx]# oc create -f nginx-plus-ingress.yaml
deployment.apps/nginx-ingress created

Expose Application

Application : Coffee & Tea
[root@bastion01 app]# oc get pods,svc
NAME                         READY     STATUS     RESTARTS     AGE
pod/coffee-9759f989b-6dpcj   1/1       Running    0            7d5h
pod/coffee-9759f989b-8jd9t   1/1       Running    0            7d5h
pod/coffee-9759f989b-cdrxr   1/1       Running    0            7d5h
pod/tea-7464cc487b-7cthd     1/1       Running    0            7d5h
pod/tea-7464cc487b-vr5vz     1/1       Running    0            7d5h
pod/tea-7464cc487b-z8z2d     1/1       Running    0            7d5h
NAME                TYPE      CLUSTER-IP    EXTERNAL-IP PORT(S) AGE
service/coffee-svc  ClusterIP 172.30.141.25 <none>      80/TCP  26d
service/tea-svc     ClusterIP 172.30.90.69  <none>      80/TCP  26d
F5 CIS CR (Cluster Operator)

TLS Profile: 定義要使用的 SSL Profile

apiVersion: cis.f5.com/v1
kind: TLSProfile
metadata:
  name: reencrypt-tls
  namespace: nginx-ingress
  labels:
    f5cr: "true"
spec:
  tls:
    termination: reencrypt
    clientSSL: /Common/clientssl   ### Client SSL Profile
    serverSSL: /Common/serverssl   ### Server SSL Profile
    reference: bigip
  hosts:
  - cafe.example.com
[root@bastion01 cr]# oc create -f tls.yaml
tlsprofile.cis.f5.com/reencrypt-tls created

Policy: 定義要使用的 WAF 策略和設定 Persistence

apiVersion: cis.f5.com/v1
kind: Policy
metadata:
  labels:
    f5cr: "true"
  name: cafe-policy
  namespace: nginx-ingress
spec:
  l7Policies:
    waf: /Common/cafe_sp          ### 預先建立好的 WAF Policy
  profiles:
    persistenceProfile: none      ### 不做 Persistence
    logProfiles:
      - /Common/Log all requests  ### WAF Log Profile
[root@bastion01 cr]# oc create -f policy_f5.yaml
policy.cis.f5.com/cafe-policy created

Virtual Server: 基於主機名稱的方式來發布服務

apiVersion: "cis.f5.com/v1"
kind: VirtualServer
metadata:
  name: cafe-vs
  namespace: nginx-ingress
  labels:
    f5cr: "true"
spec:
  host: cafe.example.com
  virtualServerHTTPSPort: 443
  virtualServerName: cafe_demo_vs
  ipamLabel: test                   ### IPAM Pool
  tlsProfileName: reencrypt-tls     ### TLS Profile
  policyName: cafe-policy           ### Policy
  snat: auto
  allowVlans: ["/Common/vlan_90"]
  pools:
  - monitor:
      interval: 15
      recv: ""
      send: "GET /coffee HTTP/1.1\r\nHost: cafe.example.com\r\n"
      timeout: 46
      type: https
    service: nginx-ingress
    servicePort: 443
[root@bastion01 cr]# oc create -f vs_f5.yaml
virtualserver.cis.f5.com/cafe-vs created

ExternalDNS: 將服務的 DNS 紀錄新增至 F5 BIG-IP DNS 智能解析 (GSLB)

在上一步驟中建立 Virtual Server 時,已經配置了對後端進行健康狀態監控的設定(使用 HTTP Monitor)。因此,不需要進一步設定額外的 DNS 監控,只需使用預設的 BIG-IP 監控(透過 I-Query)來獲取 Virtual Server 的狀態即可。

apiVersion: "cis.f5.com/v1"
kind: ExternalDNS
metadata:
  name: cafe-edns
  namespace: nginx-ingress
  labels:
    f5cr: "true"
spec:
  domainName: cafe.example.com
  dnsRecordType: A
  loadBalanceMethod: round-robin
  pools:
  - dnsRecordType: A
    loadBalanceMethod: round-robin
    dataServerName: /Common/demo-f5
[root@bastion01 cr]# oc create -f edns.yaml
externaldns.cis.f5.com/cafe-edns created
NGINX IC CR (Application Developer)

Policy: 定義請求速率限制策略

除了 Rate Limit 策略之外,Policy 還可以定義 WAF 等相關安全策略 (APP Protect)。

apiVersion: k8s.nginx.org/v1
kind: Policy
metadata:
  name: rate-limit
spec:
  rateLimit:
    rate: 5r/s            ### 限制每秒五次請求
    zoneSize: 1M
    key: ${request_uri}   ### 針對每個 URI 做計數
[root@bastion01 nginx]# oc create -f policy_nginx.yaml
policy.k8s.nginx.org/rate-limit created

Virtual Server: 基於路徑的方式來發布服務

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: cafe-vs
spec:
  host: cafe.example.com
  tls:
    secret: cafe-secret
  policies:
  - name: rate-limit     ### Policy
  upstreams:
  - name: tea
    service: tea-svc
    port: 80
    healthCheck:
      enable: true
      path: /tea
      interval: 20s
      jitter: 3s
      fails: 5
      passes: 2
      connect-timeout: 30s
      read-timeout: 20s
  - name: coffee
    service: coffee-svc
    port: 80
    healthCheck:
      enable: true
      path: /coffee
      interval: 10s
      jitter: 3s
      fails: 3
      passes: 2
      connect-timeout: 30s
      read-timeout: 20s
  routes:
  - path: /tea
    action:
      pass: tea
  - path: /coffee
    action:
      pass: coffee
  - path: /milk
    action:
      return:
        code: 200
        type: text/html
        body: "Welcome to Nginx Plus KIC Workshop!!"
[root@bastion01 app]# oc create -f vs_nginx.yaml
virtualserver.k8s.nginx.org/cafe-vs created
NGINX IC Configuration

Virtual Server 以及後端 Pod

Rate Limit Zone

BIG-IP Configuration

透過 F5 IPAM 可以取得可用的 IP 地址,並且可以自動完成 Virtual Server(VS)及 Pool 的設定

自動套用預先定義好的 WAF 策略

自動註冊服務 DNS 紀錄


Testing

Application

存取應用路徑 /coffee

Security

嘗試輸入疑似違規參數,被 F5 WAF 阻擋

查看 F5 WAF Log,被判定為 XSS 攻擊

Rate limit

若每秒存取次數超過 5 次,將觸發請求限制門檻,NGINX 會回覆 503 狀態碼,表示暫時無法存取

查看 NGINX IC Pod 日誌

[root@bastion01 ~]# oc logs -f -n nginx-ingress nginx-ingress-674ddcc9dc-pvxmd | grep limiting
2022/04/05 07:55:16 [error] 29#29: *83496 limiting requests, excess: 0.105 by zone “pol_rl_default_rate-limit_default_cafe-vs”, client: 10.142.2.60, server: cafe.example.com, request: “GET /coffee HTTP/1.1”, host: “cafe.example.com”

查看 NGINX IC 儀表板,紅色區塊表示被限制的請求


References