kubernetes.md · 6.3 KiB · Markdown Raw

# kubernetes

* get all secrets without column headers:

```
kubectl get secrets -o custom-columns=NAME:.metadata.name --no-headers=true
```

* get all pods on a node:

```
kg po --field-selector spec.nodeName=worker-c7j8z-7bc5db6557-dbcfz
```

* get events for a specific resource:

```
kubectl get events --field-selector involvedObject.kind=Pod,involvedObject.name=kube-proxy-4x6h2
```

* show every resource in a namespace:

```
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n NAMESPACE
```

* impersonate a service account:

```
kubectl auth can-i $verb $resource/$name --as=system:serviceaccount:$namespace:$serviceaccount

kubectl auth can-i use psp/kong-kong-app-psp --as=system:serviceaccount:giantswarm:kong-kong-app
```

```
kubectl auth can-i get DNSEndpoint/sidero -n kube-system --as=system:serviceaccount:external-dns:external-dns
```

* get logs from previous pod if it has died:

```
kubectl logs --previous traefik-ingress-controller-zcspk --namespace kube-system
```

* temporarily disable a daemonset, and re-enable it later:

```
kubectl patch ds dsname -p '{"spec": {"template": {"spec": {"nodeSelector": {"non-existing": "true"}}}}}'
kubectl patch ds dsname -p --type json -p='[{"op": "remove", "path": "/spec/template/spec/nodeSelector/non-existing"}]'
k patch app rook-operator --type='json'  -p='[{"op": "replace", "path": "/spec/version", "value": "2.5.0"}]'
```

* roll back a deployment:

```
kubectl rollout history deploy/name

deployment.extensions/name

REVISION  CHANGE-CAUSE

276       <none>
277       <none>
278       <none>
279       <none>
kubectl rollout undo deployment/happa --to-revision 279

deployment.extensions/happa rolled back
```

* get all evicted pods:

```
kubectl get pods --all-namespaces -o json | jq '.items[] | select(.status.reason!=null) | select(.status.reason | contains("Evicted"))'
```

* delete all failed pods:

```
kubectl -n default delete pods --field-selector=status.phase=Failed
```

* find pods consuming disk space:

```
TOP_STORAGE=$(du -hs /var/lib/docker/overlay2/* | grep -Ee '^[0-9]{3}[M]+|[0-9]G' | sort -h |tail -n 10 |tee -a /dev/stderr |awk '{print $2}'|xargs|sed 's/ /|/g')

docker inspect $(docker ps -q) | jq '.[]|.Config.Hostname,.Config.Labels."io.kubernetes.pod.name",.GraphDriver.Data.MergedDir,.hovno' | egrep -B2 "$TOP_STORAGE"

docker image inspect $(docker images -aq) | jq '.[]|.RepoDigests[0],.GraphDriver.Data.MergedDir,.hovno' |egrep -B2 "$TOP_STORAGE" 
```

* find service pods which can use a specific PSP:

```
kubectl get rolebindings,clusterrolebindings -A -o custom-columns='KIND:kind,NAMESPACE:metadata.namespace,NAME:metadata.name,SERVICE_ACCOUNTS:subjects[?(@.kind=="ServiceAccount")].name' | grep privileged
```

* get all nodes and their taints:

```
kubectl get nodes -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.kubernetes\.io/role}{"\t"}{.spec.taints}{"\n"}{end}'
```

* show tainted nodes and their taints:

```
kubectl get nodes -o=jsonpath='{range .items[?(@.spec.taints)]}{.metadata.name}{"\t"}{.metadata.labels.kubernetes\.io/role}{"\t"}{.spec.taints}{"\n"}{end}'
```

* find all pods which have unhealthy containers:

```
kubectl get po --all-namespaces | gawk 'match($3, /([0-9]+)\/([0-9]+)/, a) {if (a[1] < a[2] && $4 != "Completed") print}'
```

* find all pods with `hostNetwork: true`:

```
ykg po | yq -r '.items[] | select(.spec.hostNetwork == true).metadata.name'
```

* work around bug in `sops`:

```
sops --config <(echo '') --encrypt --input-type yaml --output-type yaml ./my-secret.yaml
```

* view old CRD version:

```
k explain helmrepository.spec --api-version source.toolkit.fluxcd.io/v1beta1
```

* check out node resources:

```
kdes no -l giantswarm.io/node-pool=worker |grep "Allocated resources" -A6

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests          Limits
  --------           --------          ------
  cpu                3678m (91%)       4295m (107%)
  memory             4327953122 (26%)  7271217318 (43%)
  ephemeral-storage  0 (0%)            0 (0%)
```

* delete namespace stuck finalising:

```
NS=broken-namespace

k get namespace $NS -ojson | k neat | jq '.spec.finalizers=null' > ns.json

kubectl proxy

# from another terminal in the same context:

curl -k -H "Content-Type: application/json" -X PUT --data-binary @ns.json http://127.0.0.1:8001/api/v1/namespaces/$NS/finalize
```

or as a one-liner:

```
NS=test

k get namespace $NS -ojson | k neat | jq '.spec.finalizers=null' > /tmp/ns.json && k proxy& sleep 2 && curl -k -H "Content-Type: application/json" -X PUT --data-binary @/tmp/ns.json http://127.0.0.1:8001/api/v1/namespaces/$NS/finalize && pkill kubectl
```

observing traffic with hubble

```
 hubble observe -4 -f --to-pod vpn-proxy/vpn-proxy-ccbcb9bb7-8v5rq
```

```
k port-forward -n kube-system svc/hubble-relay --address 0.0.0.0 --address :: 4245:80

hubble observe -4 -f --to-pod vpn-proxy/vpn-proxy-ccbcb9bb7-8v5rq
```

* remove finalizers from a lot of resources at once:

```
for app in $(k get app | awk '!/NAME/ {print $1}') ; do kubectl patch app $app -p '{"metadata":{"finalizers":null}}' --type=merge ; done
```

* find all pods in the cluster with proxy vars set:

* fix pods stuck due to kyverno cluster policy:

```
kubectl get clusterpolicies.kyverno.io inject-proxy-env --output yaml > /tmp/inject-proxy-env.yaml && kubectl delete clusterpolicies.kyverno.io inject-proxy-env && kubectl edit --namespace giantswarm $(kubectl get pods --namespace giantswarm --output name | grep scan-vulnerabilityreport) && sleep 1 && kubectl get pods --namespace giantswarm --output name | grep scan-vulnerabilityreport && kubectl create --filename /tmp/inject-proxy-env.yaml
```

## kubectl plugins

#### pexec

<https://github.com/ssup2/kpexec>

```
kubectl pexec -it -T capz-controller-manager-6b44b58485-nscf7 -c manager -- bash
```

kubernetes

get all secrets without column headers:

kubectl get secrets -o custom-columns=NAME:.metadata.name --no-headers=true

get all pods on a node:

kg po --field-selector spec.nodeName=worker-c7j8z-7bc5db6557-dbcfz

get events for a specific resource:

kubectl get events --field-selector involvedObject.kind=Pod,involvedObject.name=kube-proxy-4x6h2

show every resource in a namespace:

kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n NAMESPACE

impersonate a service account:

kubectl auth can-i $verb $resource/$name --as=system:serviceaccount:$namespace:$serviceaccount

kubectl auth can-i use psp/kong-kong-app-psp --as=system:serviceaccount:giantswarm:kong-kong-app

kubectl auth can-i get DNSEndpoint/sidero -n kube-system --as=system:serviceaccount:external-dns:external-dns

get logs from previous pod if it has died:

kubectl logs --previous traefik-ingress-controller-zcspk --namespace kube-system

temporarily disable a daemonset, and re-enable it later:

kubectl patch ds dsname -p '{"spec": {"template": {"spec": {"nodeSelector": {"non-existing": "true"}}}}}'
kubectl patch ds dsname -p --type json -p='[{"op": "remove", "path": "/spec/template/spec/nodeSelector/non-existing"}]'
k patch app rook-operator --type='json'  -p='[{"op": "replace", "path": "/spec/version", "value": "2.5.0"}]'

roll back a deployment:

kubectl rollout history deploy/name

deployment.extensions/name

REVISION  CHANGE-CAUSE

276       <none>
277       <none>
278       <none>
279       <none>
kubectl rollout undo deployment/happa --to-revision 279

deployment.extensions/happa rolled back

get all evicted pods:

kubectl get pods --all-namespaces -o json | jq '.items[] | select(.status.reason!=null) | select(.status.reason | contains("Evicted"))'

delete all failed pods:

kubectl -n default delete pods --field-selector=status.phase=Failed

find pods consuming disk space:

TOP_STORAGE=$(du -hs /var/lib/docker/overlay2/* | grep -Ee '^[0-9]{3}[M]+|[0-9]G' | sort -h |tail -n 10 |tee -a /dev/stderr |awk '{print $2}'|xargs|sed 's/ /|/g')

docker inspect $(docker ps -q) | jq '.[]|.Config.Hostname,.Config.Labels."io.kubernetes.pod.name",.GraphDriver.Data.MergedDir,.hovno' | egrep -B2 "$TOP_STORAGE"

docker image inspect $(docker images -aq) | jq '.[]|.RepoDigests[0],.GraphDriver.Data.MergedDir,.hovno' |egrep -B2 "$TOP_STORAGE"

find service pods which can use a specific PSP:

kubectl get rolebindings,clusterrolebindings -A -o custom-columns='KIND:kind,NAMESPACE:metadata.namespace,NAME:metadata.name,SERVICE_ACCOUNTS:subjects[?(@.kind=="ServiceAccount")].name' | grep privileged

get all nodes and their taints:

kubectl get nodes -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.kubernetes\.io/role}{"\t"}{.spec.taints}{"\n"}{end}'

show tainted nodes and their taints:

kubectl get nodes -o=jsonpath='{range .items[?(@.spec.taints)]}{.metadata.name}{"\t"}{.metadata.labels.kubernetes\.io/role}{"\t"}{.spec.taints}{"\n"}{end}'

find all pods which have unhealthy containers:

kubectl get po --all-namespaces | gawk 'match($3, /([0-9]+)\/([0-9]+)/, a) {if (a[1] < a[2] && $4 != "Completed") print}'

find all pods with hostNetwork: true:

ykg po | yq -r '.items[] | select(.spec.hostNetwork == true).metadata.name'

work around bug in sops:

sops --config <(echo '') --encrypt --input-type yaml --output-type yaml ./my-secret.yaml

view old CRD version:

k explain helmrepository.spec --api-version source.toolkit.fluxcd.io/v1beta1

check out node resources:

kdes no -l giantswarm.io/node-pool=worker |grep "Allocated resources" -A6

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests          Limits
  --------           --------          ------
  cpu                3678m (91%)       4295m (107%)
  memory             4327953122 (26%)  7271217318 (43%)
  ephemeral-storage  0 (0%)            0 (0%)

delete namespace stuck finalising:

NS=broken-namespace

k get namespace $NS -ojson | k neat | jq '.spec.finalizers=null' > ns.json

kubectl proxy

# from another terminal in the same context:

curl -k -H "Content-Type: application/json" -X PUT --data-binary @ns.json http://127.0.0.1:8001/api/v1/namespaces/$NS/finalize

or as a one-liner:

NS=test

k get namespace $NS -ojson | k neat | jq '.spec.finalizers=null' > /tmp/ns.json && k proxy& sleep 2 && curl -k -H "Content-Type: application/json" -X PUT --data-binary @/tmp/ns.json http://127.0.0.1:8001/api/v1/namespaces/$NS/finalize && pkill kubectl

observing traffic with hubble

 hubble observe -4 -f --to-pod vpn-proxy/vpn-proxy-ccbcb9bb7-8v5rq

k port-forward -n kube-system svc/hubble-relay --address 0.0.0.0 --address :: 4245:80

hubble observe -4 -f --to-pod vpn-proxy/vpn-proxy-ccbcb9bb7-8v5rq

remove finalizers from a lot of resources at once:

for app in $(k get app | awk '!/NAME/ {print $1}') ; do kubectl patch app $app -p '{"metadata":{"finalizers":null}}' --type=merge ; done

find all pods in the cluster with proxy vars set:

kubectl get pods --all-namespaces --output yaml | yq '
  .items[]
  | select((.metadata.name | test("^(capi|dex-k8s)-")) | not)
  | select((.metadata.namespace | test("^(kube-system|kyverno)$")) | not)
  | select(
      [ (.spec.containers // []) + (.spec.initContainers // [])
        | .[]
        | ([ (.env // [])[].name ] | any_c(. == "http_proxy" or . == "HTTP_PROXY"))
      ] | any_c(. == false)
    )
  | .metadata.namespace + "/" + .metadata.name
'

fix pods stuck due to kyverno cluster policy:

kubectl get clusterpolicies.kyverno.io inject-proxy-env --output yaml > /tmp/inject-proxy-env.yaml && kubectl delete clusterpolicies.kyverno.io inject-proxy-env && kubectl edit --namespace giantswarm $(kubectl get pods --namespace giantswarm --output name | grep scan-vulnerabilityreport) && sleep 1 && kubectl get pods --namespace giantswarm --output name | grep scan-vulnerabilityreport && kubectl create --filename /tmp/inject-proxy-env.yaml

kubectl plugins

pexec

https://github.com/ssup2/kpexec

kubectl pexec -it -T capz-controller-manager-6b44b58485-nscf7 -c manager -- bash