Deleting a Hung Namespace in Kubernetes

I've been working a bit with Kubeflow the past few weeks and getting it installed proved challenging. While working through the install I deleted various namespaces and pods while troubleshooting and ran across a situation where sometimes my namespaces would hang after deletion using kubectl delete namespace <namespace> with a state of Terminating. The rest of this post talks about how to move past that issue.

Note: this post originated from this great post by Craig Newton. I changed to reflect Kubeflow and posted here so I have a quick place to find it next time I need it.

First you have to get dump the namespace config into a json file:

kubectl get namespace <namespace> -o json > <namespace>.json

You should see something like:

{
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "creationTimestamp": "2020-01-14T21:05:16Z",
        "labels": {
            "control-plane": "kubeflow",
            "katib-metricscollector-injection": "enabled"
        },
        "name": "kubeflow",
        "resourceVersion": "884821",
        "selfLink": "/api/v1/namespaces/kubeflow",
        "uid": "28e3078b-ce62-425e-8145-a0606c028173"
    },
    "spec": {
        "finalizers": [
            "kubernetes"
        ]
    },
    "status": {
        "phase": "Active"
    }
}

You want to open the file for editing and remove "kubernetes" from the finalizers array:

{
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "creationTimestamp": "2020-01-14T21:05:16Z",
        "labels": {
            "control-plane": "kubeflow",
            "katib-metricscollector-injection": "enabled"
        },
        "name": "kubeflow",
        "resourceVersion": "884821",
        "selfLink": "/api/v1/namespaces/kubeflow",
        "uid": "28e3078b-ce62-425e-8145-a0606c028173"
    },
    "spec": {
        "finalizers": [

        ]
    },
    "status": {
        "phase": "Active"
    }
}

Next you'll execute your cleanup command:

kubectl replace --raw "/api/v1/namespaces/<namespace>/finalize" -f ./<namespace>.json

Check to ensure that the hung namespace is now removed from your cluster:

kubectl get namespaces -all-namespaces

Comments