Skip to main content

Good morning,


Recently our middletier where the prod environment works required a reboot because all services crashed. It is installed on ubuntu linux. Commands from the server mgmt, e.g. kubectl, reported no communication with kubernetes.

Is it recommended to reboot the middletier periodically? If so, is there technical documentation for this?

Hi,

I have never heard of any similar issues… 
how do you mean - “all services crashed.”  Which state were the pods in? “Pending”/”CrashLoopBack”/”Error” ?

if kubectl don’t work it sounds like a network issue, or that microk8s API also “crashed” - did you try “sudo microk8s kubectl” on the linux server itself?

There is no recommendation from IFS to periodically restart the k8s cluster server. Some might say it’s good practice to restart any server e.g. monthly - often as part of OS patching. 


Hi, 


I could not read the status of the pods because I could not execute the kubectl get pods -namespace command. I am 100% sure that was not a network problem, NMS did not return information that it could have been caused by the network. End users could not access the application, it stopped working. 

Performing a reboot on Vmware helped. So I'm wondering, if the machine hasn't been rebooted for a long time, should we reboot it more often, e.g. every month-three? Isn't there a recommendation for this in the documentation?


If you are or have access to a skilled linux admin you might find reasons for the “crash” in journalctl. It will be very verbose… even if the issue is logged there it might be hard to find… 

e.g.
journalctl
journalctl -u snap.microk8s.daemon-kubelite
journalctl -u snap.microk8s.daemon-containerd.service
journalctl --list-boots
dmesg

If this would happen again (hope not) - please login to the linux server and verify status on all pods.
sudo microk8s kubectl get pods -A -o wide

There is no recommendation (documentation) from IFS to periodically restart the k8s cluster server. Some might say it’s good practice to restart any server e.g. monthly - often as part of OS patching. 


Reply