Skip to main content

Good morning,


Recently our middletier where the prod environment works required a reboot because all services crashed. It is installed on ubuntu linux. Commands from the server mgmt, e.g. kubectl, reported no communication with kubernetes.

Is it recommended to reboot the middletier periodically? If so, is there technical documentation for this?

Hi,

I have never heard of any similar issues… 
how do you mean - “all services crashed.”  Which state were the pods in? “Pending”/”CrashLoopBack”/”Error” ?

if kubectl don’t work it sounds like a network issue, or that microk8s API also “crashed” - did you try “sudo microk8s kubectl” on the linux server itself?

There is no recommendation from IFS to periodically restart the k8s cluster server. Some might say it’s good practice to restart any server e.g. monthly - often as part of OS patching. 


Hi, 


I could not read the status of the pods because I could not execute the kubectl get pods -namespace command. I am 100% sure that was not a network problem, NMS did not return information that it could have been caused by the network. End users could not access the application, it stopped working. 

Performing a reboot on Vmware helped. So I'm wondering, if the machine hasn't been rebooted for a long time, should we reboot it more often, e.g. every month-three? Isn't there a recommendation for this in the documentation?


If you are or have access to a skilled linux admin you might find reasons for the “crash” in journalctl. It will be very verbose… even if the issue is logged there it might be hard to find… 

e.g.
journalctl
journalctl -u snap.microk8s.daemon-kubelite
journalctl -u snap.microk8s.daemon-containerd.service
journalctl --list-boots
dmesg

If this would happen again (hope not) - please login to the linux server and verify status on all pods.
sudo microk8s kubectl get pods -A -o wide

There is no recommendation (documentation) from IFS to periodically restart the k8s cluster server. Some might say it’s good practice to restart any server e.g. monthly - often as part of OS patching. 


Usual process is

  • for windows servers: 2 reboots for pathc tuesday
  • for linux: when updates are available

It is wierd not to have a suggestion for a reboot of the whole IFS Cloud remote, wich is in my opinion the basic of opérationnal start, stop and maintenance on any industry appliance.

As a customer, we do not now what the order is in case of shutdown and restart as it was with IFSv8 procedures

 


If you are or have access to a skilled linux admin you might find reasons for the “crash” in journalctl. It will be very verbose… even if the issue is logged there it might be hard to find… 

e.g.
journalctl
journalctl -u snap.microk8s.daemon-kubelite
journalctl -u snap.microk8s.daemon-containerd.service
journalctl --list-boots
dmesg

If this would happen again (hope not) - please login to the linux server and verify status on all pods.
sudo microk8s kubectl get pods -A -o wide

There is no recommendation (documentation) from IFS to periodically restart the k8s cluster server. Some might say it’s good practice to restart any server e.g. monthly - often as part of OS patching. 

 

In case of an unattended reboot, kubernetes pods are totally out of service. I found no other solution than reinstalling the MT, but it should be nice to have a bunch of shutdown scripts for the MT (and included the Kubernetes plateform) because the script mctl only start of stop the ifs namespace, but not the undelying kubernetes.

I’m willing to share my experience ;)


Reply