Hello ​@Charith Epitawatta /team,

can you please help here identify the root cause?
what could have caused the change in expiry dates?

Also the output of the below command, correct way to check the validity of the certificate?
 

Powershell:

kubectl exec -it --namespace=ifs-ingress ((kubectl get pods -A) -match 'nginx-controller' -split '\s+')s1] -- /bin/bash -c "openssl x509 -in /etc/ingress-controller/ssl/default-fake-certificate.pem -text"

Response is highly appreciated here.

Thanks,
Hardik

Hi Hardik,

Other than the HTTPS certificate, there are multiple Kubernetes certificates in use. Please check the validity of these. You can do this by logging in to your Kubernetes node and executing the following command:

sudo microk8s refresh-certs -c

Please share the output.

Best regards -- Ben

Thanks ​@Ben Monroe !

Here’s some follow-up based on the latest findings.

When I ran the command:

sudo microk8s refresh-certs -c

I received the following output:

The CA certificate will expire in 3631 days. The server certificate will expire in 346 days. The front proxy client certificate will expire in 346 days.

However, previously when I checked using the following command:

kubectl exec -it --namespace=ifs-ingress $(kubectl get pods -n ifs-ingress | grep nginx-controller | awk '{print $1}' | head -1) -- openssl x509 -in /etc/ingress-controller/ssl/default-fake-certificate.pem -text

…it showed that the certificate was valid until 15th November 2025. Despite that, we suddenly received an X.509 certificate error on 23rd May in our PROD environment.

Questions:

1. Could this issue have been caused by one of the other certificates (e.g., server or front proxy client) expiring or becoming invalid, even though the default-fake-certificate.pem was still valid?

2. Are there other certificate sources or dependencies (possibly from ingress or internal services) that may lead to such X.509 errors, even if the fake certificate appears valid?

Additional Question:

Is there a way to renew or refresh the certificate(s) (especially for ingress) without reinstalling the full Kubernetes stack or the ingress namespace? We'd like to avoid downtime or full redeployment if possible.

Any insight or guidance would be greatly appreciated as the customer is seeking a root cause analysis to prevent recurrence.

Regards,
Hardik

Hi Hardik,

The internal Kubernetes certificate appear to be valid.
When the ingress namespace is first installed it defaults to a fake certificate.
It is only after running installer.cmd that the certificate in ifsroot\config\certs as defined in ifsroot\config\ifscloud-values.yaml is applied.
You may update the certificdate and definition here and re-run installer.cmd with the mtinstaller to apply it.
The ingress namespace does not need to be reinstalled.

I do not think you should be checking against "default-fake=certificate.pem". Presumably you have applied a valid TLS certificate. Please try to verify the certificate applied to your server.

openssl s_client -connect <host name>:443 | openssl x509 -noout -dates

This assumes that openssl is available. You can execute it from any machine that can access your IFS Cloud host.

Best regards -- Ben

Thanks ​@Ben Monroe ,

Issue is not about the TLS certificate and more of towards the Kubernetes internal certificates 

We had got this error all of sudden on PROD. (on 23rd may 2025)
what could have been the cause?

E0523 06:27:45.856566    4736 memcache.go:265] couldn't get current server API group list: Get "https://<middelware-IP>:16443/api?timeout=32s": x509: certificate has expired or is not yet valid: current time 2025-05-23T06:27:45Z is after 2025-05-22T18:33:24Z
E0523 06:27:45.862380    4736 memcache.go:265] couldn't get current server API group list: Get "https://<middelware-IP>:16443/api?timeout=32s": x509: certificate has expired or is not yet valid: current time 2025-05-23T06:27:45Z is after 2025-05-22T18:33:24Z
E0523 06:27:45.867595    4736 memcache.go:265] couldn't get current server API group list: Get "https://<middelware-IP>:16443/api?timeout=32s": x509: certificate has expired or is not yet valid: current time 2025-05-23T06:27:45Z is after 2025-05-22T18:33:24Z
E0523 06:27:45.872803    4736 memcache.go:265] couldn't get current server API group list: Get "https://<middelware-IP>:16443/api?timeout=32s": x509: certificate has expired or is not yet valid: current time 2025-05-23T06:27:45Z is after 2025-05-22T18:33:24Z
E0523 06:27:45.879065    4736 memcache.go:265] couldn't get current server API group list: Get "https://<middelware-IP>:16443/api?timeout=32s": x509: certificate has expired or is not yet valid: current time 2025-05-23T06:27:45Z is after 2025-05-22T18:33:24Z
Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2025-05-23T06:27:45Z is after 2025-05-22T18:33:24Z
error: pod, type/name or --filename must be specified


NOTE:- after reinstalling the Kubernetes from scratch, this was resolved and we had new certificate expiry date valid for one year, but we want to know the root cause.

Regards,
Hardik
 

​@hardik We have encountered a similar issue where environments went down due to expiring Kubernetes certificates. We've found that some certificates seem to expire before their stated expiration date, which may be due to the method used to extract the expiry information.

The Knowledge Base (KB) article below can help you monitor some of these certificates; however, please be aware that it may not cover all of them.

 https://support.ifs.com/csm?id=kb_article_view&sys_kb_id=811fee9ac307d21465365083e4013170&table=kb_knowledge&searchTerm=x509