Question

Upgrade microk8s fails with certificate error

  • 11 December 2023
  • 4 replies
  • 198 views
roklde
Rank
Userlevel 6
Badge +23
  • roklde
  • Superhero (Employee)
  • 613 replies

Hi Community,

I’m having issues updating an internal microk8s cluster. 

The command I’m trying to run is:
 

sudo snap refresh microk8s --channel=1.23/stable

It prompts following error:
 

error: cannot refresh "microk8s": Post https://api.snapcraft.io/v2/snaps/refresh: x509: certificate
       is valid for ingress.local, not api.snapcraft.io

 

When issuing the following command to check on the certificate I’m also getting a strange response back:

openssl s_client -connect api.snapcraft.io:443

CONNECTED(00000003)
depth=0 O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
   i:O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDbzCCAlegAwIBAgIQINrsNaTDS8eAPt8YOke7PDANBgkqhkiG9w0BAQsFADBL
MRAwDgYDVQQKEwdBY21lIENvMTcwNQYDVQQDEy5LdWJlcm5ldGVzIEluZ3Jlc3Mg
Q29udHJvbGxlciBGYWtlIENlcnRpZmljYXRlMB4XDTIzMTIwODA4MzIzOFoXDTI0
MTIwNzA4MzIzOFowSzEQMA4GA1UEChMHQWNtZSBDbzE3MDUGA1UEAxMuS3ViZXJu
ZXRlcyBJbmdyZXNzIENvbnRyb2xsZXIgRmFrZSBDZXJ0aWZpY2F0ZTCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBAJsNJUM3DRNlnceWvV1oRDjohPpAmo1+
2u9OSRLfvfwNHKp+F5b53ZlXbf5ME+iAEkw0U3ng9UroryGZT5wyuDdjgl/ecbJv
1p9LwkqNfVMqwQrZ2yXk1FpoikSUjOyJg7BTGhnJITHe0A/OqQjsnSVFsuqxjx/0
PeMLNrsPm9hS9McThT9+luEg5eyS8mFHuMmjC4NQSIubdVFoUJ6736qE8qKszhOd
nt9oF2V0WHnjXe0ERuonFPsmpTVU0HfphOav0JxQHc5KBVJEpns54jdYX22A7F6n
yEduWrVqQTFPC345yzaLc2Y/TxvNzs46+BobUYHSLK8fXvUk6QCimIMCAwEAAaNP
ME0wDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB
/wQCMAAwGAYDVR0RBBEwD4INaW5ncmVzcy5sb2NhbDANBgkqhkiG9w0BAQsFAAOC
AQEAjs6Gn0HiWissUT2OPCvRSSafuZeh6xlyKp46GGatlG61cepdpYP/8zAEANsG
rco/GLpwExFcql2IOXAXpZ5BPMulR6sfEY03vnxuXvhPFnfWUN+o4m4dBokST8Lb
UD/Jn9z6HLmceBLo4cgrGV2D0JEaVv/PsNCoKzvYPtCi7KL4yMSW1cAsDHt7VFr3
USvVSQayXCc2rNHpgBpvsu3/XxMIRPmaKbgOst7nTbeCvWNnyEPG5YhZQ3gl0LsF
kRtHFZdcb5cz/BQNZJPR/PJ70WTsdI9Bi0FURuu6CbE2L4fna4SIgsuSjWkbkV6e
7W76oiVFAvIgHT3+OYp2akd9GA==
-----END CERTIFICATE-----
subject=O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate

issuer=O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1365 bytes and written 401 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 1D8EF098FFCB4F45AE54FE92960114F4207D4633A67B1D942CF7F4D50195B389
    Session-ID-ctx:
    Master-Key: B5073B8FEAD9ECB42330B63E1230A478673422ACB0B76A208591CC60D5FBDD6BD854E75ACAECFDD0F38A319AFC61FEA1
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1702288150
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: yes
---
closed


Obviously, this looks like a certificate issue.

I tried all sort of things to troubleshoot this issue but haven’t found a solution yet. Did someone else had those kind of issues when upgrading microk8s?


OS: Ubuntu 20.04.6 LTS (GNU/Linux 6.6.4-060604-generic x86_64)


Best regards
Roman

4 replies

durette
Rank
Userlevel 7
Badge +18

One of the IFS installer scripts writes a line to /etc/hosts on the middleware server:

127.0.0.1       api.snapcraft.io

This means when you reach out to SnapCraft, it bypasses your DNS server and uses the local machine instead, so when you reach out to get that certificate, you’re getting it from your own middleware server instead of Canonical’s Snapcraft server.

I would guess IFS added this to prevent an upgrade that breaks the application, but I would love if an architect could chime in.

If you’re familiar with editing text files from the terminal, does it work if you remove that line?

If you find this forum useful, please consider paying back to the community in your own areas of expertise. -- Kevin Durette - https://www.linkedin.com/in/kevindurette
durette
Rank
Userlevel 7
Badge +18

In case you don't know the vi editor, there are running jokes on the internet about this text editor being hard to use (and especially hard to exit), so use caution.

From the terminal:

sudo vi /etc/hosts

(sudo = "superuser do", i.e. as root, and with great power comes great responsibility.)

This isn't Notepad; you don't drop into the editor in editing mode right away. You start in command mode, where the letters on your keyboard do commands. As a very important warning if you're accustomed to Windows, these commands are case-sensitive.

To begin, typing a capital G will put you on the bottom of the file where your api.snapcraft.io line likely exists.

Type dd (lowercase d twice) to remove that line. The first d means you want to delete something, and the second d means you want the whole line.

Now type :wq{Enter} (colon, w, q, Enter) to write the file and quit.

Then try again.

 

This editor comes pre-installed on pretty much every Linux machine on the planet, so it’s worth learning.

If you find this forum useful, please consider paying back to the community in your own areas of expertise. -- Kevin Durette - https://www.linkedin.com/in/kevindurette
H
Rank
Userlevel 5
Badge +10

Have you tried a complete reinstall 😁
If e.g. a cert expires i have not found a way of renewing them… !?
I guess the same goes for “corrupted” certs.


You better wipe out any traces of the old mk8s before reinstalling on a server.

sudo snap remove --purge microk8s
sudo snap remove --purge kubectl
sudo rm -R /root/snap/microk8s
sudo rm -R /home/ifs/microk8s
sudo rm -R /home/ifs/snap
sudo rm -R /home/ifs/remote-scripts
sudo rm /home/ifs/install-k8s.sh
sudo reboot

roklde
Rank
Userlevel 6
Badge +23

Thanks for the response, Henrik. No, I didn’t try to completely uninstall and reinstall yet. This would be the “last restort”! ...yeah, I also tried to update certs with some commands but without luck.

 

Best regards
Roman

Reply


Terms & ConditionsCookie settings