Skip to main content
Question

Odata pod instability

  • September 13, 2024
  • 5 replies
  • 211 views
K
Superhero (Partner)
Forum|alt.badge.img+16

Hi Everyone

Do you notice lastly problems with Odata pod? Time to time it restarted automaticaly after reaching some memory pick. This is quite strange because many of environments are affected by this(PROD, CFG and even DEV envrionments with poor usage). It is observed by many toast message which are shown in IFS Cloud.

 

Few hours ago I faced problem like this. It disapeared automatically after Odata pod is up again. By the way when I list the pods last restart was 24d ago.

I know that process is killed because one time after deep investigation I got this:

Sep  4 07:28:34 clouddemo kernel: [2415668.649318] Memory cgroup out of memory: Killed process 2070719 (java) total-vm:3125508kB, anon-rss:1542888kB, file-rss:38396kB, shmem-rss:0kB, UID:1000 pgtables:3400kB oom_score_adj:953

Piotr --> https://www.linkedin.com/in/piotr-skowronek-8425aa6/

5 replies

H
Hero (Employee)
Forum|alt.badge.img+10
  • hhanse
  • Hero (Employee)
  • 175 replies
  • September 16, 2024

Hmm - 

It is a bit strange as it seems Linux kills the container as it exeeds it cgroup memory allocation.
The java heap (xmx) will never exceed the cgroup memory allocation. Possible java allocates too much metaspace memory… which has not upper limit.

is this an old version 23r2? … on a new Ubuntu 22.04? or is cgroup v2 enabled in Ubuntu?
Older java had a bug on cgroup v2, where it though the container had access to the entrie VM memory. Then Linux killed java once it used more than the cgroup allocation for that container.

K
Superhero (Partner)
Forum|alt.badge.img+16
  • knepiosko
  • Author
  • Superhero (Partner)
  • 405 replies
  • September 17, 2024

Hi

Problem observed on 23.2.5 and 24.1.3. I was also redirected to problem mentioned here:

https://community.ifs.com/framework-and-experience-101/error-pod-ephemeral-local-storage-usage-exceeds-the-total-limit-of-containers-is-seen-in-the-middle-tier-deployment-logs-related-to-odata-provider-30267?tid=30267&fid=101

On one environment I changed suggested Excel parameter to check possible improvement.

I will back with more detail.

Piotr --> https://www.linkedin.com/in/piotr-skowronek-8425aa6/
E
Hero (Customer)
Forum|alt.badge.img+8
  • Edwards
  • Hero (Customer)
  • 44 replies
  • February 6, 2025

​@knepiosko did you see any improvement?

E
Hero (Customer)
Forum|alt.badge.img+8
  • Edwards
  • Hero (Customer)
  • 44 replies
  • February 6, 2025

ADDON TO ABOVE : We were experiencing the same issue. In our case we were able to recreate this issue on command by generating large outputs and sending to excel. We had the same error message indicated in the other forum post Pod ephemeral local storage usage exceed the total limit of containers in our odata pod logs.

Setting the EXCEL_EXTENTION_MODE system parameter to STREAMED remedied this for us

K
Superhero (Partner)
Forum|alt.badge.img+16
  • knepiosko
  • Author
  • Superhero (Partner)
  • 405 replies
  • February 6, 2025

It was a problem related to pod resources.

Piotr --> https://www.linkedin.com/in/piotr-skowronek-8425aa6/

Reply


Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings