Skip to main content

Hi Everyone

Do you notice lastly problems with Odata pod? Time to time it restarted automaticaly after reaching some memory pick. This is quite strange because many of environments are affected by this(PROD, CFG and even DEV envrionments with poor usage). It is observed by many toast message which are shown in IFS Cloud.

 

Few hours ago I faced problem like this. It disapeared automatically after Odata pod is up again. By the way when I list the pods last restart was 24d ago.

I know that process is killed because one time after deep investigation I got this:

Sep  4 07:28:34 clouddemo kernel: :2415668.649318] Memory cgroup out of memory: Killed process 2070719 (java) total-vm:3125508kB, anon-rss:1542888kB, file-rss:38396kB, shmem-rss:0kB, UID:1000 pgtables:3400kB oom_score_adj:953

Hmm - 

It is a bit strange as it seems Linux kills the container as it exeeds it cgroup memory allocation.
The java heap (xmx) will never exceed the cgroup memory allocation. Possible java allocates too much metaspace memory… which has not upper limit.

is this an old version 23r2? … on a new Ubuntu 22.04? or is cgroup v2 enabled in Ubuntu?
Older java had a bug on cgroup v2, where it though the container had access to the entrie VM memory. Then Linux killed java once it used more than the cgroup allocation for that container.

Hi

Problem observed on 23.2.5 and 24.1.3. I was also redirected to problem mentioned here:

On one environment I changed suggested Excel parameter to check possible improvement.

I will back with more detail.

​@knepiosko did you see any improvement?

ADDON TO ABOVE : We were experiencing the same issue. In our case we were able to recreate this issue on command by generating large outputs and sending to excel. We had the same error message indicated in the other forum post Pod ephemeral local storage usage exceed the total limit of containers in our odata pod logs.

Setting the EXCEL_EXTENTION_MODE system parameter to STREAMED remedied this for us

It was a problem related to pod resources.

Reply


Terms & ConditionsCookie settings