IFS Cloud K8s PODs Garbage memory collection

​@HashanD is it possible to share how have you applied the heap dump and manually call the CG please? 

I tried that myself.

1. Run “jsp” to find process ID
2. Create dump “jcmd <ID> GC.heap_dump dump.hprof”

In my case, the used memory dind’t change much, less than 10% for sure.

Linux
kubectl exec -n <namespace> <pod name> -c <deployment name> -- jps | grep .jar
    
Windows
kubectl exec -n <namespace> <pod name> -c <deployment name> -- jps | Select-String ".jar”

in the output first column is the PID

Linux
kubectl exec -n <namespace> -it <pod name> -c <deployment name> -- jmap -dump:live,format=b,file=/tmp/heap-ifsapp-<container name>.hprof <PID>

Windows
kubectl exec -n <namespace> -it <pod name> -c <deployment name> -- jmap "-dump:live,format=b,file=/tmp/heap-ifsapp-<container name>.hprof" <PID>

Heap dump will be saved inside the particular pod "/tmp/heap-ifsapp-<container name>.hprof"

Copy the heap dump file into the local host
Linux / Windows
kubectl -n <namespace> -c <Deployment name> cp <Pod name>:<Full path of the file inside pod> <Target location and file name in local server>

kubectl exec -n <namespace> -it <pod name> -c <deployment name> -- jmap -dump:live,format=b,file=/tmp/heap-ifsapp-<container name>.hprof <PID>

Have you used dump:live?
When you use live, the JVM must first run a Full Garbage Collection to determine which objects are still reachable. If you include live temporary collections, buffer, caches with weak/soft refs, etc will be forced by GC and then JVM releases memory back to OS.

Also you will have to wait for few seconds to see the significat diff from kubectl top command

That does indeed make a difference, 75% used memory to 25%

Interesting! Definitely something worth analysing!

Yup
But I forgot to mention it's better to execute in non-peak hours, as heap dump generation may cause some latency for the current executing services of the POD

I have dispatched a IFS Support case as well. I hope I would recive positive feedback from them 🙂

First - global.scale is set to 100 i assume?

If you get OOM that mean that the pod is out of memory not that the GC is not working as it should.
In my experience OOM happens when large java metapace structures are created. Java metaspace is not included in GC and is not part of the memory sizing of java at pod startup.

monitor it with:
jcmd <pid> VM.metaspace 

You should not get frequent OOM’s in pods. Increase the number of pods to spread the metaspace over more pods. If issue persist report as bugs.

 

Thank you for your input. I understand what an OOM condition means. If garbage collection is working correctly, the pod should have additional memory available to use.

For some of our customers, we have configured four replicas for the ifsapp-odata deployment and increased both memory requests and memory limits. Despite this, the pods still encounter OOM restarts.

My main point is that when I generate a Java heap dump, a full garbage collection is triggered, and a significant amount of memory is immediately released back to the system. This clearly shows that a large amount of reclaimable memory was being retained.

Another user has confirmed observing the same behavior as described above.

What i’m trying to say is that the oData pod is out of the memory that is not handled by the GC. Memory areas like this:
 

• Metaspace
• Code cache
• Thread stacks
• JIT compiler memory
• Direct ByteBuffers (off‑heap)
• GC native structures
• NIO buffers
• Class data sharing
• Other JVM native allocations
• …. and other Linux OS memory 

Even if you set the pods memory limit really large, that will not affect these memory areas. If you look at your GC snapshots  you can see that none of the metaspace and classspace memory are released by your GC, just the “normal” heap memory. In your pic we can also see that 166Mb MetaSpace memory is used (which is a big chunk of the available 200Mb in oData).  

There is a hidden parameter that controls how much memory of the requested memory should be used for non-normal-heaps memory. For oData this is set to 200Mb, and only the Metaspace alone use 166 of that. That is leaving 34Mb for the rest of of the memory areas in the list above, so try increasing it to 500 to see if that reduces the OOM.

ifsappodata:
   minStatic: 500

 
This is not an official IFS recommendation as it comes through a community discussion. Using hidden parameters are not really supported. But it might help the troubleshooting via your normal support channel.

Yeah, I get what you mean, and thank you for providing this hidden parameter.

Hi all 

​@HashanD Could the minStatic Parameter solve your issue with the OOM of the Pods? 

We are having the same issue that Odata Pod gobbles up every bit of memory and as soon as it is over the limit of 95% of the Pods Memory Ressources it restarts with a OOM.
Is the restart of the pods the intended solution from IFS for a not working garbage collecions? ;) Or can we do something else there?

Thank you really much for your help.