Hi Kacper,

> Why is it caused that those pods are still restarting?

There are many reasons that a pod may restart, including an unhandled exception (=error), insufficient CPU or memory, or a failed health probe.

> If POD has been restarted we should delete it and let it get up again?

Most pods are managed by a deployment, which manages the pod lifetime and replicas. If a pod fails and the replica count is not met, the deployment will create a replacement pod. You can optionally cleanup after failed pods but it is not required. Typically you only remove pods to force a new one to be created.

> How we can see what happening and what is the cause of restart on this certain pod?

Please check the pod logs and node description.

Node description:

kubectl describe node <node name if there are multiple>

Please pay particular attention to the CPU and memory/resources and events sections near the end.

Pod logs:

kubectl logs deployments/ifsapp-connect -n <ifs namespace>
kubectl logs pod/<pod name> -n <ifs namespace>

Pods generally container multiple containers. Please append "-c <container name>" if you need to focus on logs for a particular container.

Best regards -- Ben

Fine, I attached logs from restarting and no-restarting pod. Can you help me with investigating them or just point line which might be a reason for restarting pod? Im attaching both of dumps to compare: 

I used AI to verify these logs and it returned a message: 

The analysis of the logs from both Kubernetes pods shows that the restarting pod ("2_bad.txt") contains errors not present in the logs of the stable pod ("1_good.txt"). Key differences and issues detected in the "2_bad.txt" logs include:

1. Errors related to ExecutionException and PermanentFailureException: There's an error message indicating a permanent failure during data transmission, suggesting that the pod attempted an operation that failed and cannot be automatically recovered. Specifically, the error points to a problem with ifs.fnd.connect.senders.ConnectSenderManager attempting to send data, resulting in an ExecutionException and a PermanentFailureException​​.
2. ConcurrentAccessTimeoutException: Another critical error occurs when the pod tries to access a resource but cannot obtain it within the designated time. Notably, ConnectTimerPostForwarder cannot obtain a lock within 5000 milliseconds, leading to a ConcurrentAccessTimeoutException​​.

These errors are critical and can directly contribute to the pod's restarting, as they indicate problems that prevent the application from functioning properly. Specifically, errors related to the permanent operation failure and access time out to resources can result in pod instability, ultimately leading to its restart.

In summary, the main difference between the "1_good.txt" and "2_bad.txt" logs is the presence of critical errors in "2_bad.txt" that are not present in "1_good.txt". These errors are likely the cause of the stability issues and restarts of the pod represented by "2_bad.txt". Addressing these problems would require further analysis and possibly modifications to the configuration or application code to handle ExecutionException and ConcurrentAccessTimeoutException errors and ensure proper error handling.

Obviously, 1_good.txt is a log with no restarting pod and 2_bad.txt is a log with a restarting pod. And I wonder, what operation in system can cause errors pointed by AI in attached text?

Hi kacperifs!

Did you ever get more information about this issue? It seems that we are facing the same issue in one of our customer’s environments, with similar error messages constantly showing up on ifsapp-connect pod, and having issues with integrations.

Hi @Mikko1234 

We had the same problem, for which the solution turned out to be replication of pods according to the documentation that support provided. In addition, we also increased the resources on Vmware, which we allocated to the environment accordingly. The POD is still able to crash, although it does not affect the performance of the integration.

Even with plenty of memory in your VM, pods can continue crashing, because the default values for pod memorys are not sufficient. 

Manual Memory Allocation for Pods in ifscloud-values.yaml

You can manually allocate memory to specific pods using the addMemory directive in the ifscloud-values.yaml file. For example:

Note

We encountered issues with pods being OOMKilled (Out of Memory). The default memory settings were insufficient, even on servers with 128 GB of RAM.

After analyzing memory usage via Grafana, we observed that only about 10% of the available memory was being utilized. To mitigate this, we manually added the addMemory directive to increase the memory allocation for the affected pods.

Hey ​@MCIPSTEV ,

I'm currently shocked because I was looking for the memory allocation step for a specific pod and couldn't find any information on it anywhere. Could you give me a link to specific documentation on how to enter this correctly in yaml? Alternatively, could you send me a snippet where this directive is included, please?

Hello,

Unfortunately, this behavior is not documented anywhere . I'm currently facing major issues with IFS support, as the default memory values for the pods are unknown, and my pods started crashing right after a simple SU8 upgrade on my 24R2 remote cloud local instance.

Initially, I suspected a memory leak, but it now seems more likely that the default memory allocations are simply too low for certain pods. Without the addMemory directive, they fail with OOMKilled errors, even though the system has sufficient physical memory.

This lack of transparency and predictability is a serious concern, especially in production environments.

We have a 128GB expensive server in EC2 Amazon cloud, and we were facing issues like OOMKILLED and pods restarting.

At the end we added these rules in ifscloud-values.yaml, and we barely use 30% of the available memory.

# Disabled because not needed
ifsappsigningservice:
  replicas: 0

ifsappreportingbr:
  replicas: 1

ifsappreportingweb:
  replicas: 1
  addMemory: 4000 

ifsappnative:
  replicas: 1
  addMemory: 2000 

ifsappdoc:
  replicas: 1
  addMemory: 2000 

#This is a collection of small applications like - apidoc - demand server proxy - gisint - svcsch
ifsappapplicationsvc:
  #replicas: 0
  addMemory: 4000 

# Handles generation of the output of Operational Report Layouts.
ifsappreporting:
  #replicas: 0 
  addMemory: 5000  

ifsappreportingren:
  replicas: 2
  addMemory: 10000  
# Used for generating the final output of Report Designer Layouts.


#IFS OData Provider. This is the OData Endpoint for the server logic and additionally provides the projection API's for native mobile clients.
ifsappodata:
  replicas: 2
  addMemory: 5000 

ifsappconnect:  
  replicas: 2
  addMemory: 3000 

ifsappextensibility:
  addMemory: 3000  

ifsappnativeexecutor:
  replicas: 1

ifsappnativenotification:
  replicas: 1 

ifsappnativeserver:
  replicas: 1

ifsappdocmanesign:
  replicas: 0

ifsappscim:
  replicas: 0

ifsappreportingwebdesignerren:
  replicas: 1

ifsappreportingwebdesigner:
  replicas: 1

ifsvirusscanner:
  replicas: 0

ifsappreportingwebtransformer: 
  replicas: 1

ifsappreportingwebruntimeren:
  replicas: 1

Please share memory and cpu details of your MT server:
Memory: 128 GB and pods were crashing before addMemory rules

Hey, ​@MCIPSTEV 

In mine MT it looks like this:

ok, 
I have

and currenly having crashes for the reporting-ren pod

How can you be sure that the addMemory directive works if it is not even mentioned in the Kubernetes documentation? Isn't it the case that you can add it in yaml, but it is completely ignored?

Another important thing is what vCPU allocation do you have and what is your GlobalScale parameter?

I have the pod handling the report-rendering, I allocated 10 GB
 

Above is the output of  journalctl --since "2025-08-06 XX:XX" and I see exactely the OOMKILLED for the pod, wich means it ran out of mémory of 10 GB !

By default the ifsapp-report-rendering pod has less memory.

Send me your values from global section especially scale, cpuRatio and replicas.

You can find it in ifscloud-values.yaml

In the global section (If we put scale  1, 20 or 50) it is soooooo slowww

replicas: 1
scale: 100
 

cpuRatio is unchanged

Set the scale to 100, try cpuRatio at 50, and leave replicas=1. You can adjust the ifsappreporting replicas set to 2. If one fails, the other will take over its tasks. To be honest, I'm not sure about this addMemory directive. I think you'll need to remove the namespace and run the installation again after these changes.

We have applied the process of adjusting scale and CPURatio together with the IFS support consultant without succes. The only way of getting rid of most of OOMKILLEd is the addMemory directive.

So I am rather sure about the directive addMemory because it was suggested by R&D of IFS support.

Hi ​@MCIPSTEV,

> The default memory settings were insufficient, even on servers with 128 GB of RAM.

Pods are configured to use memory limits.
You can check the memory limit for a pod by describing it. For example:

kubectl describe pod -l app=ifsapp-odata -n <your namespace>

...
    Limits:
      cpu:                10
      ephemeral-storage:  1G
      memory:             1300M
    Requests:
      cpu:                225m
      ephemeral-storage:  750M
      memory:             1300M
...

Kubernetes will terminate a pod if it tries to use more memory than defined. As such it does not matter if the server has extra unused memory.

I am not able to comment on undocumented configurations. However, you should be able to confirm any changes to the memory limits.

Best regards -- Ben

For information:

Pods crashing:  
  
1. ifs-app- connect:
IFS connect - messaged broker integrates to external system using files/ftp/mail/soap/json
2. ifsapp-reporting-ren:
Used for generating the final output of Report Designer Layouts.
3. ifsapp-doc
The end user documentation
4. ifsapp-reporting:
Handles generation of the output of Operational Report Layouts.
5. ifsapp-client-services
Cloud Web client services - serves up metadata for Cloud Web and lobby pages.

Default memory allocation

After applying the 24R2 SU08: the default values for the pods are:

Hi ​@MCIPSTEV 

So the problem is still there, right?

Yes