Java Taking all CPU on Application server

What IFS version is this?  We find that Apps10 takes a lot of RAM in order to run well.  Apps9 isn’t so RAM hungry.

From a CPU perspective it can do that when starting up (which can take 20 mins sometimes) or if it is just thrashing due to the server being too small.  What CPU/cores are available on the machine?  Perhaps your server is undersized.

Nick

I am running Apps 9 update 9 currently. The server has 4 cores and 12GB RAM. This seems to start fine after a reboot, but after only a day or so the java grows to 6GB a the CPU runs to 99%. I did rebuild the instance and increased the HEAP memory. 

How many users in this environment?  What is your java heap max set to right now?

I’m wondering if the total of allowed Java for the instance as well as the OS requirements + other processes is too much for your server.  If so, increasing the java heap max will likely make your performance actually worse since it can try to take RAM that it can’t actually get.

We get weird situations in Apps10 where going above a certain threshold of RAM use on the server means that any additional instance cannot be started, even though there is more than enough showing as free.

We’ve also had problems with active antivirus/anti-malware scans causing issues when running on the instance, if that might be something else you’re seeing.

I bumped the max java heap to 5GB a couple months ago. It seemed to fix an out of memory issue that the clients would get intermittently.

We only have 180 users and not all on at the same time.

When I was trouble shooting the memory problems with IFS they could never see anything from the dump files pointing to a culprit in the environment. I found an article posted about the error I was getting and t increase the heap to prevent it. Maybe that wasn’t a good idea.

That server doesn’t seem big enough to me for 180 users.  It likely meets the minimum specs but not much more.  If this is a VM I would suggest adding another 4 virtual processors and maybe push the RAM up to 16 or 20GB if possible.

Assuming a min of 5GB needed to host your instance, plus the OS overhead and whatever else you have on there, 12GB RAM is probably cutting it too close.

4 cores is not much to run any recent supported version of Windows along with an IFS application server (again, plus whatever else you have on there).

HTH,

Nick

For a rough comparison, our Apps10 prod environment is running on 8 cores, 40GB RAM, with approximately the same user count as you have.  Memory use is around 24GB in general.  Apps 10 is certainly more resource hungry than Apps9 but as you can see there is a big difference between the server specs.

Looking back over the last month of CPU usage on the server I think somebody has caused this. The usage goes from a steady 2-5% to 99% in one day, then stays there. Now the fun part of trying to find the cause.

We run Apps 9 across 2 x middleware servers (4 x Managed Services). So it would be difficult to  give you any useful guidance. Node1 of the middleware server uses 12.3 Gb, and node2 uses 8.4Gb. We do have 400users. I would probably agree with Nick in that perhaps 12Gb total might be a little tight.

Perhaps the network boys have rolled something out without a change control! ;)

Hope you find the solution soon.

Regards

Mike

Working with IFS support we did a thread dump of the application server and it turned out to be an image file we were using for the reports. Somehow the system is trying to animate the .gif file and starts a java thread to run the animation and then never closes it.

We updated our company logo so that is when it was changed. Issue was self inflicted.

We replaced the image with a non-animated image filet, .jpg. After we restated the application server the threads have cleared and the system is back to averaging 100 threads in the java heap and down to 3-5% CPU usage.

Himasha Kapugeekiyanage and his team helped with the analyzing of the heap dump.

We had the same issue in Apps 8. We had to make changes to Weblogic, this is how we finally got out system to calm down.  We have been good for about 5 months after making this change.