Hi Richard,

by default IIS is set to recycle the App Pools every 29 hours. As far as I’m aware most of our customers do not change the value - unless instructed otherwise. For example, we have also customers recycling the App Pools daily during non-business hours or disabled it completly. I’m not aware that customers reboot the servers on a specific schedule to tackle performance issues.

What are the exact issue you are facing when the App Pool recycles? In general, a App Pool recycle stops all current processing as the IIS worker task is restarted. That can eventually cause issue for e.g. long running processes. Thus, if a recycle is enabled it has to be scheduled when no such processes occur.

Best regards
Roman

Thanks, Roman, for the reply.  The attached images show the profile of the web server average percent memory used from the last two weeks which includes an episode on Sept 18 @ 9pm when the scheduled processes stopped running and we performance a full shutdown and restart of all servers which also restarted the scheduled processes.  You will note that in both of these charts the web server memory increases throughout the day, plateaus and then drops which I am assuming is because the app pool was restarted due to this default setting (with the exception of the forced restart on 9/18 @ 10:50 pm.  Between the back office and mobile users, our typical usage period for the system is Monday - Friday 6 am - 8 pm. The main issue is that by about 3pm web client users start experiencing extremely sluggish performance of the web client application which more or less aligns to the memory plateau.  There is a definite pattern here and we are looking for options on what we can do to mitigate this issue.

The Web Server is running at 3.0 GHz with 32GB memory and 12vCPU.

Would appreciate any additional insight you can offer.

Cheers,

Richard

Hi Richard,

I suppose IFS already did a Review of the current Sizing during the Health Check? By reviewing the metrics, it isn’t like the Web Client is steady running at max memory and I would say it’s normal that you see an increase during the day. Of course, the memory should be also released at some point in time and if that only happens when the App Pool is recycled then something is strange.

I suggest to ask your end users, which screens are slow in the afternoon and then review if those screens have some configurations (e.g. Business Rules, Client Scripts, Custom Screens, etc.) to review them.

The memory consumption on your PSO I/O Server is quite high - eventually adding another I/O Server would be beneficial if you witness issues when PSO receives data from FSM and/or broadcasts data back to FSM, or while using the PSO Workbench.

Best regards
Roman

Thanks Roman.  We do have an engagement with IFS now to review the business rules and I’m expecting the results shortly.  Hopefully that bears some fruit.  Yes, your observation that the memory not getting released is where the issue is as this seems to be a clear example of a memory leak that is not reset until the process is recycled.  We are going to try to set the App Pool to recycle everyday @ 2am rather than the 1740 minutes it has been set to to see if that helps.  One thing we did notice today during another event where the web server memory went as high as 76% (accompanied by user complaints) was that we had 3700 active threads on the web client worker process and some that were as old as 5:30am which to me is an indication that connections are being created but not released and consuming available memory.  

Regarding PSO, no issues so far but a few months ago we did disable the PSO to FSM broadcast following a recommendation.  A separate work plan will involve a significant update to our PSO infrastructure to help support some other initiatives we have going on.

Any other suggestions?

Richard

Hi Richard,

can you please raise a case with our support regarding your concerns related to high number of active threads / possible memory leak?
As of now, I’m afraid I don’t have any further suggestions for you.

Best regards
Roman

Thanks, Roman, your you help on this.  Support case has been created.

-Richard