It seems like this needs to be done through Weblogic admin console which our customers do not have access through license. Following oracle documentation explains a way to do this using odl_rotatelogs.exe which is located under IFSHome\mw_home\mws\ohs\bin. 

https://docs.oracle.com/cd/E29542_01/web.1111/e10144/man_logs.htm#HSADM912

Please try that in a TEST environment first and check whether that works. 

Is this the intended behavior of the middleware server?

I don’t think so. By default there should be a rotation and it should not consume all the disk space if the disk has enough space. 

What is the error message that HttpServer1.logs are filled with?

It is filled with ERROR:32 Active Websocket Connection Limit Reached: 4294967283

You may need to increase the number of Max Websocket Clients parameter in IFS Middleware Server Admin Console (Apps 10). This should drop the number of error messages you are seeing in the log.

NOTE: Please try this change in your test environment first.

You may need to increase the number of Max Websocket Clients parameter in IFS Middleware Server Admin Console (Apps 10). This should drop the number of error messages you are seeing in the log.

NOTE: Please try this change in your test environment first.

 

I already increased that to 5000, but I’ll try to increase it to 32768.

Thank you.

Given error could due to the causing exhaustion of OHS threads. Have you checked the increasing the "ThreadsPerChild" value from the Application installer  ?

In APPS9 this can be configured using the IFS Applications installer. 

........................................................................

Consider the following when setting values to ThreadsPerChild.

These OHS settings related to “MaxClients” or “ThreadsPerChild” is working differently with Windows and Linux.

• In Windows, you can see the "ThreadsPerChild" as the parameter name but in Linux you see the "MaxClients" in the installer. By default it use “worker MPM” in Linux where as “WinNT MPM” is used for windows so you have to look for the correct section in the httpd.exe when checking the values

how it works on Linux,

• As stated above, you can only change the "MaxClient" parameter via the reconfiguration and all other parameters will be calculated based on that.

• For example if you specify the "MaxClient" as 1000 then both "ThreadsPerChild" and "ThreadLimit" would be around 31 where as “WLMaxWebSocketClients” (defined in mod_wl_ohs_<instance>.conf in instance/conf) will be 750 (75% of MaxClient). 

• Because "MaxClient" value will be the highest value of serving requests so even if you manually specify a higher value for "ThreadsPerChild" then it will not make any advantage because it cannot go beyond "MaxClient" limit.

• On the other hand, you have to provide a sufficient value for the "MaxClient" parameter and the general figure would be 1.5 * No of logged in users.

• Also have to consider that "Streams" use web socket connections so 75% of the value that you have given for "MaxClients" will be allocated to web sockets. That means if you specify 1000 as the "MaxClients" then 750 connections are reserved for web sockets and there will only be 250 for all other traffic. Each logged in use will consume 1 web socket connection for steams.

.............................................................................

@Charith E I think @SORSITEL is referring to an APP10 instance. IN that case you should alter the HTTP Server configuration parameters via the IFS Middleware Server Admin Console. There you can increase the ‘ThreadsPerChlid’ count.

@Charith E@Isuru Gunathunga

Thus it should be:

• ThreadLimit should be 1200 * 1.5 = 1800  (1075 licenses + headroom)
• Max Websocket Clients should be 1800 * 0.75 = 1350
• Then ThreadsPerChild should be 1800 - 1350 = 450

Is my reasoning correct or horribly wrong?

The websockets are the ones which should be focused on right, because keeping these open will yield the best performance, right?

EDIT:

I see in the logs that with the configuration above, it lowers the WLMaxWebSocketClients to 337, which is 450 * 0,75. Thus, ThreadsPerChild should be equal to ThreadLimit on Windows (as only 1 process is spawned).

When setting ThreadsPerChild = 1800, it now says:

pVirtualHost: main] Lowering WLMaxWebSocketClients to 1350. Increase ThreadsPerChild to support more connections

Which is what we want, right?

@SORSITEL "ThreadsPerChlid" is the parameter you should increase as 1800. Can you please try out setting a ThreadsPerChlid somewhat higher value than "1800" and check whether the issue still on the system?

Is the streams functionality used in the system ? "Streams" use web socket connections so 75% of the value that you have given for "ThreadsPerChlid" will be allocated to web sockets. That means if you specify 1000 as the "ThreadsPerChlid" then 750 connections are reserved for web sockets and there will only be 250 for all other traffic. Each logged in use will consume 1 web socket connection for steams.

@Charith E What I originally had for ThreadLimit and ThreadsPerChild was 5000, but then I did the calculations and figured out that since we are 1075 users, these parameters should be 1800.

This seems to have stopped the errors from appearing, at least for now.

The errors that sometimes arise now, is this:

(OS 10054)An existing connection was forcibly closed by the remote host.  : ap_get_brigade() didn't read data
(OS 10054)An existing connection was forcibly closed by the remote host.  : Error processing websocket connection

Is this normal?

(We only use the built-in IFS functionality, EE and Aurena. Thus the remaining 450 connections reserved for all other traffic I assume will be plenty, considering these are only used for normal HTTP requests, correct?)

@SORSITEL Glad to hear the number of errors reduced on the HTTP log.

Since the HTTP Server has the error ‘connection forcibly closed by remote host’, I suspect this could be due to an external device (just a thought). Is there any load balancer configured to balance the amount of traffic between the nodes? Could there be any other devices between client and server which might also be forcibly closing the underlying TCP connection?

If so can you please check the timeout settings on the load balancer as well. (This suggests the default timeout on load balancer side is perhaps less?). Kindly check whether LB timeout is bit higher for WebSocket type connections?

I don’t think we are running any load balancers, and we are not running with multiple nodes.

The logs have stayed small (500KB), so I’m happy .

Thanks a lot for the invaluable information!

@Charith E@Isuru Gunathunga

Thus it should be:

• ThreadLimit should be 1200 * 1.5 = 1800  (1075 licenses + headroom)
• Max Websocket Clients should be 1800 * 0.75 = 1350
• Then ThreadsPerChild should be 1800 - 1350 = 450

Is my reasoning correct or horribly wrong?

The websockets are the ones which should be focused on right, because keeping these open will yield the best performance, right?

 

EDIT:

I see in the logs that with the configuration above, it lowers the WLMaxWebSocketClients to 337, which is 450 * 0,75. Thus, ThreadsPerChild should be equal to ThreadLimit on Windows (as only 1 process is spawned).

When setting ThreadsPerChild = 1800, it now says:

pVirtualHost: main] Lowering WLMaxWebSocketClients to 1350. Increase ThreadsPerChild to support more connections

Which is what we want, right?

 @SORSITEL @Charith Epasinghe

We are experiencing a similar issue. In our instance the HttpServer1.log files are growing and we are seeing error messages around WebSocket connection Limit Error.  I followed the steps above in our Apps 10 environment, once we save the changes in the middleware. Do we need to do anything else i.e. restart something or do a re-config?

Thanks in advance!
Chris