Question

Streams 503 error (Notifications)

  • 7 January 2020
  • 3 replies
  • 1745 views
dgralek
Rank
Userlevel 1
Badge +3

Hello.

Today I had a new issue with my application server. I’ve noticed that my notification area showed a warning sign.

After some troubleshooting I’ve found error 503 in debug, and entries in access_log: [07/Jan/2020:10:18:40 +0100] "GET /websocket/notifications HTTP/1.1" 503 211 0

Another step was http_server log, this time different record:

[2020-01-07T15:57:33.1390+01:00] [ohs] [ERROR:32] [OH99999] [weblogic] [client_id: *****] [host_id: *****] [host_addr: *****] [pid: 3092] [tid: 5496] [user: SYSTEM] [VirtualHost: ********] <3092157840903686> *******Exception type [CONNECTION_REFUSED] (apr_socket_connect call failed with error=730061, host=::1, port=58100 ) raised at line 1692 of URL.cpp
and:

[2020-01-07T15:55:56.0807+01:00] [ohs] [ERROR:32] [OH99999] [weblogic] [client_id: *****] [host_id: ****] [host_addr: *****] [pid: 2664] [tid: 3548] [user: SYSTEM] [VirtualHost: *****:58080] Active Websocket Connection Limit Reached: 4294967295

After some more investigation I’ve found thousands of sessions in netstat calling [::1]:58100 - so I’ve decided to reboot the whole server and the problem disappeared.

Have you ever seen such issue - is there any easier way to fix it (with no server reboot)?

Best! Damian

Vanitas vanitatum et omnia vanitas.

3 replies

Kasun Balasooriya
Rank
Userlevel 7
Badge +20

Hi,

I have seen this issue reported by customers in the past. The issue here is the Oracle HTTP Server consumes and hold on to a failed WebSocket connection indefinitely, causing exhaustion of OHS threads. When this happens, client notifications (Streams) no longer works (no notifications sent to client)

To fix this issue in APPS9, the following patches have been released: 

  • 122055 - OHS not able to drop WebSocket connection (UPD1)
  • 133516 - Oracle Critical Patch Updates for Middle Tier (UPD6)
  • 135598 - Oracle Critical Patch Updates for Middle Tier (UPD7)

If you are currently in a higher UPD level for APPS9 or using APPS10 and still getting this error, you can try increasing the ThreadsPerChild value. In APPS9 this can be configured using the IFS Applications installer. 

 

 

In APPS10, this can be configured using the IFS middle-ware server admin console. 

 

 

Please consider the following when setting values to ThreadsPerChild 

 

1. These OHS settings related to MaxClients or ThreadsPerChild is working differently with Windows and Linux.

 

2. In Windows, you can see the "ThreadsPerChild" as the parameter name but in Linux you see the "MaxClients" in the installer. By default it use “worker MPM” in Linux where as “WinNT MPM” is used for windows so you have to look for the correct section in the httpd.exe when checking the values.

 

3. As we have a Linux setup here, I will explain on how it works on Linux in below.

 

4. As stated above, you can only change the "MaxClient" parameter via the reconfiguration and all other parameters will be calculated based on that. 

 

5. For example if you specify the "MaxClient" as 1000 then both "ThreadsPerChild" and "ThreadLimit" would be around 31 where as “WLMaxWebSocketClients” (defined in mod_wl_ohs_<instance>.conf in instance/conf) will be 750 (75% of MaxClient). 

 

6. The rule is that "ThreadsPerChild" * "ThreadLimit" < "MaxClient".

 

7. Because "MaxClient" value will be the highest value of serving requests so even if you manually specify a higher value for "ThreadsPerChild" then it will not make any advantage because it cannot go beyond "MaxClient" limit.

 

8. On the other hand, you have to provide a sufficient value for the "MaxClient" parameter and the general figure would be 1.5 * No of logged in users.

 

9. You also have to consider that "Streams" use web socket connections so 75% of the value that you have given for "MaxClients" will be allocated to web sockets. That means if you specify 1000 as the "MaxClients" then 750 connections are reserved for web sockets and there will only be 250 for all other traffic. Each logged in use will consume 1 web socket connection for steams.

 

Hope this helps!

Kasun B - https://www.linkedin.com/in/kasunbalasooriya/
Charith Epitawatta
Rank
Userlevel 7
Badge +31

@dgralek Apart from what @KasunBalasooriya has mentioned in his reply, you may also look into following when troubleshooting streams issues where you see a warning sign in the notification area.

 

The warning sign usually means that the web socket connection is terminated. If it is not resolved automatically after a few minutes, open the Debug Console and check the log information in the sub category “FndWebSocketService”. Here are a couple of possible errors you might see:

  • HTTP/1.1 404 or 503 : This means that the Streams application running on IFS Middleware Sever is not deployed properly. To remedy this, login to IFS Middleware Server Admin Console and check deployments. If the streams deployment is not active, doing a reconfiguration of the application server will redeploy it. 
  • HTTP/1.1 401: This means that the Websocket authentication is failing. Check if there are adequate database sessions and processes available for authentication. There should at least be around 50-100 sessions/processes available.

Hope this helps!

 

Charith - https://charith.xyz
KamuranCanakli
Rank
Userlevel 2
Badge +5

Hello where can I find IFS reconfigure instance tools

Reply


Terms & ConditionsCookie settings