Solved

MainServer1 logs are growing in size - Apps 10 UPD8

  • 7 June 2021
  • 7 replies
  • 534 views

Userlevel 6
Badge +18

Hi,

The MainServer logs are continuously growing and causing the drive hosting IFS Application to go out of space with the following repetitive errors:

####<Jun 6, 2021, 6:16:29,948 PM PDT> <Error> <com.ifsworld.jms.IfsWebStreamsTopicBean> <DIVIFSAPP02> <MainServer1> <[ACTIVE] ExecuteThread: '27' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-55E98D048B51F37E17FE> <ea976e53-587f-48f7-854f-00e338b9f183-6d5822c7> <1623028589948> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000000> <null
java.lang.NullPointerException

The MainServer1.out log can grow up to 35 GB before I manually clear it out:

 

Has someone come across the same issue. I noticed this happens in other non-production environment too but the logs aren’t growing as big & fast, as just a few users access it.

Thanks in advance!

icon

Best answer by Imal Thiunuwan 7 June 2021, 13:57

View original

This topic has been closed for comments

7 replies

Userlevel 6
Badge +16

Hello @Srikanth,

This issue happens due to failed Stream messages. could you please check if there are any failed JMS Stream messages available in the WebLogic Admin Console? If there are any available please delete them first.

The logs are getting bigger and bigger due to the repetition of mentioned error and  it causes for a huge CPU consumption of both MainServer1 and MainServer2. As a solution, you can change the value of the parameter “Redelivery limit” by logging into the Middleware Server Admin Console. Please find below screenshot.

-Login to IFS Middleware Server Admin Console.

-Navigate to Common>JMS Modules> StreamJMSModule> StreamsJMSTopic.

-Change the value of "Redelivery Limit" from -1 to 5 -Restart the Middleware Server for the change to take effect.

 

The  default value of the field “Redelivery Limit” has been set into -1, which means unlimited retries. This would have resulted in getting logs written and consuming a large number of CPU usage as well as Logs growth. By changing the value, set to “Redelivery Limit”, the stream message would drop after several tries and the CPU utilization should come down. Hence, change the value into like 5.

Also please follow below steps as well. 

1). Stop MainServer1.

2). Delete  *.dat files under

wls_domain\<INSTANCE_NAME>\servers\MainServer1\data\store\default

wls_domain\<INSTANCE_NAME>\servers\MainServer1\data\store\diagnostics

3). Restart the server and check if the error is still available.

Please check the solutions 286167 and 281112 (UPD9) as well.

Cheers!

Userlevel 6
Badge +13

Hi @Srikanth,

 

As Imal nicely explained, 

This issue is caused by an incorrect stream message which doesn't specify a to user or specifies an incorrect user. This will cause the Stream app running on the server to throw an error which is then written to the log file. 

 

Cheers!

Novishan

Userlevel 6
Badge +18

Hello @Srikanth,

This issue happens due to failed Stream messages. could you please check if there are any failed JMS Stream messages available in the WebLogic Admin Console? If there are any available please delete them first.

The logs are getting bigger and bigger due to the repetition of mentioned error and  it causes for a huge CPU consumption of both MainServer1 and MainServer2. As a solution, you can change the value of the parameter “Redelivery limit” by logging into the Middleware Server Admin Console. Please find below screenshot.

-Login to IFS Middleware Server Admin Console.

-Navigate to Common>JMS Modules> StreamJMSModule> StreamsJMSTopic.

-Change the value of "Redelivery Limit" from -1 to 5 -Restart the Middleware Server for the change to take effect.

 

The  default value of the field “Redelivery Limit” has been set into -1, which means unlimited retries. This would have resulted in getting logs written and consuming a large number of CPU usage as well as Logs growth. By changing the value, set to “Redelivery Limit”, the stream message would drop after several tries and the CPU utilization should come down. Hence, change the value into like 5.

Also please follow below steps as well. 

1). Stop MainServer1.

2). Delete  *.dat files under

wls_domain\<INSTANCE_NAME>\servers\MainServer1\data\store\default

wls_domain\<INSTANCE_NAME>\servers\MainServer1\data\store\diagnostics

3). Restart the server and check if the error is still available.

Please check the solutions 286167 and 281112 (UPD9) as well.

Cheers!

I will give this a try now and keep you posted. Thank you!

Userlevel 6
Badge +18

@Imal Thiunuwan The recommended fix is working nicely and we have applied it to the Production instance as well. Thanks for that.

Before I mark your response as the Best Answer, I would like to know why the Stream Messages failed in the first place. I don’t think the client is using this functionality. 

Is this a known issue in UPD8 which has been fixed in UPD9?

Any additional information will be helpful.

Regards, Srikanth

Userlevel 6
Badge +13

Hi @Srikanth,

 

Have you tried a restart of the MWS to see if it address the concern?

 

Cheers!

Novishan

Userlevel 6
Badge +18

Yes, a restart and reconfiguration of the application has been tried already. 

Userlevel 6
Badge +16

Hello @Srikanth ,

Yes, these messages are not created or assigned by a user. These are generated from WebLogic end. There is a fix available in UPD9; the solution 281112 (Bug ID: 153502). Furthermore, to figure out where this message is originating from you can execute the following query in the environment and see if it has any results.

 

SELECT * FROM fnd_stream_tab t 
WHERE t.to_user IS NULL OR
NOT EXISTS (SELECT 1 FROM fnd_user_tab u where u.identity =  t.to_user)

 

Cheers!