Skip to main content
Solved

MainServer1 logs are growing in size - Apps 10 UPD8

Srikanth
Superhero (Partner)
Forum|alt.badge.img+18

Hi,

The MainServer logs are continuously growing and causing the drive hosting IFS Application to go out of space with the following repetitive errors:

####<Jun 6, 2021, 6:16:29,948 PM PDT> <Error> <com.ifsworld.jms.IfsWebStreamsTopicBean> <DIVIFSAPP02> <MainServer1> <[ACTIVE] ExecuteThread: '27' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-55E98D048B51F37E17FE> <ea976e53-587f-48f7-854f-00e338b9f183-6d5822c7> <1623028589948> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000000> <null
java.lang.NullPointerException
> 

The MainServer1.out log can grow up to 35 GB before I manually clear it out:

 

Has someone come across the same issue. I noticed this happens in other non-production environment too but the logs aren’t growing as big & fast, as just a few users access it.

Thanks in advance!

Best answer by Imal Thiunuwan

Hello @Srikanth,

This issue happens due to failed Stream messages. could you please check if there are any failed JMS Stream messages available in the WebLogic Admin Console? If there are any available please delete them first.

The logs are getting bigger and bigger due to the repetition of mentioned error and  it causes for a huge CPU consumption of both MainServer1 and MainServer2. As a solution, you can change the value of the parameter “Redelivery limit” by logging into the Middleware Server Admin Console. Please find below screenshot.

-Login to IFS Middleware Server Admin Console.

-Navigate to Common>JMS Modules> StreamJMSModule> StreamsJMSTopic.

-Change the value of "Redelivery Limit" from -1 to 5 -Restart the Middleware Server for the change to take effect.

 

The  default value of the field “Redelivery Limit” has been set into -1, which means unlimited retries. This would have resulted in getting logs written and consuming a large number of CPU usage as well as Logs growth. By changing the value, set to “Redelivery Limit”, the stream message would drop after several tries and the CPU utilization should come down. Hence, change the value into like 5.

Also please follow below steps as well. 

1). Stop MainServer1.

2). Delete  *.dat files under

wls_domain\<INSTANCE_NAME>\servers\MainServer1\data\store\default

wls_domain\<INSTANCE_NAME>\servers\MainServer1\data\store\diagnostics

3). Restart the server and check if the error is still available.

Please check the solutions 286167 and 281112 (UPD9) as well.

Cheers!

View original
Cheers, Srikanth
T
Chanu_Yazi
2 people like this
  • T
    Tharindu Nadeeshan
  • Chanu_Yazi
    Chanu_Yazi
Did this topic help you find an answer to your question?
This topic has been closed for comments

7 replies

Novishan Dissanayake
Hero (Partner)
Forum|alt.badge.img+13

Hi @Srikanth,

 

Have you tried a restart of the MWS to see if it address the concern?

 

Cheers!

Novishan

Srikanth
Superhero (Partner)
Forum|alt.badge.img+18
  • Srikanth
  • Author
  • Superhero (Partner)
  • 291 replies
  • June 7, 2021

Yes, a restart and reconfiguration of the application has been tried already. 

Cheers, Srikanth
Imal Thiunuwan
Superhero (Employee)
Forum|alt.badge.img+16
  • Imal Thiunuwan
  • Superhero (Employee)
  • 117 replies
  • Answer
  • June 7, 2021

Hello @Srikanth,

This issue happens due to failed Stream messages. could you please check if there are any failed JMS Stream messages available in the WebLogic Admin Console? If there are any available please delete them first.

The logs are getting bigger and bigger due to the repetition of mentioned error and  it causes for a huge CPU consumption of both MainServer1 and MainServer2. As a solution, you can change the value of the parameter “Redelivery limit” by logging into the Middleware Server Admin Console. Please find below screenshot.

-Login to IFS Middleware Server Admin Console.

-Navigate to Common>JMS Modules> StreamJMSModule> StreamsJMSTopic.

-Change the value of "Redelivery Limit" from -1 to 5 -Restart the Middleware Server for the change to take effect.

 

The  default value of the field “Redelivery Limit” has been set into -1, which means unlimited retries. This would have resulted in getting logs written and consuming a large number of CPU usage as well as Logs growth. By changing the value, set to “Redelivery Limit”, the stream message would drop after several tries and the CPU utilization should come down. Hence, change the value into like 5.

Also please follow below steps as well. 

1). Stop MainServer1.

2). Delete  *.dat files under

wls_domain\<INSTANCE_NAME>\servers\MainServer1\data\store\default

wls_domain\<INSTANCE_NAME>\servers\MainServer1\data\store\diagnostics

3). Restart the server and check if the error is still available.

Please check the solutions 286167 and 281112 (UPD9) as well.

Cheers!

Charith Epitawatta
S
Novishan Dissanayake
7 people like this
  • Charith Epitawatta
    Charith Epitawatta
  • S
    Sahan Udana
  • Novishan Dissanayake
    Novishan Dissanayake
  • Ragaventhan Sathananda
    Ragaventhan Sathananda
  • Srikanth
    Srikanth
  • B
    Ben Monroe
  • Rusiiiru
    Rusiiiru
Novishan Dissanayake
Hero (Partner)
Forum|alt.badge.img+13

Hi @Srikanth,

 

As Imal nicely explained, 

This issue is caused by an incorrect stream message which doesn't specify a to user or specifies an incorrect user. This will cause the Stream app running on the server to throw an error which is then written to the log file. 

 

Cheers!

Novishan

Charith Epitawatta
Srikanth
2 people like this
  • Charith Epitawatta
    Charith Epitawatta
  • Srikanth
    Srikanth
Srikanth
Superhero (Partner)
Forum|alt.badge.img+18
  • Srikanth
  • Author
  • Superhero (Partner)
  • 291 replies
  • June 7, 2021
Imal Thiunuwan wrote:

Hello @Srikanth,

This issue happens due to failed Stream messages. could you please check if there are any failed JMS Stream messages available in the WebLogic Admin Console? If there are any available please delete them first.

The logs are getting bigger and bigger due to the repetition of mentioned error and  it causes for a huge CPU consumption of both MainServer1 and MainServer2. As a solution, you can change the value of the parameter “Redelivery limit” by logging into the Middleware Server Admin Console. Please find below screenshot.

-Login to IFS Middleware Server Admin Console.

-Navigate to Common>JMS Modules> StreamJMSModule> StreamsJMSTopic.

-Change the value of "Redelivery Limit" from -1 to 5 -Restart the Middleware Server for the change to take effect.

 

The  default value of the field “Redelivery Limit” has been set into -1, which means unlimited retries. This would have resulted in getting logs written and consuming a large number of CPU usage as well as Logs growth. By changing the value, set to “Redelivery Limit”, the stream message would drop after several tries and the CPU utilization should come down. Hence, change the value into like 5.

Also please follow below steps as well. 

1). Stop MainServer1.

2). Delete  *.dat files under

wls_domain\<INSTANCE_NAME>\servers\MainServer1\data\store\default

wls_domain\<INSTANCE_NAME>\servers\MainServer1\data\store\diagnostics

3). Restart the server and check if the error is still available.

Please check the solutions 286167 and 281112 (UPD9) as well.

Cheers!

I will give this a try now and keep you posted. Thank you!

Cheers, Srikanth
Imal Thiunuwan
1 person likes this
  • Imal Thiunuwan
    Imal Thiunuwan
Srikanth
Superhero (Partner)
Forum|alt.badge.img+18
  • Srikanth
  • Author
  • Superhero (Partner)
  • 291 replies
  • June 9, 2021

@Imal Thiunuwan The recommended fix is working nicely and we have applied it to the Production instance as well. Thanks for that.

Before I mark your response as the Best Answer, I would like to know why the Stream Messages failed in the first place. I don’t think the client is using this functionality. 

Is this a known issue in UPD8 which has been fixed in UPD9?

Any additional information will be helpful.

Regards, Srikanth

Cheers, Srikanth
Imal Thiunuwan
1 person likes this
  • Imal Thiunuwan
    Imal Thiunuwan
Imal Thiunuwan
Superhero (Employee)
Forum|alt.badge.img+16
  • Imal Thiunuwan
  • Superhero (Employee)
  • 117 replies
  • June 9, 2021

Hello @Srikanth ,

Yes, these messages are not created or assigned by a user. These are generated from WebLogic end. There is a fix available in UPD9; the solution 281112 (Bug ID: 153502). Furthermore, to figure out where this message is originating from you can execute the following query in the environment and see if it has any results.

 

SELECT * FROM fnd_stream_tab t 
WHERE t.to_user IS NULL OR
NOT EXISTS (SELECT 1 FROM fnd_user_tab u where u.identity =  t.to_user)

 

Cheers!

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings