Question

APPS9 UD17 failed updating AS, MWS directory missing

  • 16 July 2021
  • 6 replies
  • 328 views

Userlevel 3
Badge +7

Good day.

 

I have been trying to apply update 17 to our APPS9 UD14 instances (I have tried 4 different instances) and he end result is the same.

Failed to update AS & the MWS directory never recreates.

 

 

Db space seems fine

Application server has 40G free on C: and 65G on D where ifshome resides.

 

The application runs fine prior to the update.

I run reconfiguration prior to the update

The database updates with no errors.

This appears as a Middleware issue.

 

Any clue as to why AS will not startup?

 

Prior to executing the update install the following services are stopped

IFSConnect

IFS Solr

ManagedServer1

 

AdminServer is left running

 

I appreciate any insight.

 

Thank you,

Marjie

 


This topic has been closed for comments

6 replies

Userlevel 3
Badge +5

Hi @CUCSOLUTIONS 

There could be several entries under the <HOME_LIST> tag because, it seems you have several IFS_HOME instances in your environment. But make sure, REMOVE=”T” is not inside any instances, listed under tag <HOME_IDX ….>.

Incase if you have stopped all the servers, make sure node_manager and adminserver are running :)

 

Userlevel 3
Badge +5

Hi @CUCSOLUTIONS,

Well, regarding the following question you asked, 

On item #3 - how do you know if this file is corrupted? I do see references there to old environments, but existing environment entries look good.

Following is an example of a healthy inventory.xml file.

If this is corrupted, the <HOME_LIST> will not be appear according to the above format and most probably at the end of <HOME_IDX tag of the corrupted IFS_HOME instance, there will be an additional REMOVE=”T” tag. So to run the installation in a healthy manner, you can replace it with the backups which is in path ex: C:\Program Files\Oracle\Inventory\backup. 

 

Userlevel 5
Badge +8

Hi @CUCSOLUTIONS ,


Please check all log files and follow below steps before run installation 

  1. Check whether there is a mws_bak folder inside IFS_HOME -> mw_home. Please restore ifs_home using backup
  2. ​​​​​​Check if there are any files in following folder: <<IFS_HOME>>\wls_domain\<<INSTANCE_NAME>>\pending. If it is empty you can proceed with deployment however on finding anything present, please make sure to run reconfiguration in installer and validate that all content of pending folder is removed.
  3.  Check the inventory.xml(Location - C:\Program Files\Oracle\Inventory\ContentsXML) is not corrupted. if it is corrupted restore the oracle inventory file from backups
  4. Restart the VM and start the all services and reconfiguration finally do the application test

Thanks,
​​​​​​​Kalana

 

 

Userlevel 3
Badge +7

Kalana;

Sorry for the delayed response.

When you refer to checking log files prior to installing, which of these files am I to check and what am I looking for?

The environments run fine with no errors, however I must be missing something as I have been unable to apply this update to any of the 4 dev/test environments without them failing at the same point in time.

Prior to applying the update I verified #1 & 2 look good. no mws_bak file and no items in \pending folder.

On item #3 - how do you know if this file is corrupted? I do see references there to old environments, but existing environment entries look good.

 

I applied the update again today and it once again failed in the same place, including a missing mws folder under mw_home. Thus a web logic error trying to start services even after reboot.

While the update ran, I monitored resources to ensure I didn’t max out on memory or processor resources

Is there something in update 15 or 16 that may reference changes that I didn’t see in the UD17 installation? 

Any processes or tricks to applying updates I can refer to would be truly appreciated.

Thank you

 

 

Userlevel 3
Badge +7

Hiroshi;

Thank you very much for your feedback.

WOW - there were several entries and I had to go back to 2018 to find a non-corrupted inventory file.

So what creates the corruption in the Oracle Inventory?

I plan to try and apply an update today with hopes I am successful.

More to follow.

Userlevel 3
Badge +7

Ok group.

 

I tried another update and it failed once again with unable to update AS and the mws directory never rebuilt.

Here is what was seen during Install.

Reported error is: The following error occurred while executing this line:
D:\ifs\extsvr\SANITY\repository\server\install_fndmws.xml:457: Java returned: 1
Finalize process IFS Base Functionality 6.0.0
Configuring Oracle Users
Configuring Remote Connect Server
Error in Ant-script: server\install_fndbas.xml
Reported error is: The following error occurred while executing this line:
D:\ifs\extsvr\SANITY\repository\server\install_fndbas.xml:1364: D:\ifs\extsvr\SANITY\mw_home\mws\wlserver\server\lib does not exist.
Creating the Key Store
Configuring PrintAgent
Configuring Services
Starting ConnectServer Service
Adding Application Links to Index Page
Finalize process Documentation 
Indexing script started in a separate window
Finalize process Installer 6.0.0
-------------------------------------------------- 
Installation has partly failed, check the logfile.
Logfile for Installation is:
 D:\ifs\extsvr\SANITY\installer.log
Logfile for Instance SANITY is:
 D:\ifs\extsvr\SANITY\instance\SANITY\SANITY_configuration.log

 

 

I ensured the Oracle Inventory was not corrupted (fixed)

Ensured the IFS application was functioning properly with no errors.

Recompiled invalid objects as appowner & as sys using plsql

Disconnected from plsql

Set job_queue_processes=0

Started all servers as Administrator

Stopped SOLR

Stopped Connect

Stopped ManagedServer1 prior to executing

db install went fine - no errors

 

So now where to look?

I did see a reference to use of Antimalware/Antivirus.

I’m on a VM running windows server 2012 R2, plenty of RAM, plenty or hard drive space.

Confirmed with IFS file structure is the same as the ones provided.

We are running Anti-virus applications (Carbon Black, Ivanti Security and McAfee Agent - none of which I can disable)

I have created an ifsdump file prior to rolling back the VM.

Where do I start to dissect why this may be occurring?

I feel like I am missing something here. I am going from APPS9 UD14 to APPS9 UD17

I do appreciate any assistance one can provide.

 

If the db updates fine, can I keep the updated db and just rollback the mws to try installing again? 

I keep rolling back the db when I roll back the mws to ensure the environment works well.

Thank you