Solved

Prevent timeout during Application Server clone?

  • 13 December 2019
  • 6 replies
  • 903 views
NickPorter
Rank
Userlevel 6
Badge +18

When restoring an Application Server clone to create a new environment using restore_clone -instance=<NewInstance> we often - but not always - run into a timeout that prevents the process from completing cleanly.  Seems to happen around the 1 hour mark.

We have removed all of the CPU and RAM load that we can possibly do on the server and the machine is nowhere near fully utilized (only around ~20% on both while the restore is in process) but the issue remains.

Does anyone know if it is possible to extend the timeout threshold during the restore process to clone an AS cleanly?

This is with Apps10 UPD5.

Here's the output from a failed attempt and timeout:

C:\IFS\EBUPD5>.\restore_clone -instance=EBUPD5
Dec 13, 2019 3:07:25 PM ifs.application.installer.util.main.CloneMain restoreDomain
INFO: Changing instance
Dec 13, 2019 3:07:25 PM ifs.application.installer.util.main.CloneMain restoreDomain
INFO: Creating MW Home..
Dec 13, 2019 3:07:25 PM ifs.application.installer.util.main.InstallMain smokeTest
INFO: Smoke test passed!
Dec 13, 2019 3:07:25 PM ifs.application.installer.util.main.InstallMain createMwHome
INFO: Installing Java and Middleware Home at C:/IFS/EBUPD5/mw_home
Dec 13, 2019 3:07:25 PM ifs.application.installer.util.main.InstallMain createMwHome
INFO: Installing Middleware Home...
Dec 13, 2019 3:07:25 PM ifs.application.installer.util.main.InstallMain install
INFO: include MWS installation.
Dec 13, 2019 3:07:26 PM ifs.application.installer.util.ant.AntRunner$MwsAntLogger messageLogged
INFO: Installing Middleware Server (Windows)
Dec 13, 2019 3:07:26 PM ifs.application.installer.util.ant.AntRunner$MwsAntLogger messageLogged
INFO: Created dir: C:\IFS\EBUPD5\mw_home\tmp
Dec 13, 2019 3:22:36 PM ifs.application.installer.util.ant.AntRunner$MwsAntLogger messageLogged
INFO:
Dec 13, 2019 3:22:42 PM ifs.application.installer.util.ant.AntRunner$MwsAntLogger messageLogged
INFO: Deleting directory C:\IFS\EBUPD5\mw_home\tmp
Dec 13, 2019 3:22:42 PM ifs.application.installer.util.main.InstallMain install
INFO: Clean unwanted files in installation.
Dec 13, 2019 3:22:42 PM ifs.application.installer.util.main.Cleanup start
INFO: Start cleaning shortcut files after installation.
Dec 13, 2019 3:22:42 PM ifs.application.installer.util.main.Cleanup delete
INFO: delete: C:\Documents and Settings\All Users\Start Menu\Programs\Oracle FMW - 12.2.1.3.0 (33)\Eclipse.lnk
Dec 13, 2019 3:22:42 PM ifs.application.installer.util.main.Cleanup delete
INFO: delete: C:\Documents and Settings\All Users\Start Menu\Programs\Oracle FMW - 12.2.1.3.0 (33)\Online Documentation.lnk
Dec 13, 2019 3:22:42 PM ifs.application.installer.util.main.Cleanup delete
INFO: delete: C:\Documents and Settings\All Users\Start Menu\Programs\Oracle FMW - 12.2.1.3.0 (33)\Uninstall OraHome8.lnk
Dec 13, 2019 3:22:42 PM ifs.application.installer.util.main.Cleanup delete
INFO: delete: C:\Documents and Settings\All Users\Start Menu\Programs\Oracle FMW - 12.2.1.3.0 (33)
Dec 13, 2019 3:22:42 PM ifs.application.installer.util.main.InstallMain createMwHome
INFO: Middleware Home installed!
Dec 13, 2019 3:22:42 PM ifs.application.installer.util.main.CloneMain restoreDomain
INFO: MW Home created!
Dec 13, 2019 3:22:42 PM ifs.application.installer.util.main.CloneMain restoreDomain
INFO: Preparing configuration files..
Dec 13, 2019 3:22:42 PM ifs.application.installer.util.main.CloneMain replaceInstallerValues
INFO: Changing to self-signed certificate for HTTP Server
Dec 13, 2019 3:22:42 PM ifs.application.installer.util.main.CloneMain restoreDomain
INFO: Configuration files prepared
Dec 13, 2019 3:22:42 PM ifs.application.installer.util.main.CloneMain restoreDomain
INFO: Encrypting passwords..
Dec 13, 2019 3:22:42 PM ifs.application.installer.util.main.CloneMain restoreDomain
INFO: Passwords encrypted
Dec 13, 2019 3:22:42 PM ifs.application.installer.util.main.CloneMain restoreDomain
INFO: Launching installer..
Dec 13, 2019 3:22:42 PM ifs.application.installer.util.main.CloneMain restoreDomain
INFO: Waiting for process to complete...
Dec 13, 2019 4:07:25 PM ifs.application.installer.util.main.MainBase onPanic
SEVERE: Panic message: Watchdog kicked in, operation takes too long time! Aborting...
Watchdog kicked in, operation takes too long time! Aborting...

icon

Best answer by NickPorter 14 December 2019, 00:13

View original

6 replies

Himasha Abeywickrama
Rank
Userlevel 7
Badge +18

Hi @NickPorter ,

 

Could you try reinstalling the Microsoft C++ Redistributable Package and see if it resolves the issue?

https://www.linkedin.com/in/himashaabeywickrama
NickPorter
Rank
Userlevel 6
Badge +18

Since it is sporadic, I don’t believe it is C++ redist.

We also have many other AS instances that can run OK on that same server.

After shutting down everything I could, including antivirus (temporarily), I was able to get the restore process to complete cleanly before it reached the timeout threshold.

Even though there was minimal CPU and RAM load it looks like the virus scanning slowed the process of the restore down beyond the tolerance of the timeout threshold.

paguin
Rank
Userlevel 5
Badge +11

Normally its also due to firewall enabled on application server

Regards

Pankaj

E
Rank
Userlevel 4
Badge +6

You can set the watchdog timeout in mws.properties file (in ifshome/instance/<instance>/bin dir) by adding the line:
WATCHDOG_TIMEOUT=7200

/ Erik

R
Rank
Badge +2

During the clone process, is this the old instance folder where we need to add the WATCHDOG_TIMEOUT value? The new instance based on overrideConfig is not yet created, so can’t add it there.

Also, there is nothing going on when it starts the “Launching Installer” step. Everything has been disabled including the antivirus ( including antimalware service, windows server 2016). It still stops on exactly 1 hour mark.

INFO: Launching installer..
Aug 27, 2022 11:08:24 PM ifs.application.installer.util.main.CloneMain restoreDomain
INFO: Waiting for process to complete...

Any help on how to extend the time frame or why the process is just hanging without any logging is very much appreciated.

 

/Rami

E
Rank
Userlevel 4
Badge +6

Yes, you should change in the old instance folder if I remember correctly (assuming you’re changing instance name).

 

Check mws.log (in instance/logs) if that contains more information.

 

BR

/ Erik

Reply


Terms & ConditionsCookie settings