Solved

Server error when setting password

Technical Toby
Rank
Userlevel 5
Badge +14

Trying to change the password in Aurena (IFSCLOUD) is resulting in the following error

 

Unable to fulfill the request due to unexpected server state.

        Url: https://dewbo-icl-tmi.waldbronn.polytec.de/main/ifsapplications/projection/v1/UserHandling.svc/SetUserPassword
        Date = 2022-01-07T14:06:31.530Z
        Request Id = 092ea252-4591-404a-8bdc-c6e77fc433ff
        Client version = 21.1.5.20210806072818.0

 

Unfortunately, this does not really help.
Where can I look for the proper error? Or is something wrong here?

This is a remote installed Cloud environment on customer servers.


In the log window you can see the following

15:10:12 ERROR - ODP_ILLEGAL_STATE: (Server Error) Unable to fulfill the request due to unexpected server state.

Unable to fulfill the request due to unexpected server state.

 undefined
15:10:12 ERROR - Command error: (ODP_ILLEGAL_STATE) - Unable to fulfill the request due to unexpected server state., details: undefined undefined
15:10:12 DEBUG - CMD(1): Error: Unable to fulfill the request due to unexpected server state. 
15:10:12 DEBUG - CMD(1): Done UserDetail_EnterNewPasswordCommand. 

icon

Best answer by Technical Toby 18 January 2022, 08:25

View original
Never be afraid to ask!

10 replies

Rusiru Dharmadasa
Rank
Userlevel 7
Badge +19

Can you please get the ifsapp-odata, ifsapp-iam pod logs? There must be something additional. 

Rusiru
Technical Toby
Rank
Userlevel 5
Badge +14

Deletion and reinstallation of Middletier solved the issue.

 

@Rusiru Dharmadasa 

can you please drop me a line on how to retrieve logs from pods?

 

Kind regards

Tobias

Never be afraid to ask!
Rusiru Dharmadasa
Rank
Userlevel 7
Badge +19

Hi Tobias,

 

You can use below commands.

 

kubectl get pods

kubectl describe pods <pod-name> -n <namespace>

kubectl logs <pod-name> -n <namespace>

 

Best Regards,

Rusiru

Rusiru
H
Rank
Userlevel 5
Badge +10

From 21r2 there is a tool called mtctl which can dump all logs to a folder in the ifsinstaller folder.
This is also the folder that should be zipped and attached whenever a support issue is raised on middle tier issues.

In a ifsinstaller folder just type “mtctl dump -n <namespace>” then you can brows all logs in the created <namespace>_dump folder.

Read the techdoc about mtctl.

dsj
Rank
Userlevel 7
Badge +20

Hi @Rusiru Dharmadasa 

 

We also have the same problem in a remote installed Cloud environment, and same error can be observed in different places.

Eg:

User Details, Password change, Identity and Access Manager and System parameters ...etc

Error when loading the User Details page

IFS Aurena framework version: 21.2.1.20211105061905.0

IFS Aurena client version: 21.2.1.20211105061905.0

IFS OData provider version: 21.2.1.20211112115348.0

 

In the odata pod log we could see below error.

 

{"type":"liberty_message","host":"ifsapp-odata-8bff756cb-kw7lr","ibm_userDir":"\/opt\/ifs\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"2022-01-19 09:39:06.230 ERROR  51 f1bf25b8-1348-47a2-8964-7f082e439668 com.ifsworld.fnd.provider.processor.FndErrorProcessor.processError() : Error Stack : Error while Custom Java lookup.\ncom.ifsworld.fnd.common.exceptions.ODataProviderIllegalStateException: Error while Custom Java lookup.\n\tat com.ifsworld.fnd.storage.invoker.JavaOperationInvoker.invokeOperation(JavaOperationInvoker.java:105)\n\tat com.ifsworld.fnd.storage.invoker.JavaOperationInvoker.invoke(JavaOperationInvoker.java:68)\n\tat com.ifsworld.fnd.provider.processor.FndComplexCollectionProcessor.readComplexCollection(FndComplexCollectionProcessor.java:116)\n\tat org.apache.olingo.server.core.ODataDispatcher.handleComplexDispatching(ODataDispatcher.java:409)\n\tat org.apache.olingo.server.core.ODataDispatcher.handleFunctionDispatching(ODataDispatcher.java:205)\n\tat org.apache.olingo.server.core.ODataDispatcher.handleResourceDispatching(ODataDispatcher.java:146)\n\tat org.apache.olingo.server.core.ODataDispatcher.dispatch(ODataDispatcher.java:117)\n\tat org.apache.olingo.server.core.ODataHandlerImpl.processInternal(ODataHandlerImpl.java:164)\n\tat org.apache.olingo.server.core.ODataHandlerImpl.process(ODataHandlerImpl.java:85)\n\tat com.ifsworld.fnd.provider.ODataHandlerWrapper.process(ODataHandlerWrapper.java:64)\n\tat com.ifsworld.fnd.provider.ODataHandlerWrapper.process(ODataHandlerWrapper.java:77)\n\tat com.ifsworld.fnd.web.api.FndODataServletBase.serviceODataRequest(FndODataServletBase.java:316)\n\tat com.ifsworld.fnd.web.api.FndODataServletBase.serviceProjectionRequest(FndODataServletBase.java:161)\n\tat com.ifsworld.fnd.web.api.FndODataServletBase.service(FndODataServletBase.java:111)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:791)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1257)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:745)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:442)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:182)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:93)\n\tat com.ifsworld.fnd.web.api.ResponseFilterBase.doFilter(ResponseFilterBase.java:60)\n\tat com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:90)\n\tat com.ifsworld.fnd.web.api.InitConfigurationFilterBase.doFilter(InitConfigurationFilterBase.java:194)\n\tat com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:90)\n\tat com.ibm.ws.security.jaspi.JaspiServletFilter.doFilter(JaspiServletFilter.java:56)\n\tat com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:90)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:1001)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1139)\n\tat com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:5057)\n\tat com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.handleRequest(DynamicVirtualHost.java:314)\n\tat com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1005)\n\tat com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.run(DynamicVirtualHost.java:279)\n\tat com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink$TaskWrapper.run(HttpDispatcherLink.java:1146)\n\tat com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.wrapHandlerAndExecute(HttpDispatcherLink.java:427)\n\tat com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.ready(HttpDispatcherLink.java:386)\n\tat com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:565)\n\tat com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleNewRequest(HttpInboundLink.java:499)\n\tat com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.processRequest(HttpInboundLink.java:359)\n\tat com.ibm.ws.http.channel.internal.inbound.HttpICLReadCallback.complete(HttpICLReadCallback.java:70)\n\tat com.ibm.ws.tcpchannel.internal.WorkQueueManager.requestComplete(WorkQueueManager.java:504)\n\tat com.ibm.ws.tcpchannel.internal.WorkQueueManager.attemptIO(WorkQueueManager.java:574)\n\tat com.ibm.ws.tcpchannel.internal.WorkQueueManager.workerRun(WorkQueueManager.java:958)\n\tat com.ibm.ws.tcpchannel.internal.WorkQueueManager$Worker.run(WorkQueueManager.java:1047)\n\tat com.ibm.ws.threading.internal.ExecutorServiceImpl$RunnableWrapper.run(ExecutorServiceImpl.java:239)\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.base\/java.lang.Thread.run(Unknown Source)\nCaused by: java.lang.ClassNotFoundException: com.ifsworld.projection.UserHandlingFunctionsImpl\n\tat com.ibm.ws.classloading.internal.AppClassLoader.findClassCommonLibraryClassLoaders(AppClassLoader.java:569)\n\tat com.ibm.ws.classloading.internal.AppClassLoader.findClass(AppClassLoader.java:291)\n\tat java.base\/java.lang.ClassLoader.loadClass(Unknown Source)\n\tat com.ibm.ws.classloading.internal.AppClassLoader.findOrDelegateLoadClass(AppClassLoader.java:547)\n\tat com.ibm.ws.classloading.internal.AppClassLoader.loadClass(AppClassLoader.java:505)\n\tat java.base\/java.lang.ClassLoader.loadClass(Unknown Source)\n\tat java.base\/java.lang.Class.forName0(Native Method)\n\tat java.base\/java.lang.Class.forName(Unknown Source)\n\tat com.ifsworld.fnd.storage.invoker.JavaOperationInvoker.invokeOperation(JavaOperationInvoker.java:92)\n\t... 49 more","ibm_threadId":"00000033","ibm_datetime":"2022-01-19T09:39:06.231+0000","module":"SystemErr","loglevel":"SystemErr","ibm_methodName":"","ibm_className":"","ibm_sequence":"1642585146231_0000000000076","ext_thread":"Default Executor-thread-7"}

 

DSJ - https://dsj23.me
Rusiru Dharmadasa
Rank
Userlevel 7
Badge +19

@dsj Hi Damith,

 

We have seen “Error while Custom Java lookup” before and could be resolved by restarting the relevant pod. In this case its ifsapp-odata. If this is resolved by just a restart of ifsapp-odata, that means this issue happens due to a known issue as per below. 

 

When the ifsapp-odata pod starts, the database connection is not available. So that the ifsapp-odata pod cannot fetch and register required custom java libraries existing in the database. After a while the database connection will be available but still ifsapp-odata doesn’t have the libraries as it finished starting. 

 

R&D is investigating on this issue to come up with a proper solution to avoid issues like this. 

Rusiru
dsj
Rank
Userlevel 7
Badge +20

@dsj Hi Damith,

 

We have seen “Error while Custom Java lookup” before and could be resolved by restarting the relevant pod. In this case its ifsapp-odata. If this is resolved by just a restart of ifsapp-odata, that means this issue happens due to a known issue as per below. 

 

When the ifsapp-odata pod starts, the database connection is not available. So that the ifsapp-odata pod cannot fetch and register required custom java libraries existing in the database. After a while the database connection will be available but still ifsapp-odata doesn’t have the libraries as it finished starting. 

 

R&D is investigating on this issue to come up with a proper solution to avoid issues like this. 

 

Hi @Rusiru Dharmadasa 

 

Thanks for the explanation.

Can you send the command to restart just the pod? :)

 

Cheers!
Damith

DSJ - https://dsj23.me
Rusiru Dharmadasa
Rank
Userlevel 7
Badge +19

Hi Damith,

 

Its by deleting the pod. 

 

kubectl delete pods <pod> -n <namespace>

 

Take the exact pod name using below command. 

kubectl get pods -n <namespace>

 

I use below for similar commands :) kubectl Cheat Sheet | Kubernetes

Rusiru
dsj
Rank
Userlevel 7
Badge +20

Hi @Rusiru Dharmadasa ,

 

Unfortunately the restart didn’t help. I’ve added the odata log after restart herewith if you are interested. We have created a support case and probably you’ll see in the queue :sunglasses:

Thanks a lot for your help as always, We’ll keep in touch.

 

Cheers

Damith

 

1 Attachment
DSJ - https://dsj23.me
E
Rank
Userlevel 2
Badge +4

 We discovered the same error and installation of Middletier didn't solved the problem.

We where asked to see if there was any data in table fnd_java_impl_libs and lob_file_import_tab, if so recreate the POD oData.
In our case even these tables where empty so we had to rerun db-import-data.cmd manually, it's placed at <build-home>\ifsinstaller\installers, the command are described in the documentation at https://docs.ifs.com/techdocs/21r2/020_installation/200_installing_ifs_cloud/035_ifs_cloud_ifsinstaller/050_installer_scripts/#db-import-datacmdsh. 
After running this script and restart of oData POD we managed to solve our problem.

//Leffe W

Reply


Terms & ConditionsCookie settings