Skip to main content

Trying to change the password in Aurena (IFSCLOUD) is resulting in the following error

 

Unable to fulfill the request due to unexpected server state.

        Url: https://dewbo-icl-tmi.waldbronn.polytec.de/main/ifsapplications/projection/v1/UserHandling.svc/SetUserPassword
        Date = 2022-01-07T14:06:31.530Z
        Request Id = 092ea252-4591-404a-8bdc-c6e77fc433ff
        Client version = 21.1.5.20210806072818.0

 

Unfortunately, this does not really help.
Where can I look for the proper error? Or is something wrong here?

This is a remote installed Cloud environment on customer servers.


In the log window you can see the following

15:10:12 ERROR - ODP_ILLEGAL_STATE: (Server Error) Unable to fulfill the request due to unexpected server state.

Unable to fulfill the request due to unexpected server state.

 undefined
15:10:12 ERROR - Command error: (ODP_ILLEGAL_STATE) - Unable to fulfill the request due to unexpected server state., details: undefined undefined
15:10:12 DEBUG - CMD(1): Error: Unable to fulfill the request due to unexpected server state. 
15:10:12 DEBUG - CMD(1): Done UserDetail_EnterNewPasswordCommand. 

Can you please get the ifsapp-odata, ifsapp-iam pod logs? There must be something additional. 


Deletion and reinstallation of Middletier solved the issue.

 

@Rusiru Dharmadasa 

can you please drop me a line on how to retrieve logs from pods?

 

Kind regards

Tobias


Hi Tobias,

 

You can use below commands.

 

kubectl get pods

kubectl describe pods <pod-name> -n <namespace>

kubectl logs <pod-name> -n <namespace>

 

Best Regards,

Rusiru


From 21r2 there is a tool called mtctl which can dump all logs to a folder in the ifsinstaller folder.
This is also the folder that should be zipped and attached whenever a support issue is raised on middle tier issues.

In a ifsinstaller folder just type “mtctl dump -n <namespace>” then you can brows all logs in the created <namespace>_dump folder.

Read the techdoc about mtctl.


Hi @Rusiru Dharmadasa 

 

We also have the same problem in a remote installed Cloud environment, and same error can be observed in different places.

Eg:

User Details, Password change, Identity and Access Manager and System parameters ...etc

Error when loading the User Details page

IFS Aurena framework version: 21.2.1.20211105061905.0

IFS Aurena client version: 21.2.1.20211105061905.0

IFS OData provider version: 21.2.1.20211112115348.0

 

In the odata pod log we could see below error.

 

{"type":"liberty_message","host":"ifsapp-odata-8bff756cb-kw7lr","ibm_userDir":"\/opt\/ifs\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"2022-01-19 09:39:06.230 ERROR  51 f1bf25b8-1348-47a2-8964-7f082e439668 com.ifsworld.fnd.provider.processor.FndErrorProcessor.processError() : Error Stack : Error while Custom Java lookup.\ncom.ifsworld.fnd.common.exceptions.ODataProviderIllegalStateException: Error while Custom Java lookup.\n\tat com.ifsworld.fnd.storage.invoker.JavaOperationInvoker.invokeOperation(JavaOperationInvoker.java:105)\n\tat com.ifsworld.fnd.storage.invoker.JavaOperationInvoker.invoke(JavaOperationInvoker.java:68)\n\tat com.ifsworld.fnd.provider.processor.FndComplexCollectionProcessor.readComplexCollection(FndComplexCollectionProcessor.java:116)\n\tat org.apache.olingo.server.core.ODataDispatcher.handleComplexDispatching(ODataDispatcher.java:409)\n\tat org.apache.olingo.server.core.ODataDispatcher.handleFunctionDispatching(ODataDispatcher.java:205)\n\tat org.apache.olingo.server.core.ODataDispatcher.handleResourceDispatching(ODataDispatcher.java:146)\n\tat org.apache.olingo.server.core.ODataDispatcher.dispatch(ODataDispatcher.java:117)\n\tat org.apache.olingo.server.core.ODataHandlerImpl.processInternal(ODataHandlerImpl.java:164)\n\tat org.apache.olingo.server.core.ODataHandlerImpl.process(ODataHandlerImpl.java:85)\n\tat com.ifsworld.fnd.provider.ODataHandlerWrapper.process(ODataHandlerWrapper.java:64)\n\tat com.ifsworld.fnd.provider.ODataHandlerWrapper.process(ODataHandlerWrapper.java:77)\n\tat com.ifsworld.fnd.web.api.FndODataServletBase.serviceODataRequest(FndODataServletBase.java:316)\n\tat com.ifsworld.fnd.web.api.FndODataServletBase.serviceProjectionRequest(FndODataServletBase.java:161)\n\tat com.ifsworld.fnd.web.api.FndODataServletBase.service(FndODataServletBase.java:111)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:791)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1257)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:745)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:442)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:182)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:93)\n\tat com.ifsworld.fnd.web.api.ResponseFilterBase.doFilter(ResponseFilterBase.java:60)\n\tat com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:90)\n\tat com.ifsworld.fnd.web.api.InitConfigurationFilterBase.doFilter(InitConfigurationFilterBase.java:194)\n\tat com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:90)\n\tat com.ibm.ws.security.jaspi.JaspiServletFilter.doFilter(JaspiServletFilter.java:56)\n\tat com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:90)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:1001)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1139)\n\tat com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:5057)\n\tat com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.handleRequest(DynamicVirtualHost.java:314)\n\tat com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1005)\n\tat com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.run(DynamicVirtualHost.java:279)\n\tat com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink$TaskWrapper.run(HttpDispatcherLink.java:1146)\n\tat com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.wrapHandlerAndExecute(HttpDispatcherLink.java:427)\n\tat com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.ready(HttpDispatcherLink.java:386)\n\tat com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:565)\n\tat com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleNewRequest(HttpInboundLink.java:499)\n\tat com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.processRequest(HttpInboundLink.java:359)\n\tat com.ibm.ws.http.channel.internal.inbound.HttpICLReadCallback.complete(HttpICLReadCallback.java:70)\n\tat com.ibm.ws.tcpchannel.internal.WorkQueueManager.requestComplete(WorkQueueManager.java:504)\n\tat com.ibm.ws.tcpchannel.internal.WorkQueueManager.attemptIO(WorkQueueManager.java:574)\n\tat com.ibm.ws.tcpchannel.internal.WorkQueueManager.workerRun(WorkQueueManager.java:958)\n\tat com.ibm.ws.tcpchannel.internal.WorkQueueManager$Worker.run(WorkQueueManager.java:1047)\n\tat com.ibm.ws.threading.internal.ExecutorServiceImpl$RunnableWrapper.run(ExecutorServiceImpl.java:239)\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.base\/java.lang.Thread.run(Unknown Source)\nCaused by: java.lang.ClassNotFoundException: com.ifsworld.projection.UserHandlingFunctionsImpl\n\tat com.ibm.ws.classloading.internal.AppClassLoader.findClassCommonLibraryClassLoaders(AppClassLoader.java:569)\n\tat com.ibm.ws.classloading.internal.AppClassLoader.findClass(AppClassLoader.java:291)\n\tat java.base\/java.lang.ClassLoader.loadClass(Unknown Source)\n\tat com.ibm.ws.classloading.internal.AppClassLoader.findOrDelegateLoadClass(AppClassLoader.java:547)\n\tat com.ibm.ws.classloading.internal.AppClassLoader.loadClass(AppClassLoader.java:505)\n\tat java.base\/java.lang.ClassLoader.loadClass(Unknown Source)\n\tat java.base\/java.lang.Class.forName0(Native Method)\n\tat java.base\/java.lang.Class.forName(Unknown Source)\n\tat com.ifsworld.fnd.storage.invoker.JavaOperationInvoker.invokeOperation(JavaOperationInvoker.java:92)\n\t... 49 more","ibm_threadId":"00000033","ibm_datetime":"2022-01-19T09:39:06.231+0000","module":"SystemErr","loglevel":"SystemErr","ibm_methodName":"","ibm_className":"","ibm_sequence":"1642585146231_0000000000076","ext_thread":"Default Executor-thread-7"}

 


@dsj Hi Damith,

 

We have seen “Error while Custom Java lookup” before and could be resolved by restarting the relevant pod. In this case its ifsapp-odata. If this is resolved by just a restart of ifsapp-odata, that means this issue happens due to a known issue as per below. 

 

When the ifsapp-odata pod starts, the database connection is not available. So that the ifsapp-odata pod cannot fetch and register required custom java libraries existing in the database. After a while the database connection will be available but still ifsapp-odata doesn’t have the libraries as it finished starting. 

 

R&D is investigating on this issue to come up with a proper solution to avoid issues like this. 


@dsj Hi Damith,

 

We have seen “Error while Custom Java lookup” before and could be resolved by restarting the relevant pod. In this case its ifsapp-odata. If this is resolved by just a restart of ifsapp-odata, that means this issue happens due to a known issue as per below. 

 

When the ifsapp-odata pod starts, the database connection is not available. So that the ifsapp-odata pod cannot fetch and register required custom java libraries existing in the database. After a while the database connection will be available but still ifsapp-odata doesn’t have the libraries as it finished starting. 

 

R&D is investigating on this issue to come up with a proper solution to avoid issues like this. 

 

Hi @Rusiru Dharmadasa 

 

Thanks for the explanation.

Can you send the command to restart just the pod? :)

 

Cheers!
Damith


Hi Damith,

 

Its by deleting the pod. 

 

kubectl delete pods <pod> -n <namespace>

 

Take the exact pod name using below command. 

kubectl get pods -n <namespace>

 

I use below for similar commands :) kubectl Cheat Sheet | Kubernetes


Hi @Rusiru Dharmadasa ,

 

Unfortunately the restart didn’t help. I’ve added the odata log after restart herewith if you are interested. We have created a support case and probably you’ll see in the queue :sunglasses:

Thanks a lot for your help as always, We’ll keep in touch.

 

Cheers

Damith

 


 We discovered the same error and installation of Middletier didn't solved the problem.

We where asked to see if there was any data in table fnd_java_impl_libs and lob_file_import_tab, if so recreate the POD oData.
In our case even these tables where empty so we had to rerun db-import-data.cmd manually, it's placed at <build-home>\ifsinstaller\installers, the command are described in the documentation at https://docs.ifs.com/techdocs/21r2/020_installation/200_installing_ifs_cloud/035_ifs_cloud_ifsinstaller/050_installer_scripts/#db-import-datacmdsh
After running this script and restart of oData POD we managed to solve our problem.


Reply