Skip to main content

We have an issue with 23r2 and it seems like it’s connected to the BPA/workflow engine spinning off and cause odata user session hanging forever in active state and consuming cpu load.
The Active sql session are running the belove sql. It dosn’t happen all the time but 10 - 20 times pr day. We have seen this issue now at 3 different production environment runing IFS 23r2s03 and su05

SELECT A.concat_order_refs, A.order_supply_demand_type_db, A.order_no, A.line_no, A.rel_no, A.part_no, A.contract, A.source_info, A.line_item_no, A.project_id, A.date_required, A.order_supply_demand_type, A.status_desc, A.qty_supply, A.qty_demand, A.qty_reserved, A.qty_pegged, A.projected_qty, A.plannable_qty, A.proj_not_res_qty, A.plan_not_res_qty, A.proj_not_peg_qty, A.qty_short, A.plan_not_peg_qty, A.proj_not_res_or_peg_qty, A.plan_not_res_or_peg_qty, A.co_plannable_qty, A.condition_code, A.project_name, A.sub_project_id, A.sub_project_description, A.activity_no, A.activity_description, A.activity_seq, A.luname, A.keyref, A.sys_guid FROM (SELECT concat_order_refs concat_order_refs, Order_Supply_Demand_Type_API.Encode(ORDER_SUPPLY_DEMAND_TYPE) order_supply_demand_type_db, order_no order_no, line_no line_no, rel_no rel_no, part_no part_no, contract contract, info source_info, line_item_no line_item_no, project_id project_id, date_required date_required, order_supply_demand_type order_supply_demand_type, status_desc status_desc, qty_supply qty_supply, qty_demand qty_demand, qty_reserved qty_reserved, qty_pegged qty_pegged, projected_qty projected_qty, plannable_qty plannable_qty, proj_not_res_qty proj_not_res_qty, plan_not_res_qty plan_not_res_qty, proj_not_peg_qty proj_not_peg_qty, qty_short qty_short, plan_not_peg_qty plan_not_peg_qty, proj_not_res_or_peg_qty proj_not_res_or_peg_qty, plan_not_res_or_peg_qty plan_not_res_or_peg_qty, co_plannable_qty co_plannable_qty, condition_code condition_code, Project_API.Get_Name(PROJECT_ID) project_name, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Sub_Project_Id(ACTIVITY_SEQ)) sub_project_id, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Sub_Project_Description(activity_seq)) sub_project_description, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Activity_No(ACTIVITY_SEQ)) activity_no, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Description(ACTIVITY_SEQ)) activity_description, activity_seq activity_seq, 'OrderSupplyDemandDetailPlanning' luname, 'SYS_GUID='||SYS_GUID()||'^' keyref, SYS_GUID() sys_guid FROM Detail_Planning WHERE ((SNAPSHOT_ID = :1 ))) A FETCH FIRST :2  ROWS ONLY, OriginalSql = SELECT A.concat_order_refs, A.order_supply_demand_type_db, A.order_no, A.line_no, A.rel_no, A.part_no, A.contract, A.source_info, A.line_item_no, A.project_id, A.date_required, A.order_supply_demand_type, A.status_desc, A.qty_supply, A.qty_demand, A.qty_reserved, A.qty_pegged, A.projected_qty, A.plannable_qty, A.proj_not_res_qty, A.plan_not_res_qty, A.proj_not_peg_qty, A.qty_short, A.plan_not_peg_qty, A.proj_not_res_or_peg_qty, A.plan_not_res_or_peg_qty, A.co_plannable_qty, A.condition_code, A.project_name, A.sub_project_id, A.sub_project_description, A.activity_no, A.activity_description, A.activity_seq, A.luname, A.keyref, A.sys_guid FROM (SELECT concat_order_refs concat_order_refs, Order_Supply_Demand_Type_API.Encode(ORDER_SUPPLY_DEMAND_TYPE) order_supply_demand_type_db, order_no order_no, line_no line_no, rel_no rel_no, part_no part_no, contract contract, info source_info, line_item_no line_item_no, project_id project_id, date_required date_required, order_supply_demand_type order_supply_demand_type, status_desc status_desc, qty_supply qty_supply, qty_demand qty_demand, qty_reserved qty_reserved, qty_pegged qty_pegged, projected_qty projected_qty, plannable_qty plannable_qty, proj_not_res_qty proj_not_res_qty, plan_not_res_qty plan_not_res_qty, proj_not_peg_qty proj_not_peg_qty, qty_short qty_short, plan_not_peg_qty plan_not_peg_qty, proj_not_res_or_peg_qty proj_not_res_or_peg_qty, plan_not_res_or_peg_qty plan_not_res_or_peg_qty, co_plannable_qty co_plannable_qty, condition_code condition_code, Project_API.Get_Name(PROJECT_ID) project_name, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Sub_Project_Id(ACTIVITY_SEQ)) sub_project_id, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Sub_Project_Description(activity_seq)) sub_project_description, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Activity_No(ACTIVITY_SEQ)) activity_no, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Description(ACTIVITY_SEQ)) activity_description, activity_seq activity_seq, 'OrderSupplyDemandDetailPlanning' luname, 'SYS_GUID='||SYS_GUID()||'^' keyref, SYS_GUID() sys_guid FROM Detail_Planning WHERE ((SNAPSHOT_ID = ?))) A FETCH FIRST ? ROWS ONLY


I tried yesterday to do a rollout restart of the odata pod but it didn’t manage to terminate the old pod. I then killed the oracle session and it went down at the end. I recordede the odata log during the terminate process


Underneath you can se the odata log from the termination process. It doesn’t manage to stop it until I kill the oracle session. You get the ORA-00028: økten er stoppet when I killed the database session and the odata pod terminates. that points against the BPA/workflow enging flipping out of control and causing the haning database sessions.

{"event.module":"liberty_message","host.name":"ifsapp-odata-6c996855dc-h7wc8","ibm_userDir":"\/opt\/ifs\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"SRVE0242I: eifs-odata-provider] d\/] pweb.CustomMetricsServlet]: Initialization successful.","ibm_threadId":"0000214f","_timestamp":"2024-06-10T14:09:29.387+0000","event.id":"SRVE0242I","module":"com.ibm.ws.webcontainer.servlet","log.level":"INFO","ibm_methodName":"init","log.logger":"com.ibm.ws.webcontainer.servlet.ServletWrapper","event.sequence":"1718028569387_000000000D670","ext_requestID":"AAGYnW4QhCm_AAAAAAACNXL","process.thread.name":"Default Executor-thread-7865"}

{"event.module":"liberty_message","host.name":"ifsapp-odata-6c996855dc-h7wc8","ibm_userDir":"\/opt\/ifs\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"2024-06-10 14:09:30.201 WARN 0BPA] 75  com.ifsworld.fnd.bpa.web.IfsBpaAsyncServletContextListener$AsyncPollingRunner.run() : Container kill signal received. Stop processing async Workflows","ibm_threadId":"0000004b","_timestamp":"2024-06-10T14:09:30.202+0000","module":"SystemOut","log.level":"SystemOut","ibm_methodName":"","log.logger":"","event.sequence":"1718028570202_000000000D671","ext_requestID":"AAGYnW4QhCm_AAAAAAACNXJ","process.thread.name":"AsyncPollingRunner"}

{"event.module":"liberty_message","host.name":"ifsapp-odata-6c996855dc-h7wc8","ibm_userDir":"\/opt\/ifs\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"TRAS0113I: Request AAGYnW4QhCm_AAAAAAACNQf on thread 00002145, which was previously detected as slow, completed after 180044.361 ms.","ibm_threadId":"00002145","_timestamp":"2024-06-10T14:10:44.374+0000","event.id":"TRAS0113I","module":"com.ibm.ws.request.timing.probeExtensionImpl.SlowRequestProbeExtension","log.level":"INFO","event.sequence":"1718028644374_000000000D672"}

{"event.module":"liberty_message","host.name":"ifsapp-odata-6c996855dc-h7wc8","ibm_userDir":"\/opt\/ifs\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"TRAS0114W: Request AAGYnW4QhCm_AAAAAAACMKb has been running on thread 00002132 for at least 1800276.697ms. The following table shows the events that have run during this request.

Duration        Operation

1800276.727ms + websphere.servlet.service | ifs-odata-provider | OData-Provider | \/InventoryPartAvailabilityPlanningAnalysis.svc\/GetSupplyDemandDetailPlanning(SnapshotId=660787)?$select=ConcatOrderRefs,OrderSupplyDemandTypeDb,OrderNo,LineNo,RelNo,PartNo,Contract,SourceInfo,LineItemNo,ProjectId,DateRequired,OrderSupplyDemandType,StatusDesc,QtySupply,QtyDemand,QtyReserved,QtyPegged,ProjectedQty,PlannableQty,ProjNotResQty,PlanNotResQty,ProjNotPegQty,QtyShort,PlanNotPegQty,ProjNotResOrPegQty,PlanNotResOrPegQty,CoPlannableQty,ConditionCode,ProjectName,SubProjectId,SubProjectDescription,ActivityNo,ActivityDescription,ActivitySeq,luname,keyref&$skip=0&$top=121

      0.998ms       websphere.datasource.psExecute | jdbc\/fndbas-odp | DECLARE

BEGIN

Bpa_Sys.Init_Session;

END;

      3.285ms       websphere.datasource.psExecute | jdbc\/fndbas-odp | {? = call Login_SYS.Init_Odp_Session_(?,?,?,?,?,?,?,?,?,?,?,?,?)}

      0.973ms       websphere.datasource.psExecuteQuery | jdbc\/fndbas-odp | select process_key, workflow_execution_db, workflow_timing_db, projection_name, action, entity_set_name, call_name, is_read_enabled, is_create_enabled, is_update_enabled, is_delete_enabled, OBJKEY   \r

from BPMN_PROJECTION\r

cross join (select ? as A, ? as B, ? as C, ? as D, ? as E,  ? as F from dual) D \r

where  (A is null or A = projection_name) \r

   AND (B is null or process_key = 😎 \r

   AND (C is null or (C = 'R' and is_read_enabled = 'TRUE') or (C = 'C' and is_create_enabled = 'TRUE') or (C = 'U' and is_update_enabled = 'TRUE') or (C = 'D' and is_delete_enabled = 'TRUE')) \r

   AND (is_enabled = 'TRUE') \r

   AND (D is null or entity_set_name = D) \r

   AND (E is null or call_name = E) \r

   AND (F is null or workflow_timing_db = F)

      0.825ms       websphere.datasource.psExecuteQuery | jdbc\/fndbas-odp | select process_key, workflow_execution_db, workflow_timing_db, projection_name, action, entity_set_name, call_name, is_read_enabled, is_create_enabled, is_update_enabled, is_delete_enabled, OBJKEY   \r

from BPMN_PROJECTION\r

cross join (select ? as A, ? as B, ? as C, ? as D, ? as E,  ? as F from dual) D \r

where  (A is null or A = projection_name) \r

   AND (B is null or process_key = 😎 \r

   AND (C is null or (C = 'R' and is_read_enabled = 'TRUE') or (C = 'C' and is_create_enabled = 'TRUE') or (C = 'U' and is_update_enabled = 'TRUE') or (C = 'D' and is_delete_enabled = 'TRUE')) \r

   AND (is_enabled = 'TRUE') \r

   AND (D is null or entity_set_name = D) \r

   AND (E is null or call_name = E) \r

   AND (F is null or workflow_timing_db = F)

      0.797ms       websphere.datasource.psExecuteQuery | jdbc\/fndbas-odp | select process_key, workflow_execution_db, workflow_timing_db, projection_name, action, entity_set_name, call_name, is_read_enabled, is_create_enabled, is_update_enabled, is_delete_enabled, OBJKEY   \r

from BPMN_PROJECTION\r

cross join (select ? as A, ? as B, ? as C, ? as D, ? as E,  ? as F from dual) D \r

where  (A is null or A = projection_name) \r

   AND (B is null or process_key = 😎 \r

   AND (C is null or (C = 'R' and is_read_enabled = 'TRUE') or (C = 'C' and is_create_enabled = 'TRUE') or (C = 'U' and is_update_enabled = 'TRUE') or (C = 'D' and is_delete_enabled = 'TRUE')) \r

   AND (is_enabled = 'TRUE') \r

   AND (D is null or entity_set_name = D) \r

   AND (E is null or call_name = E) \r

   AND (F is null or workflow_timing_db = F)

      2.206ms       websphere.datasource.psExecute | jdbc\/fndbas-odp | {? = call Login_SYS.Init_Odp_Session_(?,?,?,?,?,?,?,?,?,?,?,?,?)}

     25.345ms       websphere.datasource.psExecuteQuery | jdbc\/fndbas-odp | SELECT A.concat_order_refs, A.order_supply_demand_type_db, A.order_no, A.line_no, A.rel_no, A.part_no, A.contract, A.source_info, A.line_item_no, A.project_id, A.date_required, A.order_supply_demand_type, A.status_desc, A.qty_supply, A.qty_demand, A.qty_reserved, A.qty_pegged, A.projected_qty, A.plannable_qty, A.proj_not_res_qty, A.plan_not_res_qty, A.proj_not_peg_qty, A.qty_short, A.plan_not_peg_qty, A.proj_not_res_or_peg_qty, A.plan_not_res_or_peg_qty, A.co_plannable_qty, A.condition_code, A.project_name, A.sub_project_id, A.sub_project_description, A.activity_no, A.activity_description, A.activity_seq, A.luname, A.keyref, A.sys_guid FROM (SELECT concat_order_refs concat_order_refs, Order_Supply_Demand_Type_API.Encode(ORDER_SUPPLY_DEMAND_TYPE) order_supply_demand_type_db, order_no order_no, line_no line_no, rel_no rel_no, part_no part_no, contract contract, info source_info, line_item_no line_item_no, project_id project_id, date_required date_required, order_supply_demand_type order_supply_demand_type, status_desc status_desc, qty_supply qty_supply, qty_demand qty_demand, qty_reserved qty_reserved, qty_pegged qty_pegged, projected_qty projected_qty, plannable_qty plannable_qty, proj_not_res_qty proj_not_res_qty, plan_not_res_qty plan_not_res_qty, proj_not_peg_qty proj_not_peg_qty, qty_short qty_short, plan_not_peg_qty plan_not_peg_qty, proj_not_res_or_peg_qty proj_not_res_or_peg_qty, plan_not_res_or_peg_qty plan_not_res_or_peg_qty, co_plannable_qty co_plannable_qty, condition_code condition_code, Project_API.Get_Name(PROJECT_ID) project_name, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Sub_Project_Id(ACTIVITY_SEQ)) sub_project_id, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Sub_Project_Description(activity_seq)) sub_project_description, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Activity_No(ACTIVITY_SEQ)) activity_no, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Description(ACTIVITY_SEQ)) activity_description, activity_seq activity_seq, 'OrderSupplyDemandDetailPlanning' luname, 'SYS_GUID='||SYS_GUID()||'^' keyref, SYS_GUID() sys_guid FROM Detail_Planning WHERE ((SNAPSHOT_ID = ?))) A FETCH FIRST ? ROWS ONLY

      2.333ms       websphere.datasource.psExecute | jdbc\/fndbas-odp | {? = call Login_SYS.Init_Odp_Session_(?,?,?,?,?,?,?,?,?,?,?,?,?)}

      0.894ms       websphere.datasource.psExecute | jdbc\/fndbas-odp | DECLARE

BEGIN

? := Bpa_Sys.Get_Events;

END;","ibm_threadId":"00002157","_timestamp":"2024-06-10T14:18:46.916+0000","event.id":"TRAS0114W","module":"com.ibm.ws.request.timing.manager.HungRequestManager","log.level":"WARNING","event.sequence":"1718029126916_000000000D673"}

{"event.module":"liberty_message","host.name":"ifsapp-odata-6c996855dc-h7wc8","ibm_userDir":"\/opt\/ifs\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"J2CA0056I: The Connection Manager received a fatal connection error from the Resource Adapter for resource jdbc\/fndbas-odp. The exception is: state STATE_ACTIVE_INUSE java.sql.SQLRecoverableException: ORA-00028: økten er stoppet

:Error : 28, Position : 1232, Sql = SELECT A.concat_order_refs, A.order_supply_demand_type_db, A.order_no, A.line_no, A.rel_no, A.part_no, A.contract, A.source_info, A.line_item_no, A.project_id, A.date_required, A.order_supply_demand_type, A.status_desc, A.qty_supply, A.qty_demand, A.qty_reserved, A.qty_pegged, A.projected_qty, A.plannable_qty, A.proj_not_res_qty, A.plan_not_res_qty, A.proj_not_peg_qty, A.qty_short, A.plan_not_peg_qty, A.proj_not_res_or_peg_qty, A.plan_not_res_or_peg_qty, A.co_plannable_qty, A.condition_code, A.project_name, A.sub_project_id, A.sub_project_description, A.activity_no, A.activity_description, A.activity_seq, A.luname, A.keyref, A.sys_guid FROM (SELECT concat_order_refs concat_order_refs, Order_Supply_Demand_Type_API.Encode(ORDER_SUPPLY_DEMAND_TYPE) order_supply_demand_type_db, order_no order_no, line_no line_no, rel_no rel_no, part_no part_no, contract contract, info source_info, line_item_no line_item_no, project_id project_id, date_required date_required, order_supply_demand_type order_supply_demand_type, status_desc status_desc, qty_supply qty_supply, qty_demand qty_demand, qty_reserved qty_reserved, qty_pegged qty_pegged, projected_qty projected_qty, plannable_qty plannable_qty, proj_not_res_qty proj_not_res_qty, plan_not_res_qty plan_not_res_qty, proj_not_peg_qty proj_not_peg_qty, qty_short qty_short, plan_not_peg_qty plan_not_peg_qty, proj_not_res_or_peg_qty proj_not_res_or_peg_qty, plan_not_res_or_peg_qty plan_not_res_or_peg_qty, co_plannable_qty co_plannable_qty, condition_code condition_code, Project_API.Get_Name(PROJECT_ID) project_name, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Sub_Project_Id(ACTIVITY_SEQ)) sub_project_id, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Sub_Project_Description(activity_seq)) sub_project_description, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Activity_No(ACTIVITY_SEQ)) activity_no, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Description(ACTIVITY_SEQ)) activity_description, activity_seq activity_seq, 'OrderSupplyDemandDetailPlanning' luname, 'SYS_GUID='||SYS_GUID()||'^' keyref, SYS_GUID() sys_guid FROM Detail_Planning WHERE ((SNAPSHOT_ID = :1 ))) A FETCH FIRST :2  ROWS ONLY, OriginalSql = SELECT A.concat_order_refs, A.order_supply_demand_type_db, A.order_no, A.line_no, A.rel_no, A.part_no, A.contract, A.source_info, A.line_item_no, A.project_id, A.date_required, A.order_supply_demand_type, A.status_desc, A.qty_supply, A.qty_demand, A.qty_reserved, A.qty_pegged, A.projected_qty, A.plannable_qty, A.proj_not_res_qty, A.plan_not_res_qty, A.proj_not_peg_qty, A.qty_short, A.plan_not_peg_qty, A.proj_not_res_or_peg_qty, A.plan_not_res_or_peg_qty, A.co_plannable_qty, A.condition_code, A.project_name, A.sub_project_id, A.sub_project_description, A.activity_no, A.activity_description, A.activity_seq, A.luname, A.keyref, A.sys_guid FROM (SELECT concat_order_refs concat_order_refs, Order_Supply_Demand_Type_API.Encode(ORDER_SUPPLY_DEMAND_TYPE) order_supply_demand_type_db, order_no order_no, line_no line_no, rel_no rel_no, part_no part_no, contract contract, info source_info, line_item_no line_item_no, project_id project_id, date_required date_required, order_supply_demand_type order_supply_demand_type, status_desc status_desc, qty_supply qty_supply, qty_demand qty_demand, qty_reserved qty_reserved, qty_pegged qty_pegged, projected_qty projected_qty, plannable_qty plannable_qty, proj_not_res_qty proj_not_res_qty, plan_not_res_qty plan_not_res_qty, proj_not_peg_qty proj_not_peg_qty, qty_short qty_short, plan_not_peg_qty plan_not_peg_qty, proj_not_res_or_peg_qty proj_not_res_or_peg_qty, plan_not_res_or_peg_qty plan_not_res_or_peg_qty, co_plannable_qty co_plannable_qty, condition_code condition_code, Project_API.Get_Name(PROJECT_ID) project_name, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Sub_Project_Id(ACTIVITY_SEQ)) sub_project_id, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Sub_Project_Description(activity_seq)) sub_project_description, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Activity_No(ACTIVITY_SEQ)) activity_no, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Description(ACTIVITY_SEQ)) activity_description, activity_seq activity_seq, 'OrderSupplyDemandDetailPlanning' luname, 'SYS_GUID='||SYS_GUID()||'^' keyref, SYS_GUID() sys_guid FROM Detail_Planning WHERE ((SNAPSHOT_ID = ?))) A FETCH FIRST ? ROWS ONLY, Error Msg = ORA-00028: økten er stoppet

","ibm_threadId":"00002132","_timestamp":"2024-06-10T14:22:10.095+0000","event.id":"J2CA0056I","module":"com.ibm.ejs.j2c.ConnectionEventListener","log.level":"AUDIT","event.sequence":"1718029330095_000000000D674"}

{"event.module":"liberty_message","host.name":"ifsapp-odata-6c996855dc-h7wc8","ibm_userDir":"\/opt\/ifs\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"2024-06-10 14:22:10.098 ERROR 3WEBAPP] 8498 b47df5a0-215b-46ee-8fa0-e066524c3105 com.ifsworld.fnd.web.api.FndODataServletBase.service() : Server Error occurred in OData Servlet.

com.ifsworld.fnd.common.exceptions.DatabaseResultSetException: Error while mapping Entity Collection Stream.

\tat com.ifsworld.fnd.storage.mapper.EntityCollectionStreamDataMapper$EntityIteratorImpl.hasNext(EntityCollectionStreamDataMapper.java:86)

\tat org.apache.olingo.server.core.serializer.json.ODataJsonSerializer.writeEntitySet(ODataJsonSerializer.java:340)

\tat org.apache.olingo.server.core.serializer.json.ODataJsonSerializer.entityCollectionIntoStream(ODataJsonSerializer.java:282)

\tat org.apache.olingo.server.core.serializer.stream.EntityStreamContent.writeInternal(EntityStreamContent.java:46)

\tat org.apache.olingo.server.core.serializer.stream.StreamContent.writeCollection(StreamContent.java:74)

\tat org.apache.olingo.server.core.serializer.stream.StreamContent.write(StreamContent.java:69)

\tat com.ifsworld.fnd.provider.util.ODataHttpHandlerUtil.writeContent(ODataHttpHandlerUtil.java:296)

\tat com.ifsworld.fnd.provider.util.ODataHttpHandlerUtil.convertToHttp(ODataHttpHandlerUtil.java:224)

\tat com.ifsworld.fnd.web.api.FndODataServletBase.serviceProjectionRequest(FndODataServletBase.java:176)

\tat com.ifsworld.fnd.web.api.FndODataServletBase.service(FndODataServletBase.java:114)

\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:791)

\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1258)

\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:746)

\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:443)

\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:193)

\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:98)

\tat com.ifsworld.fnd.web.api.ResponseFilterBase.doFilter(ResponseFilterBase.java:60)

\tat com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)

\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)

\tat com.ifsworld.fnd.web.api.InitConfigurationFilterBase.doFilter(InitConfigurationFilterBase.java:202)

\tat com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)

\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)

\tat com.ibm.ws.security.jaspi.JaspiServletFilter.doFilter(JaspiServletFilter.java:56)

\tat com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)

\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)

\tat com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:1002)

\tat com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1140)

\tat com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:5049)

\tat com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.handleRequest(DynamicVirtualHost.java:316)

\tat com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1007)

\tat com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.run(DynamicVirtualHost.java:281)

\tat com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink$TaskWrapper.run(HttpDispatcherLink.java:1184)

\tat com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.wrapHandlerAndExecute(HttpDispatcherLink.java:453)

\tat com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.ready(HttpDispatcherLink.java:412)

\tat com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:566)

\tat com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleNewRequest(HttpInboundLink.java:500)

\tat com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.processRequest(HttpInboundLink.java:360)

\tat com.ibm.ws.http.channel.internal.inbound.HttpICLReadCallback.complete(HttpICLReadCallback.java:70)

\tat com.ibm.ws.tcpchannel.internal.WorkQueueManager.requestComplete(WorkQueueManager.java:504)

\tat com.ibm.ws.tcpchannel.internal.WorkQueueManager.attemptIO(WorkQueueManager.java:574)

\tat com.ibm.ws.tcpchannel.internal.WorkQueueManager.workerRun(WorkQueueManager.java:958)

\tat com.ibm.ws.tcpchannel.internal.WorkQueueManager$Worker.run(WorkQueueManager.java:1047)

\tat com.ibm.ws.threading.internal.ExecutorServiceImpl$RunnableWrapper.run(ExecutorServiceImpl.java:238)

\tat java.base\/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

\tat java.base\/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

\tat java.base\/java.lang.Thread.run(Unknown Source)

Caused by: java.sql.SQLRecoverableException: ORA-00028: økten er stoppet

\tat oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:526)

\tat oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:462)

\tat oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1104)

\tat oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:553)

\tat oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:269)

\tat oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:655)

\tat oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:270)

\tat oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:1079)

\tat oracle.jdbc.driver.OracleStatement.fetchMoreRows(OracleStatement.java:3469)

\tat oracle.jdbc.driver.InsensitiveScrollableResultSet.fetchMoreRows(InsensitiveScrollableResultSet.java:742)

\tat oracle.jdbc.driver.InsensitiveScrollableResultSet.absoluteInternal(InsensitiveScrollableResultSet.java:698)

\tat oracle.jdbc.driver.InsensitiveScrollableResultSet.next(InsensitiveScrollableResultSet.java:412)

\tat oracle.ucp.jdbc.proxy.oracle$1ucp$1jdbc$1proxy$1oracle$1ResultSetProxy$2oracle$1jdbc$1internal$1OracleResultSet$$$Proxy.next(Unknown Source)

\tat com.ibm.ws.rsadapter.jdbc.WSJdbcResultSet.next(WSJdbcResultSet.java:2253)

\tat com.ifsworld.fnd.storage.mapper.EntityCollectionStreamDataMapper$EntityIteratorImpl.hasNext(EntityCollectionStreamDataMapper.java:84)

\t... 45 more

Caused by: Error : 28, Position : 1232, Sql = SELECT A.concat_order_refs, A.order_supply_demand_type_db, A.order_no, A.line_no, A.rel_no, A.part_no, A.contract, A.source_info, A.line_item_no, A.project_id, A.date_required, A.order_supply_demand_type, A.status_desc, A.qty_supply, A.qty_demand, A.qty_reserved, A.qty_pegged, A.projected_qty, A.plannable_qty, A.proj_not_res_qty, A.plan_not_res_qty, A.proj_not_peg_qty, A.qty_short, A.plan_not_peg_qty, A.proj_not_res_or_peg_qty, A.plan_not_res_or_peg_qty, A.co_plannable_qty, A.condition_code, A.project_name, A.sub_project_id, A.sub_project_description, A.activity_no, A.activity_description, A.activity_seq, A.luname, A.keyref, A.sys_guid FROM (SELECT concat_order_refs concat_order_refs, Order_Supply_Demand_Type_API.Encode(ORDER_SUPPLY_DEMAND_TYPE) order_supply_demand_type_db, order_no order_no, line_no line_no, rel_no rel_no, part_no part_no, contract contract, info source_info, line_item_no line_item_no, project_id project_id, date_required date_required, order_supply_demand_type order_supply_demand_type, status_desc status_desc, qty_supply qty_supply, qty_demand qty_demand, qty_reserved qty_reserved, qty_pegged qty_pegged, projected_qty projected_qty, plannable_qty plannable_qty, proj_not_res_qty proj_not_res_qty, plan_not_res_qty plan_not_res_qty, proj_not_peg_qty proj_not_peg_qty, qty_short qty_short, plan_not_peg_qty plan_not_peg_qty, proj_not_res_or_peg_qty proj_not_res_or_peg_qty, plan_not_res_or_peg_qty plan_not_res_or_peg_qty, co_plannable_qty co_plannable_qty, condition_code condition_code, Project_API.Get_Name(PROJECT_ID) project_name, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Sub_Project_Id(ACTIVITY_SEQ)) sub_project_id, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Sub_Project_Description(activity_seq)) sub_project_description, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Activity_No(ACTIVITY_SEQ)) activity_no, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Description(ACTIVITY_SEQ)) activity_description, activity_seq activity_seq, 'OrderSupplyDemandDetailPlanning' luname, 'SYS_GUID='||SYS_GUID()||'^' keyref,","ibm_threadId":"00002132","_timestamp":"2024-06-10T14:22:10.098+0000","module":"SystemErr","log.level":"SystemErr","ibm_methodName":"","log.logger":"","event.sequence":"1718029330098_000000000D675","ext_requestID":"AAGYnW4QhCm_AAAAAAACMKb","process.thread.name":"Default Executor-thread-7843"}

{"event.module":"liberty_message","host.name":"ifsapp-odata-6c996855dc-h7wc8","ibm_userDir":"\/opt\/ifs\/wlp\/usr\/","ibm_serverName":"defaultServer","message":" SYS_GUID() sys_guid FROM Detail_Planning WHERE ((SNAPSHOT_ID = :1 ))) A FETCH FIRST :2  ROWS ONLY, OriginalSql = SELECT A.concat_order_refs, A.order_supply_demand_type_db, A.order_no, A.line_no, A.rel_no, A.part_no, A.contract, A.source_info, A.line_item_no, A.project_id, A.date_required, A.order_supply_demand_type, A.status_desc, A.qty_supply, A.qty_demand, A.qty_reserved, A.qty_pegged, A.projected_qty, A.plannable_qty, A.proj_not_res_qty, A.plan_not_res_qty, A.proj_not_peg_qty, A.qty_short, A.plan_not_peg_qty, A.proj_not_res_or_peg_qty, A.plan_not_res_or_peg_qty, A.co_plannable_qty, A.condition_code, A.project_name, A.sub_project_id, A.sub_project_description, A.activity_no, A.activity_description, A.activity_seq, A.luname, A.keyref, A.sys_guid FROM (SELECT concat_order_refs concat_order_refs, Order_Supply_Demand_Type_API.Encode(ORDER_SUPPLY_DEMAND_TYPE) order_supply_demand_type_db, order_no order_no, line_no line_no, rel_no rel_no, part_no part_no, contract contract, info source_info, line_item_no line_item_no, project_id project_id, date_required date_required, order_supply_demand_type order_supply_demand_type, status_desc status_desc, qty_supply qty_supply, qty_demand qty_demand, qty_reserved qty_reserved, qty_pegged qty_pegged, projected_qty projected_qty, plannable_qty plannable_qty, proj_not_res_qty proj_not_res_qty, plan_not_res_qty plan_not_res_qty, proj_not_peg_qty proj_not_peg_qty, qty_short qty_short, plan_not_peg_qty plan_not_peg_qty, proj_not_res_or_peg_qty proj_not_res_or_peg_qty, plan_not_res_or_peg_qty plan_not_res_or_peg_qty, co_plannable_qty co_plannable_qty, condition_code condition_code, Project_API.Get_Name(PROJECT_ID) project_name, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Sub_Project_Id(ACTIVITY_SEQ)) sub_project_id, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Sub_Project_Description(activity_seq)) sub_project_description, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Activity_No(ACTIVITY_SEQ)) activity_no, DECODE(ACTIVITY_SEQ, 0, NULL, Activity_API.Get_Description(ACTIVITY_SEQ)) activity_description, activity_seq activity_seq, 'OrderSupplyDemandDetailPlanning' luname, 'SYS_GUID='||SYS_GUID()||'^' keyref, SYS_GUID() sys_guid FROM Detail_Planning WHERE ((SNAPSHOT_ID = ?))) A FETCH FIRST ? ROWS ONLY, Error Msg = ORA-00028: økten er stoppet

\tat oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:530)

\t... 59 more","ibm_threadId":"00002132","_timestamp":"2024-06-10T14:22:10.098+0000","module":"SystemErr","log.level":"SystemErr","ibm_methodName":"","log.logger":"","event.sequence":"1718029330098_000000000D676","ext_requestID":"AAGYnW4QhCm_AAAAAAACMKb","process.thread.name":"Default Executor-thread-7843"}

….

From this log it is not possible to narrow down an issue related to Workflows. I suggest raising a support ticket to IFS to further investigate. 


@Lahirumala de Mel we working on creating a support support case. We have seen the same sql, behavior at 3 customers running 23r2 (no issues when they used 22r2. I suppose this is hitting a lot of customers running 23r2 and uses  InventoryPartAvailabilityPlanningAnalysis.svc 

 

If the sessions isn’t killed it will at the end with a not responsive system whit the database host out of cpu capasity. The query timeout isn’t able to kill the database query either and that why we think it caused by the BPA/Workflow as that is stated in the odata log when we kill the session. For two of the customer they have never touched or added workflows.

 

 


Hi ​@kjro! Did you find a solution to this issue? We are running into the exact same problem with our customer, the same SQL query gets stuck and once they accumulate, they eat up all of the processing power of the database.


@Mikko1234 We reported it as an case  and ifs resolved it as it was a bug in IFS. The bugfixed reverted a bug correction they had done on a earlier SU. 
A later su on 23.r2 or upgrade to 24.r.1 will fix this issue. 

I the meantime develop a script that kills this hanging db sessions is another option until you get the su or ru installed

Kjell Åge


Reply