Solved

What is the process of loading data in batches?

  • 25 July 2022
  • 6 replies
  • 146 views
Ann Degroat
Rank
Userlevel 4
Badge +12

Partner tried to PUSH data into FSM using perform_inbound_integration to integrate large amount of data into IFS FSM. 
The data is being pushed in batches. Each batch size is 100. It was noticed that It's successfully integrates up to 15 cycles but after that it throws a “500 Internal Server Error” error.
Thus, we would like to understand how much load IFS FSM can handle?
We tried with different batch sizes and it was noticed that when the batch size increases the cycles number is reduced.
Is there a specific solution to handle this process of loading data in batches?

icon

Best answer by sachinthaw 28 July 2022, 07:25

View original
Ann DeGroat

6 replies

Jon Reid
Rank
Userlevel 6
Badge +17

the amount of load is highly dependent on the entities being created and the business rules and other automatic processes triggered.  use the TR=6 server log level to see what XML messages are being pushed into FSM and how long each takes.  I assume you are waiting some amount of time between each batch?  if so how long, and is the wait time long enough to let the server catch up after processing the previous batch.

Ann Degroat
Rank
Userlevel 4
Badge +12

It was noticed that it successfully integrates up to 15 cycles but after that, it throws a “500 Internal Server Error” error

Ann DeGroat
Jon Reid
Rank
Userlevel 6
Badge +17

this response doesn’t provide any new information.  have the customer/partner use the TR=6 server log level to see what XML messages are being pushed into FSM and how long each takes.  get the FSM server log to see the perform_inbound_integration messages and elapsed time to process each.   Do they take longer and longer to process? for the 15th batch that gives a 500 error you should be able to see the inbound XML and the exception associated with the 500 error in the server log.

S
Rank
Userlevel 1
Badge +3

Can’t we use T Table import process to import data in to FSM ? Customer first needs to insert their data to relevant T table and trigger the T Table import process.

Additionally we can use import/export wizard in smart client also to import data into FSM tables in batches.

Ann Degroat
Rank
Userlevel 4
Badge +12

The customer says “ it takes around 5 to 7 seconds process single record in xml message.
and we have more three BR which will trigger each transaction.

as suggested, currently we don't have a time gap between each batch. but we are going to have 5 seconds of time to each batch. will it help in improving the performance  or is a gap necessary?

Ann DeGroat
S
Rank
Userlevel 1
Badge +3

Adding time gap will provide app server additional time to process the last batch. However it can slow down the overall process.

Are there any errors recorded in integration log screen related failures to process integrations ?

What are errors recorded in log files of client application when receiving a http 500 error ?

HTTP 500 error can be caused by a timeout while client application (that sends xml messages) waiting for a response from server. May be we can tryout by increasing the timeout period between client and FSM server.

As requested by Jon please send server logs with TR set to 6.

Reply


Terms & ConditionsCookie settings