Solved

Incremental load outside Azure

  • 19 March 2024
  • 2 replies
  • 29 views
T
Rank
Badge +3

We are running IFS Cloud 23R1 in a, by IFS, hosted setup with self hosted BI setup in our own Azure tenant. However, our main datalake resides in Snowflake and we’re therefore pushing data from Azure to Snowflake. Currently this is done with full dataloads each time, which is not a sustainable model.

We have enabled incremental loads between IFS Cloud and our self hosted BI (Azure). We are now looking into what we can utilize to enable incremental loads between Azure and Snowflake, and noticed that mvt_created_dt was populated in Azure. However it seem to be updated with the time of the latest incremental load, for all records? My exceptation was that this would only occur for newly added/updated records.

Does anyone have any suggestion how to implement a solid incremental load funcationality between Azure and another datalake?

icon

Best answer by subashfestus 19 March 2024, 15:15

View original

2 replies

S
Rank
Badge +4

Hi @TIVMAGFRI,

mvt_created_dt column originates from the oracle incremental load related table for the particular information source. mvt_created_dt will get the timestamp of when the incremental load executed on the oracle side data mart table.

Eg :- GL_TRANSACTION_MVT is the data mart related table for FactGlTransaction.

https://docs.ifs.com/techdocs/23r2/050_reporting/500_information_sources/050_data_mart_access/010_overview/015_incr_load_of_is/

https://docs.ifs.com/techdocs/23r2/050_reporting/500_information_sources/050_data_mart_access/040_handling_incremental_load/

If you correctly configure the incremental load for the data mart, then the mvt_created_dt will get the correct timestamp for the new and modified records. Then that value of mvt_created_dt will flow to the SQL server DB through the data load run process.

If you are seeing same mvt_created_dt for all records I guess that means the data mart table in Oracle is always getting a full load instead of an incremental load.

Eg :- Check records in GL_TRANSACTION_MVT or FACT_GL_TRANSACTION_DM to see the value of mvt_created_dt for all rows.

T
Rank
Badge +3

Hi subashfestus,

 

Thanks for a very good answer. It looks like the consultants that help us to set this up did something wrong in the configuration.

Reply


Terms & ConditionsCookie settings