Why is an on prem gateway necessary for the connection between ADLS Parquet datasets and the end user’s Power BI instance?
Solved
ADLS Gateway Question
+3Best answer by AsiriEkanayake
Yes, however, by introducing a Fabric Lakehouse as an intermediary between ADLS Gen2 and Power BI, wouldn’t we avoid the need for any on prem infrastructure? IFS would still land Parquet data in ADLS as the supported extraction mechanism, but Power BI wouldn’t connect to ADLS directly. Instead, data would be ingested into Fabric Lakehouse to be secured, typed, and modeled. Power BI would then consume data natively from Fabric using Direct Lake or the SQL endpoint.
Am I incorrect in the assumption that this would avoid the need for on prem infrastructure?
Yes a Fabric Lakehouse would avoid the requirement to use an on-prem gateway but it is not supported by our architecture at the moment; features such as the 'Advanced Refresh' of Analysis Models - Power Bi reports in IFS Cloud Web would not be possible in our current solution with a Lakehouse acting as the intermediary. But we are looking into improvements related to this.
However, if you don’t plan to use Analysis Models - Power BI in IFS Cloud, the BI setup for the “Datalake only” solution described here may be suitable for your requirements: BI Infrastructure Setup: Analysis Models - Power BI. In this setup, data would be pumped to ADLS and you can consume the data from there as required.
+1Hello! There are a few reasons an on-prem data gateway is needed
- Power BI dataset requirements; The IFS Analysis Models Power BI solution creates datasets that reference Parquet files in ADLS Gen2. To refresh these datasets via Power BI Service, their data sources must be bound to a gateway and this is done through the customer’s Service Principal.
- Security; ADLS Gen2 endpoint is not publicly accessible and ADLS cannot reliably whitelist Power BI IPs, hence, a gateway is needed for the Power BI Service to safely connect with ADLS
- Our automation (binding data sources during upload of reports) relies on the setup where data gateway is used. Without this, it is not possible to upload reports in our solution.
We are planning to investigate other options to refresh datasets, and redesign, and improve our architecture.
+3Hello! There are a few reasons an on-prem data gateway is needed
- Power BI dataset requirements; The IFS Analysis Models Power BI solution creates datasets that reference Parquet files in ADLS Gen2. To refresh these datasets via Power BI Service, their data sources must be bound to a gateway and this is done through the customer’s Service Principal.
- Security; ADLS Gen2 endpoint is not publicly accessible and ADLS cannot reliably whitelist Power BI IPs, hence, a gateway is needed for the Power BI Service to safely connect with ADLS
- Our automation (binding data sources during upload of reports) relies on the setup where data gateway is used. Without this, it is not possible to upload reports in our solution.
We are planning to investigate other options to refresh datasets, and redesign, and improve our architecture.
Yes, however, by introducing a Fabric Lakehouse as an intermediary between ADLS Gen2 and Power BI, wouldn’t we avoid the need for any on prem infrastructure? IFS would still land Parquet data in ADLS as the supported extraction mechanism, but Power BI wouldn’t connect to ADLS directly. Instead, data would be ingested into Fabric Lakehouse to be secured, typed, and modeled. Power BI would then consume data natively from Fabric using Direct Lake or the SQL endpoint.
Am I incorrect in the assumption that this would avoid the need for on prem infrastructure?
+1Yes, however, by introducing a Fabric Lakehouse as an intermediary between ADLS Gen2 and Power BI, wouldn’t we avoid the need for any on prem infrastructure? IFS would still land Parquet data in ADLS as the supported extraction mechanism, but Power BI wouldn’t connect to ADLS directly. Instead, data would be ingested into Fabric Lakehouse to be secured, typed, and modeled. Power BI would then consume data natively from Fabric using Direct Lake or the SQL endpoint.
Am I incorrect in the assumption that this would avoid the need for on prem infrastructure?
Yes a Fabric Lakehouse would avoid the requirement to use an on-prem gateway but it is not supported by our architecture at the moment; features such as the 'Advanced Refresh' of Analysis Models - Power Bi reports in IFS Cloud Web would not be possible in our current solution with a Lakehouse acting as the intermediary. But we are looking into improvements related to this.
However, if you don’t plan to use Analysis Models - Power BI in IFS Cloud, the BI setup for the “Datalake only” solution described here may be suitable for your requirements: BI Infrastructure Setup: Analysis Models - Power BI. In this setup, data would be pumped to ADLS and you can consume the data from there as required.
+3Yes, however, by introducing a Fabric Lakehouse as an intermediary between ADLS Gen2 and Power BI, wouldn’t we avoid the need for any on prem infrastructure? IFS would still land Parquet data in ADLS as the supported extraction mechanism, but Power BI wouldn’t connect to ADLS directly. Instead, data would be ingested into Fabric Lakehouse to be secured, typed, and modeled. Power BI would then consume data natively from Fabric using Direct Lake or the SQL endpoint.
Am I incorrect in the assumption that this would avoid the need for on prem infrastructure?
Yes a Fabric Lakehouse would avoid the requirement to use an on-prem gateway but it is not supported by our architecture at the moment; features such as the 'Advanced Refresh' of Analysis Models - Power Bi reports in IFS Cloud Web would not be possible in our current solution with a Lakehouse acting as the intermediary. But we are looking into improvements related to this.
However, if you don’t plan to use Analysis Models - Power BI in IFS Cloud, the BI setup for the “Datalake only” solution described here may be suitable for your requirements: BI Infrastructure Setup: Analysis Models - Power BI. In this setup, data would be pumped to ADLS and you can consume the data from there as required.
Ah I see, thank you for the clarification! :)
+4
+1
Unfortunately, at the moment we do not have a feature to test the connectivity in the Setup Parameters page so it is a matter of putting thee correct parameter details and secrets.
Since you are using a Data Lake only option, if a Parquet Data Source load shows “Error”, the relevant logs page in IFS Cloud can give an idea of the issue. Note that navigating to logs via screens and log buttons filters the logs, meaning such logs are context-aware and specific to the selected data source/ workload job/ analysis model.. If nothing amiss is shown in these filtered logs, then directly navigating to the logs page via the Navigator panel can reveal issues such as errors with encrypted data or accounts being locked, which are not related to any specific data source, workload job or analysis mode. In addition to that, can also check the logs in the ifsapp-scheduler and ifsapp-am-datapump pods as well.
+4
Unfortunately, at the moment we do not have a feature to test the connectivity in the Setup Parameters page so it is a matter of putting thee correct parameter details and secrets.
Since you are using a Data Lake only option, if a Parquet Data Source load shows “Error”, the relevant logs page in IFS Cloud can give an idea of the issue. Note that navigating to logs via screens and log buttons filters the logs, meaning such logs are context-aware and specific to the selected data source/ workload job/ analysis model.. If nothing amiss is shown in these filtered logs, then directly navigating to the logs page via the Navigator panel can reveal issues such as errors with encrypted data or accounts being locked, which are not related to any specific data source, workload job or analysis mode. In addition to that, can also check the logs in the ifsapp-scheduler and ifsapp-am-datapump pods as well.
Thank you for getting back to me, that makes sense.
My assumption then is if I run a test data load from either an explicit load or via workload job and get no logs and no errors that there is an issue with IFS reaching ADLS? I currently get nothing in data service logs or workload logs.
Appreciate the help!
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.