Is it possible to insert data into fabric datawarehouse using a script that isnt in fabric notebook?

[eduardo_schmitz]

I have a small pipeline that gets data from mongo using lambda functions and glue and save these files in S3. Then i get the parquet files in S3 using fabric connectors. But I think It would be better to simply use the script to insert data into my fabric DW. Is there a way to do it from AWS?

[v-veshwara-msft]

Hi @eduardo_schmitz ,

We’re following up once more regarding your query. If it has been resolved, please mark the helpful reply as the Accepted Solution to assist others facing similar challenges.

If you still need assistance, please let us know.
Thank you.

[alia12]

Yes, it's possible to insert data into Microsoft Fabric Data Warehouse without using Fabric Notebooks. Since your pipeline already retrieves data from MongoDB using AWS Lambda and AWS Glue, you have a few options to push data directly into Fabric DW:

1. You can set up an ODBC/JDBC connection from AWS to Fabric DW and execute INSERT INTO or MERGE statements. This method requires configuring a secure connection on the AWS side.

2. If you’re using Azure services in combination with AWS, Synapse Link can streamline data movement from AWS storage to Fabric DW.

Another option is to use Microsoft Fabric’s REST APIs to push data into your DW without relying on Fabric notebooks.

If you prefer a script-based approach directly from AWS, using ODBC/JDBC or APIs would be the best choice. Have you considered trying a direct connection from Glue or Lambda via ODBC?

[eduardo_schmitz]

yes, a script-based approach would be ideal. My data requires little transformation, so I would much rather push it directly into the DW Fabric using Lambda or Glue.

However i'm not familiar with using ODBC/JDBC or fabric API just yet. Thank you very much for the suggestion. I will study this approach.
if you have an article about this, i'd appreciate, otherwise i'm alreayding accepting this as solution.

[R1k91]

you can consider also the brand new spark connector for warehouse.
Spark connector for Microsoft Fabric Data Warehouse - Microsoft Fabric | Microsoft Learn

[alia12]

Sure! Since your data doesn’t need heavy transformation, a script-based insert makes total sense. You can definitely do this using Python or Node.js from AWS Lambda or Glue, you'd just need to first exchange credentials for an access token (using Microsoft Entra ID), and then call the Fabric REST API to insert the data. It’s a bit of a setup, but once you’ve done the OAuth flow and understand the structure of the Fabric API, it becomes straightforward. I haven’t seen a full walkthrough yet, but the official docs - https://learn.microsoft.com/en-us/fabric/ cover the endpoints and auth setup well.

[v-veshwara-msft]

Hi @eduardo_schmitz ,
Thanks for the update,
To help you get started with ODBC/JDBC connections to Fabric Data Warehouse, here are some useful resources:

1. Connectivity to data warehousing - Microsoft Fabric | Microsoft Learn

2. Set up your Odbc connection - Microsoft Fabric | Microsoft Learn

3. Microsoft ODBC Driver for SQL Server - ODBC Driver for SQL Server | Microsoft Learn

4. Microsoft JDBC Driver for SQL Server - JDBC Driver for SQL Server | Microsoft Learn

If you're considering Fabric's REST API as an alternative, you can explore:
Microsoft Fabric REST API references - Microsoft Fabric REST APIs | Microsoft Learn

Hope these help. Please reach out for further assistance.

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and a kudos would be appreciated.

Thank you.

[v-veshwara-msft]

Hi @eduardo_schmitz ,

We haven’t heard from you since our last response and wanted to check if the provided solutions met your needs. If they did, please consider accepting the solution to help others benefit. Otherwise, feel free to reach out for further assistance.

Thank you.

[v-veshwara-msft]

Hi @eduardo_schmitz ,

Just checking in again as we haven’t heard back from you. If the provided responses addressed your issue, kindly mark the helpful reply as the Accepted Solution to assist others with similar queries.

If your issue is still unresolved or you need further help, feel free to reach out.
Thank you.

[v-veshwara-msft]

Hi @eduardo_schmitz ,

Thank you for reaching out in Microsoft Fabric Community.
Based on your scenario, there are multiple ways to ingest data from AWS into Microsoft Fabric Data Warehouse:

1. Direct Data Ingestion from AWS
You can establish a connection from AWS services (such as AWS Lambda, AWS Glue) to Fabric Data Warehouse using ODBC/JDBC. This allows data to be inserted directly into Fabric DW using INSERT INTO or MERGE statements. However, setting up and maintaining the connection falls under AWS configurations.

2. Using Fabric Data Pipelines.
Microsoft Fabric offers Data Pipelines to ingest data from AWS S3 using Fabric's S3 connector.
This method enables a no-code/low-code approach to move data into Fabric Data Warehouse efficiently.
Set up your Amazon S3 connection - Microsoft Fabric | Microsoft Learn

Since setting up a direct connection from AWS to Fabric DW primarily involves AWS-side configurations, I recommend posting this in the AWS Community Forum for more insights from AWS experts:
  AWS re:Post Community → https://repost.aws/

Hope this helps. Please reach out for further assistance.

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and a kudos would be appreciated.

Best Regards,
Vinay.