Dumping a file into a Datastore Blob Storage from an AzureML v2 Pipeline

Hello,

We've recently migrated our AzureML Pipelines from using Python SDK v1 to v2.
This went mostly smoothly with the exception of one datastore feature we used to use:
In v1 we were able to get write access to an registered datastore with this simple python script within a command component:

datastore = Datastore.get(workspace,datastore_name) datastore.upload_files(files=[local_path], target_path=folder_path)

With v2 this API was no longer available. Unfortunately also the azureml-fsspec library only allows reads from a datastore so that also couldn't be used. In the documentation the only example given is how to accomplish this via a command job. Our setup is using the azure.ai decorators in python so this wasn't very helpful.

Nevertheless, we tried to incorporate the command job example in our pipeline but it doesn't compile it as part of the flow (the inputs from other components are not used).

Could someone helpout with incorporating a command job that copies a file that is an output from another component Output(type="uri_file") into a Datastore Blob Storage within a @pipeline?

Saideep Anchuri 6,435Reputation pointsMicrosoft External Staff
2025-04-23T21:03:50.91+00:00
Hi Zeid Adabel
To incorporate a command job that copies a file output from another component into a Datastore Blob Storage within an AzureML v2 pipeline,
Here are some steps:
Define the Command Job: Create a command job that takes the output from the previous component as an input. You will need to specify the command to upload the file to the datastore.
Use the OutputFileDatasetConfig: This configuration allows you to specify the destination in the datastore where you want to upload the file.
Integrate the Command Job into the Pipeline: Ensure that the command job is part of the pipeline and that it receives the output from the previous component.
from azure.ai.ml import command, OutputFileDatasetConfig, pipeline @pipeline() def my_pipeline(input_file: Output(type="uri_file")): upload_job = command( name="upload_to_datastore", inputs={"input_file": input_file}, command="azcopy copy ${{inputs.input_file}} 'https://<your_storage_account>.blob.core.windows.net/<your_container>/<your_blob>?<sas_token>'" ) return upload_job
Kindly refer below link: how-to-move-data-in-out-of-pipelines
Thank You.

Zeid Adabel0

Hi, thank you for a quick response @Saideep Anchuri

what I struggle with is moving output from a component into the command in the middle of a pipeline, building on your example together with the example I shared that does the copy via a command job.

@pipeline() def my_pipeline(): generated_data = generate_data_component() data_type = AssetTypes.URI_FILE data_path = f"azureml://datastores/{datastore_name}/paths/{folder_path}/data.csv" inputs = { "data_input": Input(type=data_type, path=generated_data.outputs.data, mode=InputOutputModes.RO_MOUNT), } outputs = { "data_output": Output(type=data_type, path=data_path, mode=InputOutputModes.RW_MOUNT, name = "data", version = "1", ), } upload_data = command( command="cp ${{inputs.data_input}} ${{outputs.data_output}}", inputs=inputs, outputs=outputs, ) ... Other components

This doesn't work because the command is never picked up.

Saideep Anchuri 6,435Reputation pointsMicrosoft External Staff
2025-04-25T18:24:04.6533333+00:00
Hi Zeid Adabel
Since the previous v1 approach isn't available, you might need to explicitly define the pipeline's data flow and ensure the command job correctly references the output from the prior component.
You can check out Azure's documentation on managing inputs and outputs in pipelines for guidance on structuring data movement. Additionally, reviewing the CLI v2 pipeline job YAML schema might help refine your approach. Let me know if you need further clarification!
Thank You.

Share via

Dumping a file into a Datastore Blob Storage from an AzureML v2 Pipeline

Your answer