Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Azure Blob Storage Load Purge File option

I am a beginner to the Matillion components, i was trying to load data from Azure blob storage using Azure Blob Storage Load component to Snowflake table. I was able to achieve it, i would like to delete the source files from Blob storage once the load is completed. I have tried to set the Purge files option value to True, but it didnt work. Can you please help here to achieve this ?

9 Community Answers

Best Answer  >Matillion Agent  

Paul Johnson —

Hi Muhammad,
I think I have managed to reproduce the issue you are having, so will raise this with our engineering team. I have found a workaround you can use though.

  1. Copy the SQL from the Azure Blob Component SQL tab
  2. Paste it into a SQL component in an orchestration job
  3. Generate a SAS with read and delete permissions for that storage account
  4. Add a line in the SQL script after the blob location which reads CREDENTIALS=(AZURE_SAS_TOKEN = ‘token’)

I will update you when i hear more from engineering.

Matillion Agent  

Paul Johnson —

Hi Muhammad,
The purge files property is the correct one. If it is not working I would first make sure have delete permissions on the file. Once you have confirmed this, can you send the server logs from the Admin → Downlaod server logs to support@matillion.com and I can take a closer look.

regards,
Paul


Muhammad Muhasin Muhammadali M P —

Hi Paul

The Matillion VM is added as OWNER to the Storage account which i am using, will that be enough for delete permission of a file in the storage ? If this is true i can send you the logs else let me know if anything has to be enabled from my end

Regards
Muhasin


Muhammad Muhasin Muhammadali M P —

Hi Paul

I have sent an email with the log file attached, can you please have a look and let me know your findings ?

Regards
Muhasin


Matillion Agent  

Paul Johnson —

Hi Muhammad,
What method of authentication from the options in this document do you use in your instance to authentivcate with Azure?

Regards,
Paul


Muhammad Muhasin Muhammadali M P —

HI Paul

I have used the Using Identities (Instance Credentials) with System managed identity option

Regards
Muhasin


Best Answer  Matillion Agent  

Paul Johnson —

Hi Muhammad,
I think I have managed to reproduce the issue you are having, so will raise this with our engineering team. I have found a workaround you can use though.

  1. Copy the SQL from the Azure Blob Component SQL tab
  2. Paste it into a SQL component in an orchestration job
  3. Generate a SAS with read and delete permissions for that storage account
  4. Add a line in the SQL script after the blob location which reads CREDENTIALS=(AZURE_SAS_TOKEN = ‘token’)

I will update you when i hear more from engineering.


Muhammad Muhasin Muhammadali M P —

HI Paul

I will look forward for your update on the fix of the issue

Regards
Muhasin


Muhammad Muhasin Muhammadali M P —

Hi Paul

Is there any update on the issue fix with Azure blob storage load component

Regards
Muhasin


Matillion Agent  

Paul Johnson —

Hi Muhammad,
I cannot provide an estimate but I will update you when i do know more.

Regards,
Paul

Post Your Community Answer

To add an answer please login