I am a beginner to the Matillion components, i was trying to load data from Azure blob storage using Azure Blob Storage Load component to Snowflake table. I was able to achieve it, i would like to delete the source files from Blob storage once the load is completed. I have tried to set the Purge files option value to True, but it didnt work. Can you please help here to achieve this ?
9 Community Answers
Paul Johnson —
I think I have managed to reproduce the issue you are having, so will raise this with our engineering team. I have found a workaround you can use though.
Copy the SQL from the Azure Blob Component SQL tab
Paste it into a SQL component in an orchestration job
Generate a SAS with read and delete permissions for that storage account
Add a line in the SQL script after the blob location which reads CREDENTIALS=(AZURE_SAS_TOKEN = ‘token’)
I will update you when i hear more from engineering.
Paul Johnson —
The purge files property is the correct one. If it is not working I would first make sure have delete permissions on the file. Once you have confirmed this, can you send the server logs from the Admin → Downlaod server logs to firstname.lastname@example.org and I can take a closer look.
The Matillion VM is added as OWNER to the Storage account which i am using, will that be enough for delete permission of a file in the storage ? If this is true i can send you the logs else let me know if anything has to be enabled from my end