Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Triggering ETL from an S3 Event via AWS Lambda for multiple files.

Can I load multiple files to snowflake using "Triggering ETL from an S3 Event via AWS Lambda".
I want to use single lamda function, single SQS and single Matillion job to load multiple files.

6 Community Answers

Matillion Agent  

Jason Kane —

Hi Deb,
Thank you for contacting support. Yes you can configure a single Matillion job to load multiple files from S3 into Snowflake.

Thanks,
Jason


Rumi Deb —

Hi Jason,

Here the problem I am facing is, the files will arrive in a single S3 bucket but the folder paths are different. How to address this with a single Lamda function.

As per my plan, the job flow is like:

S3 -> AWS Lamda -> SQS -> Matillion job

Let me know if this is possible.


Matillion Agent  

Jason Kane —

This article walks you through the steps of setting this up in AWS. The path can just be passed as another variable to the Matillion job.

Thanks,
Jason


Rumi Deb —

Hi Jason,

I have already went through this article and understood that based on SQS message Matillion job can pick up the file.

Here i want to know that to send messages to SQS for file arrival, is one Lamda function can do this as my input file folder structure is different (In same S3 bucket but folders are not same).


Matillion Agent  

Jason Kane —

Hi Rumi,

Yes you can use just one lambda function for many folders in a single bucket. With the example code, the file variable section of the SQS message will look something like this: {"file_to_load": “test-files/tested-nested/catalina.out”}. This allows you to set your “S3 URL location” property to the bucket and the “S3 Object prefix” property to the ${file_to_load} variable. Using the sample, any file created in that bucket, regardless of the folder structure beneath, will trigger the same lambda.

Let me know if this answers your question.

Thanks,
Jason


Rumi Deb —

Thanks Jason.
Will work on this and will raise new question if faced any issue.

Post Your Community Answer

To add an answer please login