Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Iterate through file in S3 Load component using specific pattern

Hi,

I want to load all files with following naming sitting in a root S3 bucket. There are other files there too but I need to selecively only want the one that I defined in my pattern properties under S3 load component. I tried passing a partial name but it didn't pick up.

For example, the files I want to load have following pattern
ABC_123.gzip
ABC_234.gzip
ABC_456.gzip

When I specify ABC as the pattern, it does not pick up the files.

Any thoughts if I should be specifying the pattern some other way?

3 Community Answers

Matillion Agent  

Ian Funnell —

Hi Bishal,

In Matillion ETL for Snowflake, you should do as follows:

  • set the S3 Object Prefix to the root bucket name – either with or without the trailing /
  • set the Pattern property to a regex which will match whole filenames

ABC on its own contains no wildcards, and so will only match a file exactly named ABC.

So I think you need something like:

ABC.*.gzip

or perhaps just

ABC.*

Best regards,
Ian


Bishal Gupta —

Thanks Ian for the quick response. I am trying ABC.* and let you know if that will pick up the file with any suffix after the string ABC_. Will get back if any issues. I had tried *ABC* before I posted this questions and it gave error stating invalid reg-ex,


Bishal Gupta —

Hi Ian,

Looks like that worked. Appreciate the quick help.

Bishal

Post Your Community Answer

To add an answer please login