Iterate through file in S3 Load component using specific pattern
I want to load all files with following naming sitting in a root S3 bucket. There are other files there too but I need to selecively only want the one that I defined in my pattern properties under S3 load component. I tried passing a partial name but it didn't pick up.
For example, the files I want to load have following pattern ABC_123.gzip ABC_234.gzip ABC_456.gzip
When I specify ABC as the pattern, it does not pick up the files.
Any thoughts if I should be specifying the pattern some other way?
3 Community Answers
Ian Funnell —
In Matillion ETL for Snowflake, you should do as follows:
set the S3 Object Prefix to the root bucket name – either with or without the trailing /
set the Pattern property to a regex which will match whole filenames
ABC on its own contains no wildcards, and so will only match a file exactly named ABC.
Thanks Ian for the quick response. I am trying ABC.* and let you know if that will pick up the file with any suffix after the string ABC_. Will get back if any issues. I had tried *ABC* before I posted this questions and it gave error stating invalid reg-ex,