Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

File Iterator


I am using a file iterator with the put component. I see there is a recursive property that I can use to control whether or not I want to draw files from subdirectories. However, I don't see where I can set this property. Only 8 properties so up for the File Iterator when selected yet the documentation shows many more.

What am I missing?



3 Community Answers

Matillion Agent  

Ian Funnell —

Hi Tom,

The fields which appear when you are configuring the File Iterator are highly dependent on which Input Data Type you choose. For example, Recursive is only relevant to SFTP, FTP, HDFS and Windows Fileshare, and won’t appear if you choose S3.

For S3, you would instead need to use the Filter Regex property to restrict the search space to the prefixes you specify.

Best regards,

Tom Flynn —

So you can use RegEx to filter out subdirectories?

Also, if I'm accessing files on S3, I assume I pretty much have to use S3 as the input data type. RIght?

Matillion Agent  

Harpreet Singh —

Hi Tom

Yes you can use Regex to filter the sub directories. Once the files are copied to S3 bucket you can use the S3 Load component to load the files to your datawarehouse.


Post Your Community Answer

To add an answer please login