File iterator duplicating data
I had to generate a filename from the files uploaded to S3, hence I found a very good article here by adding a filename to table which helped me immensely achieve my purpose. However, my data is duplicating in the target table.
I have 2 files uploaded to S3 and I need to load the data from the files into a target table with the filename. i used File iterator on my orchestration job. Now it loads the data from 2 files with filename, but also in the last iteration its duplicating the last loaded file data. I am not able to figure out the exact problem here. I am using table output with append. I even tried using table update/insert, but its the same. I also added a filter condition in my sql script to not pass the file if it already exists in the target table. but still its loading again. i am storing the file in variable and passing it in my job.
Can somebody help me debug this problem? I am new to Matillion and I don't find much of resources online too.