I have an S3 bucket with two folders. S3://BKP and S3://DIMS.
I have to rename files in a way as shown below
DIM1_TOP*.TXT to DIM1_TOP.TXT DIM1_XOT*.TXT to DIM1_XOT.TXT DIM2_TOP*.TXT to DIM2_TOP.TXT DIM2_XOT*.TXT to DIM2_XOT.TXT
Basically I want to remove anything after TOP or XOT. One condition is I may not always receive all the 4 files. for example I do not have DIM2_XOT*.TXT. But still my code should not fail or it simply ignore the error and rename those files whichever are available.
3 Community Answers
Kalyan Arangam —
One option is to use a file-iterator to iterate over each file and take necessary action using the AWS CLI via the Bash or Python component. This way, you are dealing with one file at a time and you can implement your logic in a Bash/Python/child-job which may receive the single file to act on.
You may also attach a S3 Put Component to the file-iterator. It can copy files between S3 locations.