Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Passing variables into Bash script

Hi,

I have Loop Iteration followed with Bash script, based on iterations i have to pass the value into bash script.

Exp: For First iteration the value 1 should be passed to one of the variable into Bash script.

Script: echo $Iteration

So my output should be 1 for first Iteration.

15 Community Answers

Matillion Agent  

Laura Malins —

Hi Mallikarjuna

This will depend how you’ve set up the loop iterator. If your variable is called iteration and you start with value 1. Then echo $iteration will do what you want. Are you able to provide any further details on your iterator? Perhaps you could send us an example of your job?

Thanks
Laura


Mallikarjuna P —

Hi Laura,

Thank you for your valuable time, it is working fine.

Thanks & Regards,
Mallikarjuna P


Mallikarjuna P —

Hi Laura,

I want to pass the value which is coming from BASH Script output to next compoent each time (based on no of Iterations), Please help me on this.

Exp: I have multiple files like below and i have to load it into same table.

1)File_Name_${YYYYMM}.csv
2)File_Name_${YYYYMM}.csv
3)File_Name_${YYYYMM}.csv
.
.
.
Value for ${YYYYMM} will be coming from Bash script output, this value have to pass it to next component.

Thanks & Regadrs,
Mallikarjuna P


Matillion Agent  

Kalyan Arangam —

hi Mallikarjuna,

You cannot pass values out of a BASH script component. Besides iterators, Python script component is the only component that allows altering values of variables at run time.

Assuming your files are in S3, use the S3 Load component with S3 Object Prefix set to File_Name_. This will load all files with similar prefix to thesame target table in one go.

If the above doesnt suit your requirement, please share a bit more about your scenario – a copy of your job would also help.

Best
Kalyan


Mallikarjuna P —

Hi Kalyan,

Thank you for yuor quick response, really it helped us.

Thanks & Regards,
Mallikarjuna P


Mallikarjuna P —

Hi,

Can't we load the multiple .xlsx files in to redshift in one go to same table.?

Regards,
Mallikarjuna P


Matillion Agent  

Laura Malins —

Hi Mallikarjuna

Yes you can, providing those .xlsx files are all in the same format. To do this you will need to parameterise the name of the file to load, as discussed above. You can then load them to the same table by setting the Load Option “Recreate Target Table” to be “Off”. A better option would be for you to load them all to different tables and then combine them together in a transformation job.

Thanks
Laura


Mallikarjuna P —

Hi Laura,

But my file names will look like below.

1) File_Name_201801.xlsx
2) File_Name_201802.xlsx
3) File_Name_201803.xlsx
4) File_Name_201804.xlsx
.
.
.
.

Regards,
Mallikarjuna P


Matillion Agent  

Laura Malins —

Hi Mallikarjuna,

You can pass those into names into a variable and then use the variable as the filename in the Excel Query component. You may also need to parameterise the sheet name if these are different.

thanks
Laura


Mallikarjuna P —

Hi Laura,

If i parameterise the sheet name, each time i have to pass the value accordinly and if have 10 files with YYYYMM.xlsx then i ahve to run the job 10 times with 10 different values.

Thanks,
Mallikarjuna P


Matillion Agent  

Laura Malins —

Hi Mallikarjuna

You don’t need to run an iterator inside another iterator. You can just set up your fixed iterator to have 2 variables each with different values.

So 10 rows in the fixed iterator with 2 variables will only create 10 iterations not 20, or 100….

Thanks
Laura


Mallikarjuna P —

Hi Laura,

Thank you for your input, but i really did't get what you are saying.
Let me explain the details once again.

I have 5 files like below:

file_201711.xlsx
file_201712.xlsx
file_201801.xlsx
file_201802.xlsx
file_201803.xlsx

Now tell me how to load all these files in to one table at one go.

Thanks & Regards,
Mallikarjuna P


Matillion Agent  

Ian Funnell —

Hi Mallikarjuna,

Is your list of files fixed, or can it vary?

Best regards,
Ian


Mallikarjuna P —

Hi Ian,

List of files will vary.

file_201711.csv
file_201712.csv
file_201801.csv
file_201802.csv
file_201803.csv
.
.
.
.etc

Regards,
Mallikarjuna P


Matillion Agent  

Laura Malins —

Hi Mallikarjuna,

You will need to use either .csv files or .xlsx files, Matillion can’t switch between the two easily. Do you know the list of files and the sheet names in those files?

Assuming your files are on s3 and are .xlsx you can do the following:

1. Use a file iterator to identify all of the .xlsx files in your s3 bucket. Using the iterator you can pass the filename into a variable, say ${filename}
2. Connect this iterator to the Excel Query component. Your Excel File path should contain a reference to that variable, for example s3://my-bucket/${filename}
3. Set the Load option “Recreate Target table” to be off

The above is reliant on the sheet name being the same across all files.

If the sheet name is different, you need a way to determine what the sheetname is based on the file name.

If that’s not your use case please give us detail on what exactly it is you’re trying to do, including example file names and sheet names if possible and we will try to help

Thanks
Laura

Post Your Community Answer

To add an answer please login