Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Getting Filename from a S3 bucket and assigning into variable

I am trying to get the file name that resides in a S3 bucket. The filename varies every day in the S3 bucket. I need this to be assigned to a variable. Is there anyway we could do this using Matillion.

1 Community Answers

Matillion Agent  

Dan D'Orazio —

Hi Lakshmi -

You can do this with the Python Component. The Jython environment exposes the ability to get/set variables in the Job’s context. I’ve included some example Python code that can be used to do this.

# Variables are directly accessible: 
#   print myvar
# Updating a variable:
#   context.updateVariable('myvar', 'new-value')
# Grid Variables are accessible via the context:
#   print context.getGridVariable('mygridvar')
# Updating a grid variable:
#   context.updateGridVariable('mygridvar', [['list','of'],['lists','!']])
# A database cursor can be accessed from the context (Jython only):
#   cursor = context.cursor()
#   cursor.execute('select count(*) from mytable')
#   rowcount = cursor.fetchone()[0]
import boto3
s3 = boto3.resource('s3')

my_bucket = s3.Bucket(‘bucket-name’)

for file in my_bucket.objects.all():
print file.key
if file.key == ‘file name or pattern logic’:
context.updateVariable(‘variable_name’, file.key)

Best -

Post Your Community Answer

To add an answer please login