Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Can't get columns from a table into a grid variable

I'd like to be able to load the column names from a table into a grid component to them use them in a python script.

I have created a grid variable ("stg_columns") with one column - "name".

I then used the Table Metadata to grid component and loaded the "name" column with the "Name" value from the table I want. After that component, I then call a python script and do:

print len(context.getGridVariable('stg_columns'))

It prints 0. I have ensured that it's a valid and populated table. If I manually add values to the grid variable rather than using the metadata component, it works fine. It appears that that component is not actually loading the column names.

Ideas? Have others gotten that component to work? Is there some sort of timing issue that could cause this?

Thanks!

5 Community Answers

Matillion Agent  

Arawan Gajajiva —

Hi Dov -

I can’t seem to reproduce the behavior you are describing. I created a job that does what you have described and it seems to work fine for me. If you want, we can help you debug your issue if you send a copy of your job to support@matillion.com.

Best regards,
Arawan


Dov Rosenberg —

I think I've narrowed it down. Put an AND component between the table metadata component and the python script. That seems to break it. If I remove the AND component, it works fine.


Matillion Agent  

Arawan Gajajiva —

Hi Dov -

If you are using the And Component, you will need to ensure that your Grid Variable is set with Behaviour = Shared. Notice in the documentation for the And Component the following verbiage:

NOTE: Any Copied variables that exist in the workflow when going through the And component will be reset to the values they held at the beginning of the job; unless the current job is being called from another job, the variable in question will be reset to its default value. This is to prevent the need to resolve two potentially different values of a Copied variable when two parallel workflows meet the And component. If this is not the case for your workflow, consider using Shared variables instead.

Hope this helps!

Arawan


Dov Rosenberg —

That does help, thanks. But a shared variable is shared across all instances of the workflow, correct? So if I have a workflow A that calls multiple instances of workflow B (concurrently) and inside workflow B I need to pass a variable through an AND that needs to be isolated from the other instances of B that are running, there's no way to do that?

Perhaps as a future enhancement, the AND component could specify how to deal with copied variables...

In any case, thanks for the help - this will definitely solve my current issue.


Matillion Agent  

Kalyan Arangam —

Hi Dov,

The scenario you describe is valid for an Environment Variable (Project→Environment Variable). Grid variables are by default local to a job. From what I understand, the value passed to a child-job or shared-job is always copied. (Unlike an Environment variable) Any change made in the child/shared job is not reflected back to the original job-variable.

You should be careful when using shared-job-variables with iterators in “Concurrent” mode or in parallel flows in a job where the variable value could be modified.
For example, if you have a Iterator with a python-script that could alter the state of your shared-job-variable (scalar or Grid). If iterations are run in Concurrent mode, then any change made in a python script may reflect across iterations. Besides that, you should be safe in most cases.

Hope that makes sense.
I will close this ticket for now.

Best
Kalyan

Post Your Community Answer

To add an answer please login