Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Central variable repository

We are using matillion in three different instances (prod, qa, dev). We are exploring the possiblity to to store variables in a centraly managed database to be used by all environments. Do you have any suggestions in how this could be achieved. We have been thinking about using an rds instance but i cant figure out a way to assigning the varibles in the different instances in a smooth way.

Johan Forssell

1 Community Answers

Matillion Agent  

Ian Funnell —

Hi Johan,

Interesting idea there! Matillion is designed so the master default values of the Environment Variables are held inside its own metadata. You can set defaults per environment, but in your case it’s not going to help because you have separate instances.

Your idea of using RDS would work. You’d need a table with 3 columns: environment, variable and value. You can access the Environment name at runtime via the builtin “environment_name” variable, which would enable you to filter to the correct set. You could loop through the records inside a Python script, and set the values of all the Matillion variables at the start of every Orchestration job run.
You’d need to:

  • Make sure the actual variables are present in every instance
  • Remember to run the Python script at the start of every job

We do have some plans for extending Matillion’s API, and perhaps including some functionality to manipulate environment variable defaults, although there are no definite designs in place yet.

Best regards,

Post Your Community Answer

To add an answer please login