What's the best, or at least a good, practice for separating development/test setups from production? We're using one Matillion instance.
Some ideas 1) make a locked version 'Production' of the project, and do further development and testing in the default version
2) separate environments 'production' and 'test' that use different default schemas. All development, test and temporary views go to a temp schema, and only productive target tables to the production schema that will be backed up
3) create separate projects 'test' and 'production'
We've done 1) and 2). To do 3), we'd like to copy the tested version of processes from test to production. However, when we do Export & Import, we'll just get duplicate processes into the same, existing project. Is there a way to import as a new project?
2 Community Answers
Ian Funnell —
All of the approaches you’ve outlined are perfectly valid, and have been successfully used by various customers.
It’s best for you to choose whichever processes matches most closely onto your organisation’s current procedures concerning development, testing and software release.
I am also doing 1) and 2), and it's working fine. I'm the only Matillion developer, so far, at our organization. So, a "looser" dev/test/prod/promotion system is working for me. You may have reasons to have a better/more structured system.
Regarding 2), we currently are working with just one Redshift cluster; so, I set up one "dev" DB and one "prod" DB, on the same Redshift cluster. As such, my two Matillion "environments" each point to different DBs, but on the same Redshift cluster. Probably, it would be a better practice to have different Redshift clusters, and we might consider this at some point.
We might also consider 3) at some point; but for now, I'm happy with just a single project that has multiple versions.