When Matillion runs any jobs it always validates the component and then executes it. I understand, it needs to validate when developing and running for the first time but not sure whey it needs to validate every time it runs.
Even when a version is locked (no changes allowed without unlocking), it seems to validate the component before running it. Can we override this behavior? It seems like an overhead for a job with 100 different components, when it has to validate all 100 components every time (considerable overhead for NRT extracts) before running them.
2 Community Answers
Ian Funnell —
Many thanks for raising this question.
At the moment, Matillion always re-validates components before running them, in order to regenerate accurate SQL in case of changes to variables and/or database structures. There is no option to override this, even if you are confident that the database objects have not changed, and you’re not using variables.
We are, however, considering how best to possibly optimize this approach in the future, without losing the integrity of the generated code.