Access the Matillion "Task History" data, directly?
In a webinar today, I think I heard that it is possible to access my Matillion "Task History" data, in a database. I might like to use it as a data source in a Matillion ETL job, and "ELT" that data into my Redshift data warehouse. Then, report on it with our own BI tool, in addition to looking at the information within the Matillion UI.
(Looking at the "Task History" in the Matillion UI is a good experience, for myself...but if I can get the same data into my data warehouse, then I could share it with all of our end users, too.)
Is that right? It sounded like it is in a MongoDB and/or RDS database, somewhere? Could you please explain how I can access it, if indeed it is possible?
4 Community Answers
Craig Rouse —
The task history can be accessed via the “Matillion API” API Profile.
Use a Custom API component to pull the data out.
See this document
I see. I had assumed the "id" in the "Run History Details" data was a unique ID within that dataset. But now I get it.
Still, this doesn't let me "traverse" the entire tree, like when my top-level orchestration job (which is called by the scheduler) calls another orchestration job, which calls an iterator component, which iterates X times on a transformation job that has N transformation components. In the Matillion UI, I can expand/collapse the entire tree...so I feel like there must be a data point that says, for each component, what "parent component" called it.
On the other hand, I don't yet have specific requirements around how I want to analyze and report on our Matillion task history. I'm just looking at what's possible. So, I don't know yet how useful it would actually be, to be able to traverse the entire tree, like that. : )
I will just post again, if in the future I cannot do what I want to do.