Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Access the Matillion "Task History" data, directly?

Hi,

In a webinar today, I think I heard that it is possible to access my Matillion "Task History" data, in a database. I might like to use it as a data source in a Matillion ETL job, and "ELT" that data into my Redshift data warehouse. Then, report on it with our own BI tool, in addition to looking at the information within the Matillion UI.

(Looking at the "Task History" in the Matillion UI is a good experience, for myself...but if I can get the same data into my data warehouse, then I could share it with all of our end users, too.)

Is that right? It sounded like it is in a MongoDB and/or RDS database, somewhere? Could you please explain how I can access it, if indeed it is possible?

4 Community Answers

Matillion Agent  

Craig Rouse —

Hi Kevin,
The task history can be accessed via the “Matillion API” API Profile.
Use a Custom API component to pull the data out.
See this document
Thanks,
Craig


Kevin Havice —

Got it.

I think a key piece is missing, though. In the "Run History Details" data, for each record, I need the "parent" job ID -- that is, the job/component that "called" this job/component.

That must be a part of the data model, right? It's just not surfaced by the API? If that's the case, could I please request it be added?

Thanks!


Matillion Agent  

Paul Johnson —

Hi Kevin,

I wrote a blog on how to do exactly this here
You can join the detail and summary data on the ID its a one to many relationship.

Regards,
Paul


Kevin Havice —

I see. I had assumed the "id" in the "Run History Details" data was a unique ID within that dataset. But now I get it.

Still, this doesn't let me "traverse" the entire tree, like when my top-level orchestration job (which is called by the scheduler) calls another orchestration job, which calls an iterator component, which iterates X times on a transformation job that has N transformation components. In the Matillion UI, I can expand/collapse the entire tree...so I feel like there must be a data point that says, for each component, what "parent component" called it.

On the other hand, I don't yet have specific requirements around how I want to analyze and report on our Matillion task history. I'm just looking at what's possible. So, I don't know yet how useful it would actually be, to be able to traverse the entire tree, like that.
: )

I will just post again, if in the future I cannot do what I want to do.

Many thanks!

Post Your Community Answer

To add an answer please login