Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Proceed to next component, after success OR failure

Say I have an orchestration job with a series of Database Query components -- say. All of the Database Query components query the same source, and I want them to run consecutively, not in parallel.

But I also want each Database Query component in the series to be executed regardless of whether the previous Database Query component was successful or failed. This must be a common thing, right? But how? What is the best way to do this?

My only idea so far is:
1. Connect the "success" output of Database Query #1 to an "Or" component
2. Connect the "failure" output of Database Query #1 to a DIFFERENT "Or" component
3. Connect the outputs of each of the above "Or" components to a THIRD "Or" component
4. Connect the output of the third "Or" component to Database Query #2

I think this would work, but it seems silly/tedious! I have a feeling that there must be a simpler/more efficient way. Is there?

5 Community Answers

Matillion Agent  

Ian Funnell —

Hi Kevin,

You’ve correctly described the only way to achieve this. There is no option to just ignore errors, and you also can’t attach multiple connectors to a single component (which would be an “Or” in this case).

If you are able to sufficiently parameterize the Database Query, and call it inside a sequential loop, then at least you’ll only need this construct in one place.

Best regards,
Ian


Kevin Havice —

Okay, thanks Ian! I can work with that, then.


Matillion Agent  

Ian Funnell —

Hi Kevin,

Another


Malachi Witt —

I have a similar scenario but instead need to run API queries pulling the same structured data from multiple sources. If one of the API jobs fail the remaining jobs are never executed. The problem is that these jobs are pulling real time data so using a retry loop wouldn't work.

Are there any plans to develop better support for running independent jobs consecutively? The problem is that the current design is focused on dependent simultaneous jobs but there are scenarios where independent consecutive jobs are required.


Matillion Agent  

Kalyan Arangam —

Hi Malachi,

Please see an attached image of one workaround.

I have created a child job with an API component thats been parameterised and then ignores any errors it will encounter. We then have a parent job that may invoke this job multiple times passing appropriate parameters. This way, any errors in the child-job will be ignored and the parent job will continue.

Parameterised child job - ignoring errors

Another approach – also using parameterised API Query component – is to use an iterator to iterate over a set of values in a table to then call the attached API Query component. An Iterator can be configured to ignore errors so any errors in a child-job ot an attached component.

Hope you are able to use one of the above suggestions.

Best
Kalyan

Post Your Community Answer

To add an answer please login