Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Table Iterator with more than 5,000 rows?

Hi,

I am testing a new job that uses a table iterator, iterating over a large table (> 5,000 rows).

When I started testing, I noticed in the Task History that it said "5000 iterations generated" ... when I was expecting more iterations. Because there are more than 5,000 rows in the table.

My iterator turned out to be quite slow, and I canceled it before it finished, so I haven't seen what it does after those 5,000 iterations.

But what behavior should I expect?

(I don't see mention of this "5,000 rows" thing in the documentation of the Table Iterator component. Does the Table Iterator component have a strict limit of 5,000 rows...then it just stops? Or, is it just breaking things up into chunks of 5,000 iterations...such that after finishing the first 5,000, it would start the next 5,000?)

3 Community Answers

Matillion Agent  

Paul Johnson —

Hi Kevin,
All Iterator components support a maximum of 5000 iterations.

The design choice behind this is because all “bulk” operations should be pushed down to the database rather than be processed on the instance. Can you provide some detail around the use case? I might be able to offer an alternative solution.

Regards,
Paul


Kevin Havice —

Got it.

I have to call an API for each row in a table, using a value of that table as a parameter in the API call.

But I can think of various ways split it up, to do < 5,000 iterations per iterator.


Matillion Agent  

David Lipowitz —

Hi Kevin,

Glad you have a path forward here. Please let us know if you need anything else on this.

Best Regards,
Dave

Post Your Community Answer

To add an answer please login