Out of Memory
My Matillion has restarted several times due to Out Of Memory errors.
I am doing initial load of AdWords data for about 200 accounts for the past 2 years. So I have a loop over the accounts and then another one over monthly intervals. This results in about 5000 iterations in total. The job usually makes about 1000 iterations before failing. Data volume is not the issue, because when I start from where the job failed, it runs fine for another 50 accounts or so.
I have tried to monitor the memory usage - it usually starts at 37-38% and slowly crawls up to 42-43% at which point it fails, regardless of how much time it took to get there. Also the log doesn't have any indication - the only relevant message is "# java.lang.OutOfMemoryError: Java heap space".
Are there any limits, timeouts or other considerations I should be aware of when I design the jobs to avoid this? Ideally, I would like to let the job run for a day and catch up with the data.