Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Available memory for Python component

Hi. I have been experimenting with the Python component to get data from an API. Sometimes I get to much data and get an out of memory message.

What is the memory limit of the Python component and is this something i can change manually?

1 Community Answers

Matillion Agent  

Ian Funnell —

Hi Johan,

There are limits to the amount of memory that the Python component can use at runtime. The limits can’t be changed programmatically, because they are a function of the EC2 instance size that you have chosen.

In Matillion version 1.7 you can choose between Jython, Python2 and Python3. With Jython the memory limit is imposed by the Java JVM, and for the others it’s the underlying operating system.

If you’re calling an API, it might be worth looking at Matillion’s API Query component, which among other things offers support for paging. This means you don’t have to store all the data in memory at runtime. Instead it automatically pages on your behalf, storing only a small subset of the data in memory during each iteration.

Best regards,

Post Your Community Answer

To add an answer please login