Hi. I have been experimenting with the Python component to get data from an API. Sometimes I get to much data and get an out of memory message.
What is the memory limit of the Python component and is this something i can change manually?
1 Community Answers
Ian Funnell —
There are limits to the amount of memory that the Python component can use at runtime. The limits can’t be changed programmatically, because they are a function of the EC2 instance size that you have chosen.
In Matillion version 1.7 you can choose between Jython, Python2 and Python3. With Jython the memory limit is imposed by the Java JVM, and for the others it’s the underlying operating system.
If you’re calling an API, it might be worth looking at Matillion’s API Query component, which among other things offers support for paging. This means you don’t have to store all the data in memory at runtime. Instead it automatically pages on your behalf, storing only a small subset of the data in memory during each iteration.