Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Increasing S3 Load Columns Limit

Hi all,

I was using the S3 Load Generator Components on a S3 gzipped CSV when this error occurs:

Exception caught whilst getting sample: java.lang.ArrayIndexOutOfBoundsException - 512
Hint: Number of columns processed may have exceeded limit of 512 columns. Use settings.setMaxColumns(int) to define the maximum number of columns your input can have
Ensure your configuration is correct, with delimiters, quotes and escape sequences that match the input format you are trying to parse

It seems like some setting in Java is limiting the number of array being maximum 512 items. Is there any way to expand it and was it the limit to safe-guard the system from Out of Memory?

Thank you.

4 Community Answers

Matillion Agent  

Kalyan Arangam —

hi Quindici,

Thanks for bringing this to our notice.
How many columns does your file contain?

Are you able to share this file with us?

Thanks
Kalyan


Quindici Chung —

Hi Kalyan,

It's with about 700 columns. It's a file from Segment.

Thank you.

Quin


Matillion Agent  

Kalyan Arangam —

Hi Quindici,

I have raised a ticket (EMD-3983) to fix this issue.

Coming to think of it, there may still be an issue if we increase this limit. The sampling is done in the client browser and rendering the 700 odd columns and some sample data may require some effort from your browser. Also, It may be unwieldy to deal with 700 rows of column metadata in the component interface. It may be okay once we generate the CREATE TABLE and S3 LOAD components.

In the meantime, you may have to create the target table yourself. Will let you know if we manage to fix this issue.

Best
Kalyan


Quindici Chung —

Hey Kalyan,

Cool, thanks!

Post Your Community Answer

To add an answer please login