I was using the S3 Load Generator Components on a S3 gzipped CSV when this error occurs:
Exception caught whilst getting sample: java.lang.ArrayIndexOutOfBoundsException - 512 Hint: Number of columns processed may have exceeded limit of 512 columns. Use settings.setMaxColumns(int) to define the maximum number of columns your input can have Ensure your configuration is correct, with delimiters, quotes and escape sequences that match the input format you are trying to parse
It seems like some setting in Java is limiting the number of array being maximum 512 items. Is there any way to expand it and was it the limit to safe-guard the system from Out of Memory?
4 Community Answers
Kalyan Arangam —
Thanks for bringing this to our notice.
How many columns does your file contain?
I have raised a ticket (EMD-3983) to fix this issue.
Coming to think of it, there may still be an issue if we increase this limit. The sampling is done in the client browser and rendering the 700 odd columns and some sample data may require some effort from your browser. Also, It may be unwieldy to deal with 700 rows of column metadata in the component interface. It may be okay once we generate the CREATE TABLE and S3 LOAD components.
In the meantime, you may have to create the target table yourself. Will let you know if we manage to fix this issue.