Unfortunately the limit of 65535 bytes is a Redshift limit and there’s nothing that can be done to change this, from Matillion or otherwise.
I assume when you say the length isn’t static that you mean the elements in the arrays vary so Flatten Arrays won’t work? Flatten Arrays should cope with varying lengths.
You need to think about how you want this mongo data to be stored in Redshift. It’s clear that your array won’t fit into single column in a Redshift table, so can it be split up. There may be a possibility to do this from the Matillion MongoDB component.
You can use python to extract the data from MongoDB as well, but again that would require you to understand how it would be loaded into Redshift because you will still hit the issue with varchar(max) being too small.
If you’re willing to share some sample data we could take a look and possibly suggest some options?