Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Jython StringIO() error

Need create a stream in memory
import boto3
import io
f = io.StringIO()
file_key = 'someDir/b_contracts.txt'
s3 = boto3.resource('s3')
for line in obj.get()['Body']._raw_stream:
f.write(line.encode('utf-8'))

TypeError: can't write str to text stream in <script> at line number xyz

6 Community Answers

Dmitriy (Consultant) Burtsev —

import boto3
import io
BUCKET = 'someBucket'
f = io.StringIO()
file_key = FromFTP + '/b_contracts.txt'
s3 = boto3.resource('s3')
obj = s3.Object(BUCKET, file_key)
for line in obj.get()['Body']._raw_stream:
f.write(line.encode('utf-8'))


Matillion Agent  

Arawan Gajajiva —

Hi Dmitry -

Are you still having issues with this second code snippet that you have provided, or are you sharing what worked for you?

Best regards,
Arawan


Dmitriy (Consultant) Burtsev —

Yes.
TypeError: can't write str to text stream in <script> at line number xyz


Dmitriy (Consultant) Burtsev —

This is the short version. I am running jython2.7.0 in Eclipse
import io

if __name__ == '__main__':
f = io.StringIO()
line = "hgchjvjhvh"
f.write(line.encode('utf-8'))
#f.write(line)
pass

Traceback (most recent call last):
File "C:\Users\Dmitriy.Burtsev\eclipse-workspace\StreamTest\StreamTest\main.py", line 12, in <module>
f.write(line)
File "C:\jython2.7.0\Lib\_jyio.py", line 1155, in write
raise TypeError("can't write %s to text stream" %
TypeError: can't write str to text stream


Matillion Agent  

Ian Funnell —

Hi Dmitriy,

It’s difficult to debug without access to your system, but I was looking back at the context of what you’re attempting here, and just wanted to offer some general advice in a Matillion context:

  1. I’d strongly recommend you don’t use Python Scripts for code which can potentially require large amounts of memory at runtime. Being an ELT tool, Matillion instances are deliberately small, and do not have access to large amounts of memory.
  2. It looks like you’re copying data from FTP to S3, in which case I’d recommend you take a look at the Data Transfer component, which is designed for that purpose (and more)

Best regards,
Ian


Dmitriy (Consultant) Burtsev —

This one works: f.write(unicode(line))

I have to parse one big file into several small files for external tables.

Post Your Community Answer

To add an answer please login