Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Scripting error

Hi,
I used a python script to import data(content of a .sql file) from github. Script -

import base64
import requests
content=''
headers = {'Authorization': 'token <token>'}
url = 'https://api.github.com/repos/<file address>'
req = requests.get(url, headers=headers)
if req.status_code == requests.codes.ok:
req = req.json()
content= base64.decodestring(req['content'])
else:
print('Content was not found.')
cursor = context.cursor()
cursor.execute(content)

It runs properly, but a few times randomly it throws the following error -
"ValueError: Extra data: line 1 column 4788 - line 1 column 4794 (char 4787 - 4793) in <script> at line number 25"

Please help me with the issue.

6 Community Answers

Matillion Agent  

Ian Funnell —

Hi Manav,

Matillion is designed to remove the hand coding part of the ETL process, and consequently has no way to import .sql files.

If you wish to retrofit an existing solution, the best way is to use Matillion’s Create Table and SQL Script components.

Best regards,
Ian


Manav Veer Gulati —

Hi Ian,
I need to extract queries from github and run them in matillion. Is there any other way to do it?


Matillion Agent  

Paul Johnson —

Hi Manav,
Other than what you have already attempted there is no way to do this, which is as Ian mentioned by design.
Have you explored using the SQL components as Ian suggested.
Regards,
Paul


Manav Veer Gulati —

Hi Paul
Yes we have, but there are few jobs for which scripts are changed in some other tool and so we need to run the matillion job with updated queries. So we need the queries directly from github.

Does it make sense? Or is there any other way I can go forward with it?

Thanks in advance


Matillion Agent  

Ian Funnell —

Hi Manav,

Thanks for the details. Matillion’s SQL Script component requires that you include (or copy-and-paste) the SQL to run. There’s no way to have it execute SQL stored externally.

Out of interest, are you trying to migrate existing work from another ETL tool?

Best regards,
Ian


Manav Veer Gulati —

Hi Ian,
No we are just editing some queries locally in the DB runner, so we just want to maintain a version control.

Post Your Community Answer

To add an answer please login