Getting Started

Jython boto3 delete all files in S3 sub-folders

Hello I have bucket with several folders. One of this folders is ETLWork folders. ETLWork folder has several sub-folders like external_table1, external_table2 ... How can I delete all files in...

Jython Invalid bucket name s3://mocsdw01/FromFTP/

import boto3 BUCKET = FromFTP # name of the s3 bucket file_key = 'AIP5_DLY_PCO_PO_20180714202419.asc.txt' s3 = boto3.resource('s3') obj = s3.Object(BUCKET, file_key) for line in obj.get()['Body']._raw_stream: print(line) botocore.exceptions.ParamValidationError: Parameter validation failed:...

GCP Cloud SQL or just BigQuery?

We are users of the GCP platform and currently trialling Matillion. In our previous work (before Matillion) we already have invested in the use of CLoud SQL as a replication...

Migration

Hello Support, Are there instructions available if we want to migrate from mongoDB to already existing "Postgres RDS Instance with Multi-AZ Failover"? We have Matillion m4.large instance running with MongoDB,...

Mongo DB Json Extract

Hi, I have field in mongodb as greater than 65535 array of JSON data.. I'm used varchar(max). It only get 65535 character count from source to redshift. Any option to...

streaming ETL

Hi....is anyone using Matillion in a streaming fashion where it is reading directly from Kinesis or Kafka?

How to get identity from one row insert?

We would like to generate id for each job execution. CREATE TABLE public.etlbatch ( _etlbatchid INTEGER NOT NULL IDENTITY(1, 1), datetimeid TIMESTAMP DEFAULT ('now'::character varying)::timestamp without time zone NOT NULL,...

Evaluating Matillion

Hi Matillion Community, My team is in the process of evaluating ETL tools to load data from 30+ data source into our data lake. I have two questions: - Does...

MongoDB Component

Hi, I'm updating my mattillion version 1.36.7 and now iam facing some iisue in mongodb components from basic mode. Some fields in mongodb but not display in basic mode. I'm...

Do Matillion have a Split component?

My destinations: CREATE TABLE TBL_1 (DT DATETIME ); CREATE TABLE TBL_2 (INTG INT ); CREATE TABLE TBL_3 (TXT VARCHAR(30)); Source (one file, no header, pipe delimiter): TYPE1| 2018-08-13 00:00:00 TYPE2|1234...