IAM Roles & Permissions (AWS)

AWS credentials are required for Matillion ETL instance to access various services such as discovering S3 buckets and using KMS. Appropriate permissions must be given via your AWS admin console and details of your GCP account must be entered into the Matillion ETL instance via Project → Manage Credentials where credentials for other platforms may also be entered.

You can grant access to other AWS services using an Access Policy.



Adding an AWS Access Policy

There are two ways you can add give access to Matillion ETL.

1. With instance credentials - This is done by specifying an IAM Role for the EC2 instance at launch time. Once in place, you can add and remove permissions from the chosen IAM role and the changes take effect immediately. This gives you fine grained control over exactly which AWS services Matillion ETL is allowed to use. This method is the easiest to manage if you attach the role to the instance while it is being launched. However it is possible to add an IAM to a running instance, using the AWS command line.

2. With existing user defined credentials - These can be defined via the Matillion ETL user interface at any time. The credentials can be created via an IAM User, and Matillion will be able to access what the IAM user has access to. You can simply enter an Access Key ID and a Secret Access Key.

Depending on your company’s security policy, you may choose to use either:

● Coarse-grained access control - easier to administer, but typically grants many more privileges than are necessary.

 Fine-grained access control - requires you to specify a more complex policy but follows the principle of least privilege. A template for this policy type is attached to this document for the user's convenience.

 

AWS Coarse-grained access control

With this approach, you simply attach the following Managed Policies to the role you are using.
 

Managed Policy Purpose

AmazonRedshiftReadOnlyAccess

This policy allows Matillion to read metadata about the Redshift clusters, such as endpoint and port number. This makes it easier to set up new projects and environments. If you don't allow this role, you will need to type in all the information manually.

It does not allow Matillion to read data from Redshift (that privilege is granted by you providing a username and password, and by you permitting network connectivity between Matillion and Redshift).

AmazonS3FullAccess

This policy is used to read data to and from S3 buckets. It can be used in almost every load component, since they almost all either read directly from S3 or else use an S3 Staging area.

AmazonSNSFullAccess

This policy is used when creating SNS topics and publishing messages to them. This is the purpose of the SNS Message Component.

AmazonSQSFullAccess

This policy enables Matillion to read from an SQS queue, or to write an SQS message using the SQS Message Component.

CloudWatchFullAccess

This policy is required to publish metrics to Amazon CloudWatch from the CloudWatch Publish component.

AmazonRDSReadOnlyAccess

This policy is used by the RDS Query and RDS Bulk Output components to discover information about the available RDS servers in the region.
Without this policy, Matillion may still be able to connect to RDS provided the network path is set up and a username and password have been provided.

AmazonEC2FullAccess

This policy is used when performing automated backups.



Note there is no single Managed Policy which can be used to authorize KMS access in the Manage Passwords dialog. Please use the KMS table in the fine-grained access control section below.



AWS Fine-grained access control

This section lists all the IAM privileges which Matillion ETL can require during normal operation. Privileges are shown using AWS policy action metamodel version "2012-10-17". It does not take into account any privileges required by custom AWS requests made in Bash or Python scripts.

There are no mandatory IAM Actions which Matillion absolutely requires. However the recommended ones listed in the next section are the minimum required for productivity and ease of use.
 

Recommended Actions Description

redshift:DescribeClusters

During first login, allows Matillion to find Redshift clusters. Also used when creating a new environment. Note this privilege is not required in order to use a particular Redshift cluster: it’s just useful to make setup faster and less error prone.

s3:ListAllMyBuckets

Enables every S3 Staging Area drop down list to work. Used in every Orchestration data load component.
Note: Matillion ETL can run without this if you wish to keep your buckets private. However, exploration and discovery of buckets from within the client will no longer work.

s3:ListBucket

Enables operator to view folders inside buckets. Can be used in every Orchestration data load component.

s3:GetObject

Enables Matillion to read S3 files. Used in every Orchestration data load component.

s3:PutObject

Used by every Orchestration data load component to create temporary files in a nominated staging bucket.

The files are used temporarily in order to take advantage of Redshift’s bulk loader and transfer the data as quickly as possible into Redshift. This Action could be restricted to specific staging buckets.

 

s3:DeleteObject

Used for the same purpose as s3:PutObject, to remove temporary S3 files created during data loads.

s3:GetBucketLocation

Uses the subresource location to return a bucket's AWS region.


Automated backups can be enabled from the Project → Manage Backups menu. Backups require the EC2 actions detailed below.
 

EC2 Actions Description

ec2:CreateSnapshot

Used during creation of the EC2 snapshot.

ec2:CreateTags

Used to tag the generated snapshot with a name.

ec2:DescribeInstances

Required to retrieve instance metadata

ecDescribeVolumes

Required to retrieve instance metadata



This section lists the actions required in order to use the KMS option in the Manage Passwords dialog.
Note that in addition to the below, the chosen KMS key must be:
  • In the same Region as Matillion
  • Enabled
 
KMS Action Description
kms:ListAliases Enables Matillion to populate the "Master Key" dropdown by listing all the KMS aliases which are associated with a Key.
kms:Encrypt Enables Matillion to store an encrypted password.
kms:Decrypt Enables Matillion to retrieve and use an encrypted password.
 

In the following tables, we outline actions for RDS, SQS, SNS and Cloudwatch required by Matillion to perform certain functions.
 

RDS Actions Description

rds:DescribeDBInstances

Enables Matillion to list the endpoints of the chosen RDS database type. Note this privilege is not required in order to query data from any RDS instance.


 

SQS Actions Description.

sqs:ListQueues

Enables the operator to select an SQS queue in the main SQS Configuration window.

sqs:GetQueueUrl

Required in order to monitor queues.

sqs:ReceiveMessage

Enables Matillion to receive messages from SQS (e.g. using the SQS Message orchestration component).

sqs:DeleteMessage

Enables Matillion to receive messages from SQS and remove them from the queue.

sqs:SendMessage

Enables Matillion to send messages to SQS.


 

SNS Actions Description

sns:Publish

Required in order to publish messages with SNS (e.g. Using the SNS Message component).

sns:ListTopics

Enables the operator to select from a list of existing SNS topics (such as when using the SNS Message component.

sns:CreateTopic

If the user manually types in a topic name which doesn’t already exist in that region, Matillion tries to create the new topic, which requires this privilege.


 

CloudWatch Actions Description

cloudwatch:ListMetrics

Required to enable the CloudWatch Publish component to list existing metrics.

cloudwatch:PutMetricData

Required in order to publish metrics to Cloudwatch using Matillion’s CloudWatch Publish component.This Action also includes the authorization to create new Custom Namespaces and new Metrics.



 

Credentials for CDC

Below is a full list of the permissions a Matillion ETL instance requires to use CDC as described in the above sections. We provide a brief outline of what each permission allows Matillion ETL to do and which section of the CDC setup it pertains to.
 
Lambda Action Description Client Section
lambda:CreateFunction Create Lambda Functions Manage
lambda:DeleteFunction Delete Lambda Functions Manage
lambda:GetFunction Get information about the Lambda Function Manage
lambda:UpdateFunctionCode Update a Lambda Function's code Manage
lambda:UpdateFunctionConfiguration Modify the version-specifc settings of a Lambda function Manage
lambda:AddPermission Get information about the Lambda Function Manage
lambda:RemovePermission Revokes Lambda Function permission from an AWS Service Manage
lambda:GetPolicy Get the resource-based IAM policy for a Lambda Function CDC Monitor
SQS Action Description  
sqs:DeleteMessage Delete a message from a specified queue Manage
sqs:ReceiveMessage Retrieve up to 10 messages from a specified queue Manage
sqs:ListQueues (Optional) Allows discovery of available queues Manage
sqs:ChangeMessageVisibility Change the visibility timeout of a message in a queue Manage
sqs:GetQueueUrl Get the URL of an Amazon SQS queue Manage
DMS Action Description  
dms:CreateEndpoint Create a DMS endpoint CDC Task Config
dms:DeleteEndpoint Delete a DMS endpoint CDC Task Config
dms:StartReplicationTask Start a replication task CDC Monitor
dms:CreateReplicationTask Create a replication task CDC Task Config
dms:ModifyEndpoint Modify a DMS endpoint CDC Monitor
dms:DescribeConnections Get the status of the connections between a replication instance and an endpoint CDC Task Config
dms:DescribeReplicationInstances Get information on replication instances on your account (region-specific) CDC Task Config
dms:DescribeReplicationTasks Get information on replication tasks on your account (region-specific) CDC Task Config
dms:StopReplicationTask Stop a replication task CDC Monitor
dms:DeleteReplicationTask Delete a replication task CDC Monitor
dms:DescribeEndpoints Get information on DMS endpoints on your account (region specific) CDC Task Config
dms:TestConnection Tests the connection between a replication instance and a DMS endpoint CDC Task Config
S3 Permissions Description  
s3:GetBucketNotification Get the notification configuration of a bucket CDC Task Config
s3:ListBucket View folders inside S3 buckets  
s3:PutBucketNotification Set notification configuration of an S3 bucket CDC Task Config
s3:ListAllMyBuckets (Optional) Get a list of all owned S3 buckets CDC Task Config
IAM Permissions Description  
iam:ListRoles (Optional) Get a list of IAM Roles Manage
iam:PassRole Allows passing a role to an AWS service Manage
EC2 Permissions Description  
ec2:DescribeRegions (Optional) Get the regions currently available to the instance Manage