IAM Roles and Permissions (AWS)
    • Dark
      Light

    IAM Roles and Permissions (AWS)

    • Dark
      Light

    Article Summary

    Overview

    You manage access in AWS by creating policies and attaching them to IAM identities (users, groups of users, or roles) or AWS resources. A policy is an object in AWS that, when associated with an identity or resource, defines their permissions. AWS evaluates these policies when an IAM principal (user or role) makes a request. Permissions in the policies determine whether the request is allowed or denied. Most policies are stored in AWS as JSON documents.

    AWS credentials are required for Matillion ETL instance to access various services such as discovering S3 buckets and using KMS. Appropriate permissions must be given via your AWS admin console and details of your GCP account must be entered into the Matillion ETL instance via Project, then Manage Credentials where credentials for other platforms may also be entered.

    Note

    For more information on Manage Credentials, please visit the document here.


    Adding an AWS access policy

    There are two ways you can add a policy, and give access to Matillion ETL:

    1. With instance credentials: This is done by specifying an IAM Role for the EC2 instance at launch time. Once in place, you can add and remove permissions from the chosen IAM role and the changes take effect immediately. This gives you fine grained control over exactly which AWS services Matillion ETL is allowed to use. This method is the easiest to manage if you attach the role to the instance while it is being launched. However it is possible to add an IAM to a running instance, using the AWS command line.

    2. With existing user defined credentials: These can be defined via the Matillion ETL user interface at any time. The credentials can be created via an IAM User, and Matillion will be able to access what the IAM user has access to. You can simply enter an Access Key ID and a Secret Access Key.

    Depending on your company’s security policy, you may choose to use either:

    • Coarse-grained access control: easier to administer, but typically grants many more privileges than are necessary.
    • Fine-grained access control: requires you to specify a more complex policy but follows the principle of least privilege. To see the template for this policy type, refer to AWS Policy.

    AWS coarse-grained access control

    With this approach, you simply attach the following Managed Policies to the roles you are using.

    Managed PolicyManaged Policy
    AmazonRedshiftReadOnlyAccessThis policy allows Matillion to read metadata about the Redshift clusters, such as endpoint and port number. This makes it easier to set up new projects and environments. If you don't allow this role, you will need to type in all the information manually. Note: It does not allow Matillion to read data from Redshift (that privilege is granted by you providing a username and password, and by you permitting network connectivity between Matillion and Redshift).
    AmazonS3FullAccessThis policy is used to read data to and from S3 buckets. It can be used in almost every load component, since they almost all either read directly from S3 or else use an S3 Staging area.
    AmazonSNSFullAccessThis policy is used when creating SNS topics and publishing messages to them. This is the purpose of the SNS Message Component.
    AmazonSQSFullAccessThis policy enables Matillion to read from an SQS queue, or to write an SQS message using the SQS Message Component.
    CloudWatchFullAccessThis policy is required to publish metrics to CloudWatch Publish component.
    AmazonRDSReadOnlyAccessThis policy is used by the RDS Bulk Output components to discover information about the available RDS servers in the region. Without this policy, Matillion may still be able to connect to RDS provided the network path is set up and a username and password have been provided.
    AmazonEC2FullAccessThis policy is used when performing automated backups.
    SecretsManagerReadWriteProvides access to Secrets Manager operations. The policy doesn't allow the identity to configure rotation because rotation requires IAM permissions to create roles. To learn more, read AWS managed policies available for use with AWS Secrets Manager.
    Note

    There are no single Managed Policy which can be used to authorize KMS access in the Manage Passwords dialog through Project, then Manage Password. Please use the KMS table in the fine-grained access control section below.


    AWS fine-grained access control

    This section lists all the IAM privileges which Matillion ETL can require during normal operation. Privileges are shown using the AWS policy action metamodel version "2012-10-17". It doesn't take into account any privileges required by custom AWS requests made in Bash or Python scripts.

    There are no mandatory IAM Actions which Matillion requires. However the recommended ones listed in the next section are the minimum requirements for productivity and ease of use.

    Recommended ActionsDescription
    redshift:DescribeClustersDuring first login, allows Matillion to find Redshift clusters. Also used when creating a new environment. Note: this privilege is not required in order to use a particular Redshift cluster: it’s just useful to make setup faster and less error prone.
    s3:ListAllMyBucketsEnables every S3 Staging Area drop-down list to work. Used in every Orchestration data load component. Note: Matillion ETL can run without this if you wish to keep your buckets private. However, exploration and discovery of buckets from within the client will no longer work.
    s3:ListBucketThis permission allows the user to access files in the folder (and any sub-folders). Can be used in every Orchestration data load component.
    s3:GetObjectEnables Matillion to read S3 files. Used in every Orchestration data load component.
    s3:PutObjectUsed by every Orchestration data load component to create temporary files in a nominated staging bucket. The files are used temporarily in order to take advantage of Redshift’s bulk loader and transfer the data as quickly as possible into Redshift. This Action could be restricted to specific staging buckets.
    s3:DeleteObjectUsed for the same purpose as s3:PutObject, to remove temporary S3 files created during data loads.
    s3:GetBucketLocationUses the subresource location to return a bucket's AWS region.
    secretsmanager:ListSecretsAllows permission to populate the secret name drop-down menu when configuring an external password located in a secret manager.
    secretsmanager:GetSecretValueAllows permission to resolve the value associated with a secret, for example when a component is configured to use a secret from the secret manager as a password.
    Note

    For more information about permission policy examples for AWS Secrets Manager, see here.

    Automated backups can be enabled from the Admin, then the Manage Backups menu to the top right of the page. Backups require the EC2 actions detailed below.

    EC2 ActionsDescription
    ec2:CreateSnapshotUsed during creation of the EC2 snapshot.
    ec2:CreateTagsUsed to tag the generated snapshot with a name.
    ec2:DescribeInstancesRequired to retrieve instance metadata.
    ec2:DescribeVolumesRequired to retrieve instance metadata.

    This section lists the actions required to use the KMS option in the Manage Passwords dialog. In addition to the below, the chosen KMS key must be:

    • In the same region as Matillion
    • Enabled

    The following tables outline actions for KMS, RDS, SQS, SNS and Cloudwatch, and are required by Matillion to perform certain functions.

    KMS ActionDescription
    kms:ListAliasesEnables Matillion to populate the "Master Key" dropdown by listing all the KMS aliases which are associated with a Key.
    kms:EncryptEnables Matillion to store an encrypted password.
    kms:DecryptEnables Matillion to retrieve and use an encrypted password.
    RDS ActionsDescription
    rds:DescribeDBInstancesEnables Matillion to list the endpoints of the chosen RDS database type. Note this privilege isn't required in order to query data from any RDS instance.
    SQS ActionsDescription
    sqs:ListQueuesEnables the operator to select an SQS queue in the main SQS Configuration window.
    sqs:GetQueueUrlRequired in order to monitor queues.
    sqs:ReceiveMessageEnables Matillion to receive messages from SQS (e.g. using the SQS Message orchestration component).
    sqs:DeleteMessageEnables Matillion to receive messages from SQS and remove them from the queue.
    sqs:SendMessageEnables Matillion to send messages to SQS.
    SNS ActionsDescription
    sns:PublishRequired in order to publish messages with SNS (e.g. Using the SNS Message component).
    sns:ListTopicsEnables the operator to select from a list of existing SNS topics (such as when using the SNS Message component.
    sns:CreateTopicIf the user manually types in a topic name which doesn’t already exist in that region, Matillion tries to create the new topic, which requires this privilege.
    CloudWatch ActionsDescription
    cloudwatch:ListMetricsRequired to enable the CloudWatch Publish component to list existing metrics.
    cloudwatch:PutMetricDataRequired in order to publish metrics to Cloudwatch using Matillion’s CloudWatch Publish component. This Action also includes the authorization to create new Custom Namespaces and new Metrics.

    To see the template for this policy type, refer to AWS policy.


    Credentials for CDC

    Below is a full list of the permissions a Matillion ETL instance requires to use CDC as described in the above sections. We provide a brief outline of what each permission allows Matillion ETL to do and which section of the CDC setup it pertains to.

    Lambda ActionDescriptionClient Section
    lambda:CreateFunctionCreate Lambda Functions.Manage
    lambda:DeleteFunctionDelete Lambda Functions.Manage
    lambda:GetFunctionGet information about the Lambda Function.Manage
    lambda:TagResourceAdd tags to a Lambda Function.Manage
    lambda:UpdateFunctionCodeUpdate a Lambda Function's code.Manage
    lambda:UpdateFunctionConfigurationModify the version-specifc settings of a Lambda function.Manage
    lambda:AddPermissionGet information about the Lambda Function.Manage
    lambda:RemovePermissionRevokes Lambda Function permission from an AWS Service.Manage
    lambda:GetPolicyGet the resource-based IAM policy for a Lambda Function.CDC Monitor
    SQS ActionDescription
    sqs:DeleteMessageDelete a message from a specified queue.Manage
    sqs:ReceiveMessageRetrieve up to 10 messages from a specified queue.Manage
    sqs:DeleteQueueDelete a queue specified by the QueueUrl, regardless of the queue's contents.Manage
    sqs:ListQueues (Optional)Allows discovery of available queues.Manage
    sqs:ChangeMessageVisibilityChange the visibility timeout of a message in a queue.Manage
    sqs:GetQueueUrlGet the URL of an Amazon SQS queue.Manage
    DMS ActionDescription
    dms:CreateEndpointCreate a DMS endpoint.CDC Task Config
    dms:DeleteEndpointDelete a DMS endpoint.CDC Task Config
    dms:StartReplicationTaskStart a replication task.CDC Monitor
    dms:CreateReplicationTaskCreate a replication task.CDC Task Config
    dms:ModifyEndpointModify a DMS endpointCDC Monitor
    dms:DescribeConnectionsGet the status of the connections between a replication instance and an endpoint.CDC Task Config
    dms:DescribeReplicationInstancesGet information on replication instances on your account (region-specific).CDC Task Config
    dms:DescribeReplicationTasksGet information on replication tasks on your account (region-specific).CDC Task Config
    dms:StopReplicationTaskStop a replication task.CDC Monitor
    dms:DeleteReplicationTaskDelete a replication task.CDC Monitor
    dms:DescribeEndpointsGet information on DMS endpoints on your account (region specific).CDC Task Config
    dms:TestConnectionTests the connection between a replication instance and a DMS endpoint.CDC Task Config
    S3 PermissionsDescription
    s3:GetBucketNotificationGet the notification configuration of a bucket.CDC Task Config
    s3:ListBucketView folders inside S3 buckets
    s3:PutBucketNotificationSet notification configuration of an S3 bucket.CDC Task Config
    s3:ListAllMyBuckets (Optional)Get a list of all owned S3 buckets.CDC Task Config
    IAM PermissionsDescription
    iam:ListRoles (Optional)Get a list of IAM Roles.Manage
    iam:PassRoleAllows passing a role to an AWS service.Manage
    EC2 PermissionsDescription
    ec2:DescribeRegions (Optional)Get the regions currently available to the instance.Manage

    DynamoDB

    The DynamoDB data staging components require some credentials to function correctly.

    DynamoDB ActionsDescription
    dynamodb:ListTablesReturn information about the table, including the current status of the table, when it was created, the primary key schema, and any indexes on the table.
    dynamodb:DescribeTableReturn an array of table names associated with the current account and endpoint.
    dynamodb:ScanScan a table or index.

    AWS policy

    The following AWS policy template grants fine-grained access control to your Matillion ETL instance:

    {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Sid": "StmtMinRedshift",
    "Effect": "Allow",
    "Action": [
    "redshift:DescribeClusters"
    ],
    "Resource": [
    "*"
    ]
    },
    {
    "Sid": "StmtMinS3",
    "Effect": "Allow",
    "Action": [
    "s3:ListAllMyBuckets",
    "s3:ListBucket",
    "s3:GetObject",
    "s3:PutObject",
    "s3:DeleteObject",
    "s3:GetBucketLocation"
    ],
    "Resource": [
    "*"
    ]
    },
    {
    "Sid": "StmtMinSQS",
    "Effect": "Allow",
    "Action": [
    "sqs:DeleteMessage",
    "sqs:ListQueues",
    "sqs:ReceiveMessage",
    "sqs:SendMessage",
    "sqs:GetQueueUrl"
    ],
    "Resource": [
    "*"
    ]
    },
    {
    "Sid": "StmtMinRDS",
    "Effect": "Allow",
    "Action": [
    "rds:DescribeDBInstances"
    ],
    "Resource": [
    "*"
    ]
    },
    {
    "Sid": "StmtMinEC2",
    "Effect": "Allow",
    "Action": [
    "ec2:CreateSnapshot",
    "ec2:CreateTags",
    "ec2:DescribeInstances",
    "ec2:DescribeVolumes"
    ],
    "Resource": [
    "*"
    ]
    },
    {
    "Sid": "StmtMinSNS",
    "Effect": "Allow",
    "Action": [
    "sns:ListTopics",
    "sns:CreateTopic",
    "sns:Publish"
    ],
    "Resource": [
    "*"
    ]
    },
    {
    "Sid": "StmtMinCloudwatch",
    "Effect": "Allow",
    "Action": [
    "cloudwatch:PutMetricData",
    "cloudwatch:ListMetrics"
    ],
    "Resource": [
    "*"
    ]
    },
    {
    "Sid": "StmtMinKMS",
    "Effect": "Allow",
    "Action": [
    "kms:ListAliases",
    "kms:Encrypt",
    "kms:Decrypt"
    ],
    "Resource": [
    "*"
    ]
    },
    {
    "Sid": "StmtMinDDB",
    "Effect": "Allow",
    "Action": [
    "dynamodb:ListTables",
    "dynamodb:DescribeTable",
    "dynamodb:Scan"
    ],
    "Resource": [
    "*"
    ]
    },
    {
    "Effect": "Allow",
    "Action": [
    "secretsmanager:CreateSecret",
    "secretsmanager:ListSecrets"
    ],
    "Resource": [
    "*"
    ]
    }
    ]
    }