Contents x
    Non-hub: Launching Matillion ETL via Azure CLI
      • Dark
        Light
      Contents

      Non-hub: Launching Matillion ETL via Azure CLI

        • Dark
          Light
        Article Summary

        Warning

        New customers must go through the Matillion Hub to select their preferred cloud provider and data warehouse to begin their Matillion ETL journey.

        Overview

        This guide explains how to find and launch existing Matillion ETL products via the Azure command line interface (CLI). You can use the Azure CLI for the following:

        • To create a "bring your own licence" (BYOL) Matillion ETL instance. Information about setting up all required Azure resources will be covered in this guide.
        • To launch a legacy marketplace listing for customers who aren't registered with the Matillion Hub.
        Note

        Make sure you choose a Matillion ETL instance size based on the supported Azure VM size in the Azure Instance Recommendations section of the Non-hub: Instance Sizes Guide.

        Prerequisites

        Prior to launching a Matillion ETL instance you need to be an existing non-hub customer, and are required to have:

        • Adequate knowledge about the cloud service account (AWS, Azure, GCP), and Cloud Data Warehouse (Snowflake, Redshift or Google BigQuery), you want to launch.
        • A user with admin permissions who can access the intended cloud service account.
        • Access to a cloud storage bucket (S3, Azure, Blob Storage or Google Cloud Storage) to house the transient staging files Matillion used to load data to the cloud.
        • A network path to access the intended data sources. This may involve working with your network team to enable access to on-premise databases.

        You are also required to:

        Note

        The commands provided in this guide are written for Bash. However, using similar commands you can generate the same results using PowerShell or other shells.

        Locating the available images

        You are required to locate URN values to launch the published Matillion ETL images using the relevant CLI. The published images are available in the following format:

        publisher:offer:sku:version

        Displayed in the table below, are the specific offers from Matillion ETL:

        PublisherOfferSKUPayment Method
        Matillionmatillion-etl-snowflakematillion-etl-for-snowflakeMarketplace billing
        Matillionmatillion-azure-synapsematillion-etl-for-synapseMarketplace billing
        Matillionmatillion-etl-snowflakematillion-etl-byolBring your own licence (BYOL)
        Matillionmatillion-azure-synapsematillion-etl-synapse-byolBring your own licence (BYOL)
        Matillionmatillionmatillion-etl-for-snowflakeMatillion Hub
        Matillionmatillionmatillion-etl-for-synapseMatillion Hub
        Matillionmatillionmatillion-etl-for-deltalakeMatillion Hub

        To find the specific versions for the table above, run the following command:

        az vm image list --publisher matillion --all | jq -r '.[].urn'

        An example of what this command will return is printed below:

        matillion:matillion:matillion-etl-for-deltalake:1.52.0
matillion:matillion:matillion-etl-for-deltalake:1.53.8
matillion:matillion:matillion-etl-for-snowflake:1.53.8
matillion:matillion:matillion-etl-for-synapse:1.53.8
matillion:matillion-etl-azure-synapse:matillion-etl-for-synapse:1.48.4
matillion:matillion-etl-azure-synapse:matillion-etl-for-synapse:1.50.9
matillion:matillion-etl-azure-synapse:matillion-etl-for-synapse:1.51.5
matillion:matillion-etl-azure-synapse:matillion-etl-for-synapse:1.53.8
matillion:matillion-etl-azure-synapse:matillion-etl-synapse-byol:1.53.8
matillion:matillion-etl-snowflake:matillion-etl-for-snowflake:1.48.4
matillion:matillion-etl-snowflake:matillion-etl-for-snowflake:1.50.9
matillion:matillion-etl-snowflake:matillion-etl-for-snowflake:1.51.5
matillion:matillion-etl-snowflake:matillion-etl-for-snowflake:1.53.8
matillion:matillion-etl-snowflake:matillion-etl-snowflake-byol:1.53.8

        Next, select the URN/product/version combination that you wish to launch, and continue to the next step. For the purpose of this example, the following URN will be used:

        matillion:matillion-etl-snowflake:matillion-etl-snowflake-byol:1.53.8

        Launching an image: Required resources

        To launch an image, the following resources must be created:

        The following dependencies are required for you to launch a Matillion ETL virtual machine and access it successfully:

        Examples for each of these resources are displayed in this document. In many cases, users will already have a pre-existing resource group, storage account, network and subnetwork, network security group, network security rule, and identity set up. Therefore, the next section of this document will provide instructions for setting up the IP, the NIC, and the virtual machine (VM).

        Creating an IP

        Explanation of the CLI command:

        CLI call




az network public-ip create \
--name example-public-ip \
--resource-group example-resource-group \
--location UKWest \
--allocation-method Dynamic

        Example CLI command:

        az network public-ip create \
--name example-public-ip \
--resource-group example-resource-group \
--location UKWest \
--allocation-method Dynamic

        Creating a network interface (NIC)

        Explanation of the CLI command:

        Network Interface Card Creation

        Example CLI command:

        az network nic create \
--name example-nic \
--resource-group example-resource-group \
--location UKWest \
--vnet-name example-network \
--subnet example-subnetwork \
--network-security-group example-nsg \
--public-ip-address example-public-ip

        Creating a virtual machine (VM)

        Explanation of the CLI command:

        











https://learn.microsoft.com/en-us/cli/azure/vm?view=azure-cli-latest#az_vm_create)

        Example CLI command:

        az vm create \
--name example-vm \
--resource-group example-resource-group \
--location UKWest \
--image matillion:matillion-etl-snowflake:matillion-etl-snowflake-byol:1.53.8 \
--use-unmanaged-disk \
--storage-account matillionsaexample \
--storage-container-name system \
--os-disk-name example-machine-disk.vhd \
--assign-identity example-managed-identity \
--nics example-nic \
--admin-username centos \
--generate-ssh-keys \
--ssh-dest-key-path /home/centos/.ssh/authorized_keys \
--os-disk-size-gb 40 \
--size Standard_B2s

        For the --assign-identity parameter you should specify the actual User Identity you want to associate with the VM. See Roles and Permissions (Azure) for details. This assumes you want to use a user-assigned identity. If you want to use a System Assigned Identity (which is unique to the VM) instead, use the --assign-identity parameter without specifying the name of a user identity:

        --assign-identity \

        The way you configure a managed identity will be dependent on how you want to control your Matillion ETL VM's access to a storage account. This is described in the article Roles and Permissions (Azure), which also describes how to review your existing Matillion ETL VM and how its identity is set up.

        Additionally, we recommend that users read Azure's documentation on Configure managed identities for Azure resources on an Azure VM using Azure CLI for more details and examples.

        The above example assumes you are creating a VM with an Unmanaged Disk, which requires the following three parameters:

        -use-unmanaged-disk \
--storage-account matillionsaexample \
--storage-container-name system \

        You can instead choose to create the VM with a Managed Disk, in which case these three parameters should be omitted. See here for an explanation of the differences between managed and unmanaged disks.

        Note

        When deploying with an unmanaged disk, the --os-disk-name can't have the same name as an existing file in the storage container you have specified.

        The recommended typical default disk size (specified with --os-disk-size-gb) is 40GB. Matillion has found that this size suits most user's needs; however, you might need to look at the size of your existing Matillion ETL VM and see how much space is available to determine whether 40GB is sufficient for your specific use.

        The biggest consumer of disk space on a Matillion ETL VM is typically the on-instance Postgres database. This is Matillion ETL's "persistence layer", where all things created by users will be stored. Typically, the biggest consumer of space in the Postgres database is task history information. Matillion ETL will by default retain 365 days of task history, after which it will be purged. The frequency and volume of job executions and the messages generated by those jobs will be the biggest factor in the overall storage footprint of task history.

        Final launching steps

        Once all of the above has been accomplished, Matillion ETL can be accessed (you may need to wait a few minutes for the image to be prepared).

        Error

        If the End User Licence Agreement (EULA) hasn't been accepted from the product before launching, you may receive an error like the below:

        Message: Offer with PublisherId: 'matillion', OfferId: 'matillion' cannot be purchased due to validation errors. For more information see details. Correlation Id: '33a6e641-d6cb-4f6c-999a-34aaff514739' You have not accepted the legal terms on this subscription: '47b72173-e519-41b6-afd1-10018741e1d3' for this plan. Before the subscription can be used, you need to accept the legal terms of the image. To read and accept legal terms, use the Azure CLI commands described at https://go.microsoft.com/fwlink/?linkid=2110637 or the PowerShell commands available at https://go.microsoft.com/fwlink/?linkid=862451.

        To fix or prevent this error, accept the terms of the URN by running this command:

        az vm image terms accept --subscription <your-subscription-id> --urn <product urn>

        Once you have run this command, run the create virtual machine (VM) command again.

        Command responses (optional reading)

        This section includes the responses to the commands made above.

        Creating an IP response

        {
"publicIp": {
"ddosSettings": null,
"dnsSettings": null,
"etag": "W/\"a0a4aafb-ccf4-417b-979e-60313b016c1e\"",
"extendedLocation": null,
"id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/publicIPAddresses/example-public-ip",
"idleTimeoutInMinutes": 4,
"ipAddress": null,
"ipConfiguration": null,
"ipTags": [],
"location": "ukwest",
"name": "example-public-ip",
"provisioningState": "Succeeded",
"publicIpAddressVersion": "IPv4",
"publicIpAllocationMethod": "Dynamic",
"publicIpPrefix": null,
"resourceGroup": "example-resource-group",
"resourceGuid": "82455538-8034-4803-84f1-1fd10d97653e",
"sku": {
"name": "Basic",
"tier": "Regional"
},
"tags": null,
"type": "Microsoft.Network/publicIPAddresses",
"zones": null
}
}

        Creating a network interface (NIC) response

        {
"NewNIC": {
"dnsSettings": {
"appliedDnsServers": [],
"dnsServers": [],
"internalDnsNameLabel": null,
"internalDomainNameSuffix": "mqc30qfsjbvu5ebczbbi3bz3hc.cwx.internal.cloudapp.net",
"internalFqdn": null
},
"dscpConfiguration": null,
"enableAcceleratedNetworking": false,
"enableIpForwarding": false,
"etag": "W/\"aa2272f0-9f56-4cb5-b135-fd3d9a0b7d11\"",
"extendedLocation": null,
"hostedWorkloads": [],
"id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/networkInterfaces/example-nic",
"ipConfigurations": [
{
"applicationGatewayBackendAddressPools": null,
"applicationSecurityGroups": null,
"etag": "W/\"aa2272f0-9f56-4cb5-b135-fd3d9a0b7d11\"",
"id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/networkInterfaces/example-nic/ipConfigurations/ipconfig1",
"loadBalancerBackendAddressPools": null,
"loadBalancerInboundNatRules": null,
"name": "ipconfig1",
"primary": true,
"privateIpAddress": "10.0.0.4",
"privateIpAddressVersion": "IPv4",
"privateIpAllocationMethod": "Dynamic",
"privateLinkConnectionProperties": null,
"provisioningState": "Succeeded",
"publicIpAddress": {
"ddosSettings": null,
"dnsSettings": null,
"etag": null,
"extendedLocation": null,
"id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/publicIPAddresses/example-public-ip",
"idleTimeoutInMinutes": null,
"ipAddress": null,
"ipConfiguration": null,
"ipTags": null,
"location": null,
"name": null,
"provisioningState": null,
"publicIpAddressVersion": null,
"publicIpAllocationMethod": null,
"publicIpPrefix": null,
"resourceGroup": "example-resource-group",
"resourceGuid": null,
"sku": null,
"tags": null,
"type": null,
"zones": null
},
"resourceGroup": "example-resource-group",
"subnet": {
"addressPrefix": null,
"addressPrefixes": null,
"delegations": null,
"etag": null,
"id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/virtualNetworks/example-network/subnets/example-subnetwork",
"ipAllocations": null,
"ipConfigurationProfiles": null,
"ipConfigurations": null,
"name": null,
"natGateway": null,
"networkSecurityGroup": null,
"privateEndpointNetworkPolicies": null,
"privateEndpoints": null,
"privateLinkServiceNetworkPolicies": null,
"provisioningState": null,
"purpose": null,
"resourceGroup": "example-resource-group",
"resourceNavigationLinks": null,
"routeTable": null,
"serviceAssociationLinks": null,
"serviceEndpointPolicies": null,
"serviceEndpoints": null
},
"type": "Microsoft.Network/networkInterfaces/ipConfigurations",
"virtualNetworkTaps": null
}
],
"location": "ukwest",
"macAddress": null,
"name": "example-nic",
"networkSecurityGroup": {
"defaultSecurityRules": null,
"etag": null,
"flowLogs": null,
"id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/networkSecurityGroups/example-nsg",
"location": null,
"name": null,
"networkInterfaces": null,
"provisioningState": null,
"resourceGroup": "example-resource-group",
"resourceGuid": null,
"securityRules": null,
"subnets": null,
"tags": null,
"type": null
},
"primary": null,
"privateEndpoint": null,
"provisioningState": "Succeeded",
"resourceGroup": "example-resource-group",
"resourceGuid": "dce4915d-9be1-46ec-8c19-9e4d66c11ca5",
"tags": null,
"tapConfigurations": [],
"type": "Microsoft.Network/networkInterfaces",
"virtualMachine": null
}
}

        Creating a virtual machine (VM) response

        {
"fqdns": "",
"id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Compute/virtualMachines/example-vm",
"identity": {
"systemAssignedIdentity": "",
"userAssignedIdentities": {
"/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.ManagedIdentity/userAssignedIdentities/example-managed-identity": {
"clientId": "3f7da7e7-dda6-46de-82d2-8c82a8bd9d09",
"principalId": "1b0be934-b012-416f-b1a8-79b8cc40ec30"
}
}
},
"location": "ukwest",
"macAddress": "00-22-48-1A-2F-92",
"powerState": "VM running",
"privateIpAddress": "10.0.0.4",
"publicIpAddress": "51.104.44.10",
"resourceGroup": "example-resource-group",
"zones": ""
}

        Commands for additional resources

        This section provides explanations, examples, and responses for the commands for the following Azure resources:

        • Resource groups
        • Storage accounts
        • Networks and subnetworks
        • Network security groups
        • Network security rules
        • Identities

        If you already have any of these resources set up, you don't need to create another.

        Creating a resource group

        Explanation of the CLI command:

        Example CLI command:

        az group create \
--name example-resource-group \
--location UKWest

        Example response:

        {
"id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group",
"location": "ukwest",
"managedBy": null,
"name": "example-resource-group",
"properties": {
"provisioningState": "Succeeded"
},
"tags": null,
"type": "Microsoft.Resources/resourceGroups"
}

        Creating a storage account

        Explanation of the CLI command:

        Example CLI command:

        az storage account create \
--name matillionsaexample \
--resource-group example-resource-group

        Example response:

        {
"accessTier": "Hot",
"allowBlobPublicAccess": null,
"azureFilesIdentityBasedAuthentication": null,
"blobRestoreStatus": null,
"creationTime": "2021-04-24T08:13:22.972959+00:00",
"customDomain": null,
"enableHttpsTrafficOnly": true,
"encryption": {
"keySource": "Microsoft.Storage",
"keyVaultProperties": null,
"requireInfrastructureEncryption": null,
"services": {
"blob": {
"enabled": true,
"keyType": "Account",
"lastEnabledTime": "2021-04-24T08:13:23.066708+00:00"
},
"file": {
"enabled": true,
"keyType": "Account",
"lastEnabledTime": "2021-04-24T08:13:23.066708+00:00"
},
"queue": null,
"table": null
}
},
"failoverInProgress": null,
"geoReplicationStats": null,
"id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Storage/storageAccounts/matillionsaexample",
"identity": null,
"isHnsEnabled": null,
"kind": "StorageV2",
"largeFileSharesState": null,
"lastGeoFailoverTime": null,
"location": "ukwest",
"minimumTlsVersion": null,
"name": "matillionsaexample",
"networkRuleSet": {
"bypass": "AzureServices",
"defaultAction": "Allow",
"ipRules": [],
"virtualNetworkRules": []
},
"primaryEndpoints": {
"blob": "https://matillionsaexample.blob.core.windows.net/",
"dfs": "https://matillionsaexample.dfs.core.windows.net/",
"file": "https://matillionsaexample.file.core.windows.net/",
"internetEndpoints": null,
"microsoftEndpoints": null,
"queue": "https://matillionsaexample.queue.core.windows.net/",
"table": "https://matillionsaexample.table.core.windows.net/",
"web": "https://matillionsaexample.z35.web.core.windows.net/"
},
"primaryLocation": "ukwest",
"privateEndpointConnections": [],
"provisioningState": "Succeeded",
"resourceGroup": "example-resource-group",
"routingPreference": null,
"secondaryEndpoints": {
"blob": "https://matillionsaexample-secondary.blob.core.windows.net/",
"dfs": "https://matillionsaexample-secondary.dfs.core.windows.net/",
"file": null,
"internetEndpoints": null,
"microsoftEndpoints": null,
"queue": "https://matillionsaexample-secondary.queue.core.windows.net/",
"table": "https://matillionsaexample-secondary.table.core.windows.net/",
"web": "https://matillionsaexample-secondary.z35.web.core.windows.net/"
},
"secondaryLocation": "uksouth",
"sku": {
"name": "Standard_RAGRS",
"tier": "Standard"
},
"statusOfPrimary": "available",
"statusOfSecondary": "available",
"tags": {},
"type": "Microsoft.Storage/storageAccounts"
}

        Creating a network and subnetwork

        Explanation of the CLI command:

        Example CLI command:

        CLI call
az network vnet create \
--name example-network \
--resource-group example-resource-group \
--address-prefixes 10.0.0.0/16 \
--subnet-name example-subnetwork \
--subnet-prefixes 10.0.0.0/24

        Example response:

        {
"newVNet": {
"addressSpace": {
"addressPrefixes": [
"10.0.0.0/16"
]
},
"bgpCommunities": null,
"ddosProtectionPlan": null,
"dhcpOptions": {
"dnsServers": []
},
"enableDdosProtection": false,
"enableVmProtection": null,
"etag": "W/\"a1d5b34b-ebe8-42c5-a29f-4bd345876036\"",
"extendedLocation": null,
"id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/virtualNetworks/example-network",
"ipAllocations": null,
"location": "ukwest",
"name": "example-network",
"provisioningState": "Succeeded",
"resourceGroup": "example-resource-group",
"resourceGuid": "40dd0564-48b2-4f6b-9022-c8428e873d3a",
"subnets": [
{
"addressPrefix": "10.0.0.0/24",
"addressPrefixes": null,
"delegations": [],
"etag": "W/\"a1d5b34b-ebe8-42c5-a29f-4bd345876036\"",
"id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/virtualNetworks/example-network/subnets/example-subnetwork",
"ipAllocations": null,
"ipConfigurationProfiles": null,
"ipConfigurations": null,
"name": "example-subnetwork",
"natGateway": null,
"networkSecurityGroup": null,
"privateEndpointNetworkPolicies": "Enabled",
"privateEndpoints": null,
"privateLinkServiceNetworkPolicies": "Enabled",
"provisioningState": "Succeeded",
"purpose": null,
"resourceGroup": "example-resource-group",
"resourceNavigationLinks": null,
"routeTable": null,
"serviceAssociationLinks": null,
"serviceEndpointPolicies": null,
"serviceEndpoints": null,
"type": "Microsoft.Network/virtualNetworks/subnets"
}
],
"tags": {},
"type": "Microsoft.Network/virtualNetworks",
"virtualNetworkPeerings": []
}
}

        Creating a network security group

        Explanation of the CLI command:

        Example CLI command:

        az network nsg create \
--name example-nsg \
--resource-group example-resource-group

        Example response:

        {
"NewNSG": {
"defaultSecurityRules": [
{
"access": "Allow",
"description": "Allow inbound traffic from all VMs in VNET",
"destinationAddressPrefix": "VirtualNetwork",
"destinationAddressPrefixes": [],
"destinationApplicationSecurityGroups": null,
"destinationPortRange": "*",
"destinationPortRanges": [],
"direction": "Inbound",
"etag": "W/\"02367ddb-b299-413e-bd71-9ac4ca20a535\"",
"id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/networkSecurityGroups/example-nsg/defaultSecurityRules/AllowVnetInBound",
"name": "AllowVnetInBound",
"priority": 65000,
"protocol": "*",
"provisioningState": "Succeeded",
"resourceGroup": "example-resource-group",
"sourceAddressPrefix": "VirtualNetwork",
"sourceAddressPrefixes": [],
"sourceApplicationSecurityGroups": null,
"sourcePortRange": "*",
"sourcePortRanges": [],
"type": "Microsoft.Network/networkSecurityGroups/defaultSecurityRules"
},
{
"access": "Allow",
"description": "Allow inbound traffic from azure load balancer",
"destinationAddressPrefix": "*",
"destinationAddressPrefixes": [],
"destinationApplicationSecurityGroups": null,
"destinationPortRange": "*",
"destinationPortRanges": [],
"direction": "Inbound",
"etag": "W/\"02367ddb-b299-413e-bd71-9ac4ca20a535\"",
"id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/networkSecurityGroups/example-nsg/defaultSecurityRules/AllowAzureLoadBalancerInBound",
"name": "AllowAzureLoadBalancerInBound",
"priority": 65001,
"protocol": "*",
"provisioningState": "Succeeded",
"resourceGroup": "example-resource-group",
"sourceAddressPrefix": "AzureLoadBalancer",
"sourceAddressPrefixes": [],
"sourceApplicationSecurityGroups": null,
"sourcePortRange": "*",
"sourcePortRanges": [],
"type": "Microsoft.Network/networkSecurityGroups/defaultSecurityRules"
},
{
"access": "Deny",
"description": "Deny all inbound traffic",
"destinationAddressPrefix": "*",
"destinationAddressPrefixes": [],
"destinationApplicationSecurityGroups": null,
"destinationPortRange": "*",
"destinationPortRanges": [],
"direction": "Inbound",
"etag": "W/\"02367ddb-b299-413e-bd71-9ac4ca20a535\"",
"id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/networkSecurityGroups/example-nsg/defaultSecurityRules/DenyAllInBound",
"name": "DenyAllInBound",
"priority": 65500,
"protocol": "*",
"provisioningState": "Succeeded",
"resourceGroup": "example-resource-group",
"sourceAddressPrefix": "*",
"sourceAddressPrefixes": [],
"sourceApplicationSecurityGroups": null,
"sourcePortRange": "*",
"sourcePortRanges": [],
"type": "Microsoft.Network/networkSecurityGroups/defaultSecurityRules"
},
{
"access": "Allow",
"description": "Allow outbound traffic from all VMs to all VMs in VNET",
"destinationAddressPrefix": "VirtualNetwork",
"destinationAddressPrefixes": [],
"destinationApplicationSecurityGroups": null,
"destinationPortRange": "*",
"destinationPortRanges": [],
"direction": "Outbound",
"etag": "W/\"02367ddb-b299-413e-bd71-9ac4ca20a535\"",
"id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/networkSecurityGroups/example-nsg/defaultSecurityRules/AllowVnetOutBound",
"name": "AllowVnetOutBound",
"priority": 65000,
"protocol": "*",
"provisioningState": "Succeeded",
"resourceGroup": "example-resource-group",
"sourceAddressPrefix": "VirtualNetwork",
"sourceAddressPrefixes": [],
"sourceApplicationSecurityGroups": null,
"sourcePortRange": "*",
"sourcePortRanges": [],
"type": "Microsoft.Network/networkSecurityGroups/defaultSecurityRules"
},
{
"access": "Allow",
"description": "Allow outbound traffic from all VMs to Internet",
"destinationAddressPrefix": "Internet",
"destinationAddressPrefixes": [],
"destinationApplicationSecurityGroups": null,
"destinationPortRange": "*",
"destinationPortRanges": [],
"direction": "Outbound",
"etag": "W/\"02367ddb-b299-413e-bd71-9ac4ca20a535\"",
"id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/networkSecurityGroups/example-nsg/defaultSecurityRules/AllowInternetOutBound",
"name": "AllowInternetOutBound",
"priority": 65001,
"protocol": "*",
"provisioningState": "Succeeded",
"resourceGroup": "example-resource-group",
"sourceAddressPrefix": "*",
"sourceAddressPrefixes": [],
"sourceApplicationSecurityGroups": null,
"sourcePortRange": "*",
"sourcePortRanges": [],
"type": "Microsoft.Network/networkSecurityGroups/defaultSecurityRules"
},
{
"access": "Deny",
"description": "Deny all outbound traffic",
"destinationAddressPrefix": "*",
"destinationAddressPrefixes": [],
"destinationApplicationSecurityGroups": null,
"destinationPortRange": "*",
"destinationPortRanges": [],
"direction": "Outbound",
"etag": "W/\"02367ddb-b299-413e-bd71-9ac4ca20a535\"",
"id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/networkSecurityGroups/example-nsg/defaultSecurityRules/DenyAllOutBound",
"name": "DenyAllOutBound",
"priority": 65500,
"protocol": "*",
"provisioningState": "Succeeded",
"resourceGroup": "example-resource-group",
"sourceAddressPrefix": "*",
"sourceAddressPrefixes": [],
"sourceApplicationSecurityGroups": null,
"sourcePortRange": "*",
"sourcePortRanges": [],
"type": "Microsoft.Network/networkSecurityGroups/defaultSecurityRules"
}
],
"etag": "W/\"02367ddb-b299-413e-bd71-9ac4ca20a535\"",
"flowLogs": null,
"id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/networkSecurityGroups/example-nsg",
"location": "ukwest",
"name": "example-nsg",
"networkInterfaces": null,
"provisioningState": "Succeeded",
"resourceGroup": "example-resource-group",
"resourceGuid": "def8de26-7836-4bb7-8737-9a55b27c862b",
"securityRules": [],
"subnets": null,
"tags": null,
"type": "Microsoft.Network/networkSecurityGroups"
}
}

        Creating a network security rule

        Note

        The networking policy can be locked down once the initial launch is established and working as expected based upon customer-specific networking policies. Matillion ETL needs 80 or 8080 (HTTP) and 443 or 8443 (HTTPS) traffic.

        Explanation of the CLI command:

        Example CLI command:

        az network nsg rule create \
--name example-nsg-rule \
--resource-group example-resource-group \
--nsg-name example-nsg \
--description "Example description of rule for Matillion Traffic." \
--priority 1000 \
--source-address-prefixes '*' \
--source-port-ranges '*' \
--destination-address-prefixes '*' \
--destination-port-ranges '*' \
--access Allow \
--protocol Tcp

        Example response:

        {
"access": "Allow",
"description": "Example description of rule for Matillion Traffic.",
"destinationAddressPrefix": "*",
"destinationAddressPrefixes": [],
"destinationApplicationSecurityGroups": null,
"destinationPortRange": null,
"destinationPortRanges": [
"*"
],
"direction": "Inbound",
"etag": "W/\"b361c7f5-8a1c-497f-8bf5-fe4ad16e52db\"",
"id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/networkSecurityGroups/example-nsg/securityRules/example-nsg-rule",
"name": "example-nsg-rule",
"priority": 1000,
"protocol": "Tcp",
"provisioningState": "Succeeded",
"resourceGroup": "example-resource-group",
"sourceAddressPrefix": "*",
"sourceAddressPrefixes": [],
"sourceApplicationSecurityGroups": null,
"sourcePortRange": null,
"sourcePortRanges": [
"*"
],
"type": "Microsoft.Network/networkSecurityGroups/securityRules"
}

        Creating a managed identity

        Explanation of the CLI command:

        Example CLI command:

        az identity create \
--name example-managed-identity \
--resource-group example-resource-group

        Example response:

        {
"clientId": "3f7da7e7-dda6-46de-82d2-8c82a8bd9d09",
"clientSecretUrl": "<redacted>"",
"id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourcegroups/example-resource-group/providers/Microsoft.ManagedIdentity/userAssignedIdentities/example-managed-identity",
"location": "ukwest",
"name": "example-managed-identity",
"principalId": "1b0be934-b012-416f-b1a8-79b8cc40ec30",
"resourceGroup": "example-resource-group",
"tags": {},
"tenantId": "09d19996-a185-4b6c-8332-37120f9bba10",
"type": "Microsoft.ManagedIdentity/userAssignedIdentities"
}

        Viewing and managing Matillion ETL resources

        You can view your billable resources in the Azure Portal by clicking into Resource Groups, and then selecting your resource group that you created earlier.

        What's Next