Connecting to an RDS in a Private VPC
For security reasons, your RDS database might be held in a Private VPC that doesn't allow public access. Your Matillion instance, which might be in a Public VPC, can still connect to this RDS database using VPC Peering.
In this article, we're going to look at how to set up VPC Peering to allow Matillion to use the created VPC Peer to copy data from the Private VPC. The concept is the same for any data held within the Private VPC.
1. Set up the Private VPC. A VPC in AWS is private if it doesn't have an Internet Gateway or an NAT Gateway that can be used to connect into it. This can be set up from the AWS Console.
2. The Route table for the VPC will only have an entry for local routes:
3. Set up the RDS instance. The RDS instance should be private. Even if it is set to be public-facing it will not be accessible because it is in a private VPC.
4. Create a Peering Connection. A Peering Connection is required to route traffic between two VPCs. These VPCs can be in the same AWS account, or in different accounts if required. To create a Peering Connection, select "Peering Connections" in the VPC Dashboard and click Create Peering Connection.
This Peering Request will appear in Peering Connections and now needs to be accepted by the account the VPC to peer with is in:
5. Once the request has been accepted, the route tables of both of the VPCs need to be updated to direct relevant traffic to use the Peering Connection. For the VPC that Matillion is in, all traffic to the IP associated with the Private VPC, in this case 10.1.0.0/22, needs to be directed to the Peering Connection:
For the RDS VPC, all traffic to the Matillion instance needs to be directed to use the Peering Connection. This was done by directing all other traffic to the Peering Connection but this IP range could be limited:
6. Update the RDS security group to allow traffic in from both to Matillion.
Matillion now has a private route to the RDS instance without having to use the internet. The RDS Query and RDS Bulk Load components will now work as expected from the Matillion instance.