Do you have further details on the Clustered/Highly Available configuration of Matillion? I'm particularly interested in what additional AWS components are required (e.g. RDS instances) and whether there is an AWS architecture diagram showing the setup.
2 Community Answers
Ian Funnell —
A Clustered/Highly Available mode is available for our Redshift product. It fundamentally comprises multiple peer instances of Matillion which share the workload.
You’re right about extra AWS components. These include:
An RDS database to hold metadata (replacing the on-instance MongoDB which performs the same function in a single-instance deployment)
An ALB load balancer, which distributes the HTTP sessions between the two instances. You aim your browser at the ALB and not at one of the instances.
There’s more about considerations for designing a Job for a High Availability cluster in this document
When launching a HA cluster, we provide various CloudFormation templates according to what infrastructure you already have in place.
It’s possible to migrate from a single-instance deployment to an HA cluster using the Server Migration Tool