We have an NRT job kicking off using a Lambda and SQS whenever a trigger file is dropped in a certain bucket. We have seen that during Redshift maintenance window the job fails since Redshift cluster is down during that time. Is there an automated way of 'Managing SQS Configuration' Listen queue so that we can disable for about an hour (during Redshift maintenenace window) and enable it again after an hour? I understand by disabling the Listen queue tasks will just be queued up and will be resumed once SQS Listen queue is enabled again.
1 Community Answers
Ian Funnell —
AWS don’t seem to offer a way to temporarily make queues stop accepting new messages, or to temporarily hide messages from clients. So the main options I can think of for this are:
Switch off Matillion itself during the Redshift downtime. You could do this using a scheduled Lambda function
Keep Matillion running 24/7, but if it receives an SQS instruction when the maintenance window is near, invoke a 90-minute sleep before actually doing anything. Our microbatch replication article mentions avoiding maintenance windows, so you could use some logic like that. To sleep for 90 minutes you could put a Bash Script sleep 60 inside a loop from 1..90.