I am looking at the Lambda created by Matillion for CDC and I see group_id and event in the message - the lambda is sending to SQS queue. I didn't find documentation related to these two items - what are they referring to? Also, I did not see job_id in the SQS message so how Matillion is routing the correct message to the correct job? (if I create multiple CDC tasks) ?
Also, I am planning to send a Grid variable along with other variables in this queue but it is not getting sent to the CDC job. Am I not allowed to send the Grid variable just like in regular messages?
All great questions. I can’t speak directly to that functionality, though I would have expected to see something about the job as well. Perhaps that’s included in the event? The SQS Queue is different than the one Matillion listens on to start Orchestration jobs, so it’s quite conceivable that the message is different.
As for passing Grid Variables, I would expect that the Grid Variables are included in the message, but again, the process that reads the message would have to know how to consume them, along with the Shared job that’s configured with the CDC task.
I’m going to dig deeper into these questions, to see if I can get confirmation.
I did confirm that the message format used for CDC, and its associated queue, are different from the message used to start Matillion Orchestration jobs. There is no option to pass additional variables to it, as such.
The CDC tasks are differentiated by the values found in the event, so no additional work is needed to separate one CDC Task from another.
If you could provide some details about your use case, perhaps we can suggest an alternative.