Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Group_Id and event in Matillion CDC

I am looking at the Lambda created by Matillion for CDC and I see group_id and event in the message - the lambda is sending to SQS queue. I didn't find documentation related to these two items - what are they referring to? Also, I did not see job_id in the SQS message so how Matillion is routing the correct message to the correct job? (if I create multiple CDC tasks) ?

"group_id": "S3_CREATION_EVENT:bucket_name:key", "event": "S3_CREATION_EVENT"

Also, I am planning to send a Grid variable along with other variables in this queue but it is not getting sent to the CDC job. Am I not allowed to send the Grid variable just like in regular messages?

"gridVariables" : {
"<GridVar Name>": [["<R1C1 Value>", "<R1C2 Value>"], ["<R2C1 Value>", "<R2C2 Value>"]]
}

4 Community Answers

Matillion Agent  

Dan D'Orazio —

Hi Rajish -

All great questions. I can’t speak directly to that functionality, though I would have expected to see something about the job as well. Perhaps that’s included in the event? The SQS Queue is different than the one Matillion listens on to start Orchestration jobs, so it’s quite conceivable that the message is different.

As for passing Grid Variables, I would expect that the Grid Variables are included in the message, but again, the process that reads the message would have to know how to consume them, along with the Shared job that’s configured with the CDC task.

I’m going to dig deeper into these questions, to see if I can get confirmation.

Best -
Dan


Rajish Shakya —

Thank you and I am looking forward to hear about them. Thanks in advance. - Rajish


Rajish Shakya —

Also, it looks like I can't pass any other job variable - other than what it is defined by Matillion itself. Can you please confirm that as well?


Matillion Agent  

Dan D'Orazio —

Hi Rajish -

I did confirm that the message format used for CDC, and its associated queue, are different from the message used to start Matillion Orchestration jobs. There is no option to pass additional variables to it, as such.

The CDC tasks are differentiated by the values found in the event, so no additional work is needed to separate one CDC Task from another.

If you could provide some details about your use case, perhaps we can suggest an alternative.

Best -
Dan

Post Your Community Answer

To add an answer please login