Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Salesforce Query

I have an ETL that is using the salesforce query component. As i watch data being staged, it appears as if records are being staged roughly 5K rows at a time....and I am just wondering if each of those increments is a separate API call to the salesforce API?

Maybe a dumb question, but i know we have a finite amount of calls that can be made in a rolling 24 hour period, and just want to understand how the salesforce query component is working behind the scenes as it interacts with the API.

Thanks in advance!

2 Community Answers

Matillion Agent  

Ian Funnell —

Hi James,

Matillion pages through the Salesforce API calls according to a non-configurable page size, and then streams the data out to S3 according to another non-configurable byte limit. The net effect is the ~5K rows that you see in the logfile.

There is a “bulk mode” option with the Salesforce Query, which you can enable by going to the Connection Properties and setting UseBulkAPI to true. This will result in a smaller number of Salesforce API calls overall. Although note that you are not permitted to perform joins or aggregations when issuing a query with the bulk API.

Best regards,
Ian


James Nolen —

Thanks Ian, good to know.

Post Your Community Answer

To add an answer please login