I have an ETL that is using the salesforce query component. As i watch data being staged, it appears as if records are being staged roughly 5K rows at a time....and I am just wondering if each of those increments is a separate API call to the salesforce API?
Maybe a dumb question, but i know we have a finite amount of calls that can be made in a rolling 24 hour period, and just want to understand how the salesforce query component is working behind the scenes as it interacts with the API.
Thanks in advance!
2 Community Answers
Ian Funnell —
Matillion pages through the Salesforce API calls according to a non-configurable page size, and then streams the data out to S3 according to another non-configurable byte limit. The net effect is the ~5K rows that you see in the logfile.
There is a “bulk mode” option with the Salesforce Query, which you can enable by going to the Connection Properties and setting UseBulkAPI to true. This will result in a smaller number of Salesforce API calls overall. Although note that you are not permitted to perform joins or aggregations when issuing a query with the bulk API.