Recurly Extract
    • Dark
      Light

    Recurly Extract

    • Dark
      Light

    Article Summary

    This article is specific to the following platforms - Snowflake - Redshift - BigQuery.

    Recurly Extract

    The Recurly Extract component calls the Recurly API to retrieve and store data to be either referenced by an external table or loaded into a table, depending on the user's cloud data warehouse. Users can then transform their data with the Matillion ETL library of transformation components.

    Using this component may return structured data that requires flattening. For help with flattening such data, we recommend using the Nested Data Load Component for Amazon Redshift and the Extract Nested Data Component for Snowflake or Google BigQuery.


    Properties

    Snowflake Properties

    PropertySettingDescription
    NameStringA human-readable name for the component.
    Data SourceSelectSelect a data source. As noted above, once you have configured the Data Source property, one or more properties specific to that data source will become available to configure. These properties are not optional and must be configured.
    Please refer to the "Data Source Properties" table in this documentation for guidance with these additional properties.
    API KeyString or SelectThe working API key used to connect to Recurly. For more information, please visit our support site's Recurly Authentication Guide.
    Matillion recommends using the Password Manager feature to securely store passwords and keys.
    SubdomainStringThe subdomain of your Recurly account, for example:
    https://<subdomain>.recurly.com/v2/accounts
    Page LimitIntegerSet the page limit for the amount of records to be returned and staged.
    LocationStorage LocationProvide an S3 bucket path, GCS bucket path, or Azure Blob Storage path that will be used to store the data. Once on an S3 bucket, GCS bucket or Azure Blob, the data can be referenced by an external table. A folder will be created at this location with the same name as the Target Table.
    IntegrationSelect(GCP only) Choose your Google Cloud Storage Integration. Integrations are required to permit Snowflake to read data from and write to a Google Cloud Storage bucket. Integrations must be set up in advance of selecting them in Matillion ETL. To learn more about setting up a storage integration, read our Storage Integration Setup Guide.
    WarehouseSelectChoose a Snowflake warehouse that will run the load.
    DatabaseSelectChoose a database to create the new table in.
    SchemaSelectSelect the table schema. The special value, [Environment Default], will use the schema defined in the environment. For more information on using multiple schemas, see this article.
    Target TableStringProvide a new table name.
    Warning: This table will be recreated and will drop any existing table of the same name.

    Redshift Properties

    PropertySettingDescription
    NameStringA human-readable name for the component.
    Data SourceSelectSelect a data source. As noted above, once you have configured the Data Source property, one or more properties specific to that data source will become available to configure. These properties are not optional and must be configured.
    Please refer to the "Data Source Properties" table in this documentation for guidance with these additional properties.
    API KeyString or SelectThe working API key used to connect to Recurly. For more information, please visit our support site's Recurly Authentication Guide.
    Matillion recommends using the Password Manager feature to securely store passwords and keys.
    SubdomainStringThe subdomain of your Recurly account, for example:
    https://<subdomain>.recurly.com/v2/accounts
    Page LimitIntegerSet the page limit for the amount of records to be returned and staged.
    LocationStorage LocationProvide an S3 bucket path, GCS bucket path, or Azure Blob Storage path that will be used to store the data. Once on an S3 bucket, GCS bucket or Azure Blob, the data can be referenced by an external table. A folder will be created at this location with the same name as the Target Table.
    TypeDropdownSelect between a standard table and an external table.
    Standard SchemaDropdownSelect the Redshift schema. The special value, [Environment Default], will use the schema defined in the Matillion ETL environment.
    External SchemaSelectSelect the table's external schema. To learn more about external schemas, please consult the 'Configuring The Matillion ETL Client' section of the Getting Started With Amazon Redshift Spectrum documentation.
    Target TableStringProvide a name for the external table to be used.
    Warning: This table will be recreated and will drop any existing table of the same name.

    BigQuery Properties

    PropertySettingDescription
    NameStringA human-readable name for the component.
    Data SourceSelectSelect a data source. As noted above, once you have configured the Data Source property, one or more properties specific to that data source will become available to configure. These properties are not optional and must be configured.
    Please refer to the "Data Source Properties" table in this documentation for guidance with these additional properties.
    API KeyString or SelectThe working API key used to connect to Recurly. For more information, please visit our support site's Recurly Authentication Guide.
    Matillion recommends using the Password Manager feature to securely store passwords and keys.
    SubdomainStringThe subdomain of your Recurly account, for example:
    https://<subdomain>.recurly.com/v2/accounts
    Page LimitIntegerSet the page limit for the amount of records to be returned and staged.
    Table TypeSelectSelect whether the table is Native (by default in BigQuery) or an external table.
    ProjectSelectSelect the Google Bigquery project. The special value, [Environment Default], will use the project defined in the environment.
    For more information, refer to the BigQuery documentation.
    DatasetSelectSelect the Google Bigquery dataset to load data into. The special value, [Environment Default], will use the dataset defined in the environment.
    For more information, refer to the BigQuery documentation.
    Target TableStringA name for the table.
    Warning: This table will be recreated and will drop any existing table of the same name.
    Only available when the table type is Native.
    New Target TableStringA name for the new external table.
    Only available when the table type is External.
    Cloud Storage Staging AreaCloud Storage BucketSpecify the target Google Cloud Storage bucket to be used for staging the queried data. Users can either:
    1. Input the URL string of the Cloud Storage bucket following the template provided: gs://<bucket>/<path>
    2. Navigate through the file structure to select the target bucket.

    Only available when the table type is Native.
    LocationCloud Storage BucketSpecify the target Google Cloud Storage bucket to be used for staging the queried data. Users can either:
    1. Input the URL string of the Cloud Storage bucket following the template provided: gs://<bucket>/<path>
    2. Navigate through the file structure to select the target bucket.
    Only available when the table type is External.
    Load OptionsMultiple SelectClean Cloud Storage Files: Destroy staged files on Cloud Storage after loading data. Default is On.
    Cloud Storage File Prefix: Give staged file names a prefix of your choice. The default setting is an empty field.
    Recreate Target Table: Choose whether the component recreates its target table before the data load. If Off, the component will use an existing table or create one if it does not exist. Default is On.
    Use Grid Variable: Check this checkbox to use a grid variable. This box is unchecked by default.



    Data Source Properties

    The following table lists any Data Source that requires one or more unique component properties for configuration. If a Data Source is missing from this table, it does NOT have any unique component properties.

    Data SourceProperty NameTypeDescription
    accountsStateSelectSelect the state of the accounts. Select one of "active", "closed", "non_subscriber", "past_due", or "subscriber".
    Begin TimeDatetimeOperates on the attribute specified by the Sort property. Filters data records to only include those with datetimes greater than or equal to the supplied datetime. Accepts an ISO 8601 date or date and time.
    End TimeDatetimeOperates on the attributes specified by the Sort property. Filters data records to only include those with datetimes less than or equal to the supplied datetime. Accepts an ISO 8601 date or date and time.
    coupon_redemptions_for_accountBegin TimeDatetimeOperates on the attribute specified by the Sort property. Filters data records to only include those with datetimes greater than or equal to the supplied datetime. Accepts an ISO 8601 date or date and time.
    End TimeDatetimeOperates on the attributes specified by the Sort property. Filters data records to only include those with datetimes less than or equal to the supplied datetime. Accepts an ISO 8601 date or date and time.
    Account CodeStringA unique identifier used by Matillion ETL to identify the Recurly account. This string may contain the following characters:
    a-z 0-9 @-_.

    For more information, please refer to the Recurly Documentation.
    coupon_redemptions_for_invoiceBegin TimeDatetimeOperates on the attribute specified by the Sort property. Filters data records to only include those with datetimes greater than or equal to the supplied datetime. Accepts an ISO 8601 date or date and time.
    End TimeDatetimeOperates on the attributes specified by the Sort property. Filters data records to only include those with datetimes less than or equal to the supplied datetime. Accepts an ISO 8601 date or date and time.
    Invoice codeIntegerThe number of the invoice.
    coupon_redemptions_for_subscriptionBegin TimeDatetimeOperates on the attribute specified by the Sort property. Filters data records to only include those with datetimes greater than or equal to the supplied datetime. Accepts an ISO 8601 date or date and time.
    End TimeDatetimeOperates on the attributes specified by the Sort property. Filters data records to only include those with datetimes less than or equal to the supplied datetime. Accepts an ISO 8601 date or date and time.
    Subscription CodeStringThe code used to identify the subscription. This string may contain the following characters:
    a-z 0-9 @-_.
    couponsBegin TimeDatetimeOperates on the attribute specified by the Sort property. Filters data records to only include those with datetimes greater than or equal to the supplied datetime. Accepts an ISO 8601 date or date and time.
    End TimeDatetimeOperates on the attributes specified by the Sort property. Filters data records to only include those with datetimes less than or equal to the supplied datetime. Accepts an ISO 8601 date or date and time.
    StateSelectSelect the state of the accounts. Select one of "expired", "maxed_out", or "redeemable".
    invoicesBegin TimeDatetimeOperates on the attribute specified by the Sort property. Filters data records to only include those with datetimes greater than or equal to the supplied datetime. Accepts an ISO 8601 date or date and time.
    End TimeDatetimeOperates on the attributes specified by the Sort property. Filters data records to only include those with datetimes less than or equal to the supplied datetime. Accepts an ISO 8601 date or date and time.
    StateSelectSelect the state of the accounts. Select one of "closed", "failed", "open", "paid", "past_due", "pending", "processing", or "voided".
    plan_add_onsBegin TimeDatetimeOperates on the attribute specified by the Sort property. Filters data records to only include those with datetimes greater than or equal to the supplied datetime. Accepts an ISO 8601 date or date and time.
    End TimeDatetimeOperates on the attributes specified by the Sort property. Filters data records to only include those with datetimes less than or equal to the supplied datetime. Accepts an ISO 8601 date or date and time.
    Plan CodeStringThe unique code used to identify the specific plan.
    StateSelectSelect the state of the accounts. Select one of "active", "canceled", "expired", "future", "in_trial", "live", or "past_due".
    plansBegin TimeDatetimeOperates on the attribute specified by the Sort property. Filters data records to only include those with datetimes greater than or equal to the supplied datetime. Accepts an ISO 8601 date or date and time.
    End TimeDatetimeOperates on the attributes specified by the Sort property. Filters data records to only include those with datetimes less than or equal to the supplied datetime. Accepts an ISO 8601 date or date and time.
    subscriptionsBegin TimeDatetimeOperates on the attribute specified by the Sort property. Filters data records to only include those with datetimes greater than or equal to the supplied datetime. Accepts an ISO 8601 date or date and time.
    End TimeDatetimeOperates on the attributes specified by the Sort property. Filters data records to only include those with datetimes less than or equal to the supplied datetime. Accepts an ISO 8601 date or date and time.
    StateSelectSelect the state of the accounts. Select one of "active", "canceled", "expired", "future", "in_trial", "live", or "past_due".
    subscriptions_for_accountBegin TimeDatetimeOperates on the attribute specified by the Sort property. Filters data records to only include those with datetimes greater than or equal to the supplied datetime. Accepts an ISO 8601 date or date and time.
    End TimeDatetimeOperates on the attributes specified by the Sort property. Filters data records to only include those with datetimes less than or equal to the supplied datetime. Accepts an ISO 8601 date or date and time.
    Account CodeStringA unique identifier used by Matillion ETL to identify the Recurly account. This string may contain the following characters:
    a-z 0-9 @-_.

    For more information, please refer to the Recurly Documentation.
    StateSelectSelect the state of the accounts. Select one of "active", "canceled", "expired", "future", "in_trial", "live", or "past_due".
    transactionsBegin TimeDatetimeOperates on the attribute specified by the Sort property. Filters data records to only include those with datetimes greater than or equal to the supplied datetime. Accepts an ISO 8601 date or date and time.
    End TimeDatetimeOperates on the attributes specified by the Sort property. Filters data records to only include those with datetimes less than or equal to the supplied datetime. Accepts an ISO 8601 date or date and time.
    StateSelectSelect the state of the accounts. Select one of "failed", "successful", or "voided".
    TypeSelectSelect the transaction type. Select one of "authorization", "purchase", or "refund"
    transaction_for_accountBegin TimeDatetimeOperates on the attribute specified by the Sort property. Filters data records to only include those with datetimes greater than or equal to the supplied datetime. Accepts an ISO 8601 date or date and time.
    End TimeDatetimeOperates on the attributes specified by the Sort property. Filters data records to only include those with datetimes less than or equal to the supplied datetime. Accepts an ISO 8601 date or date and time.
    Account CodeStringA unique identifier used by Matillion ETL to identify the Recurly account. This string may contain the following characters:
    a-z 0-9 @-_.

    For more information, please refer to the Recurly Documentation.
    StateSelectSelect the state of the accounts. Select one of "chargeback", "failed", "successful", or "voided".