S3 Manifest Builder
    • Dark
      Light

    S3 Manifest Builder

    • Dark
      Light

    Article Summary

    This article is specific to the following platforms - Redshift.

    S3 Manifest Builder

    Build and upload a manifest file which describes a set of files (S3 objects) to load based upon a pattern using a regular expression.

    If the pattern is simply a prefix, this is not required as the S3 Load can use a prefix directly. However, for more complicated file-naming schemes, use this component to build a manifest file first, and then use an S3 Load with the manifest option and a path pointing to the generated manifest file.


    Properties

    PropertySettingDescription
    NameTextA human-readable name for the component.
    Source S3 PathS3 TreeThe S3 bucket (and optionally the path within it) to search.
    Manifest PathS3 TreeThe S3 location to write the manifest file to.
    Manifest File nameTextThe file name for the manifest.
    Filter Regular ExpressionRegular ExpressionThis is a regular expression. All names found from Source S3 Path are tested against this expression, and if they match they are included in the manifest.
    This requires an exact match, so if you are searching for text anywhere within the name you will need to include wildcards before and after the text.
    MandatorySelectYes writes the Mandatory option into the manifest, meaning the subsequent S3 load will fail if the file is not present. No makes the file optional, and no error will be raised if the file is missing.

    Variable Exports

    This component makes the following values available to export into variables:

    SourceDescription
    Matched ObjectsThe number of objects matching the filter pattern and written to the Manifest.

    What's Next