RDS Bulk Output Component
RDS Bulk OutputNote: This feature is only available for instances hosted on AWS.
Load the contents of a table (or view) into a table in an Amazon RDS database.
|Name||Text||The descriptive name for the component.|
aurora: Amazon Aurora (see the AWS Documentation).
mysql: MySQL (see the AWS Documentation).
mariadb: MariaDB (see the AWS Documentation).
postgresql: PostgreSQL (see the AWS Documentation).
|RDS Endpoint||Text||This is the RDS Database Endpoint. If the IAM role attached to the instance (or the manually entered credentials associated with the current environment) has the permissions granted to query the RDS endpoints, you may select the RDS endpoint from a list. Otherwise, you must enter it manually - it can be found in the RDS console and is a long dotted-name and port number, separated by a colon.|
|Database Name||Text||This is the name of the database within your RDS instance.|
|Username||Text||This is your RDS connection username.|
|Password||Text||This is your RDS connection password. The password is masked so it can be set, but not read.Users have the option to store their password inside the component but we highly recommend using the Password Manager option.|
|JDBC Options||Parameter||A JDBC parameter supported by the Database Driver. The available parameters
are determined automatically from the driver, and may change from
version to version.
They are usually not required as sensible defaults are assumed.
|Value||The parameter value.|
|Database||Text||The name of the database you wish to source data from.|
|Schema||Select||Select the table schema. The special value, [Environment Default] will use the schema defined in the environment. For more information on using multiple schemas, see this article.|
|Source Table||Select||The table (or view) on your cluster to copy to RDS.|
|Target Table||Text||Provide a new table name.|
|Load Columns||Select||Choose the columns to load into RDS. This parameter can be empty, which means that all columns will be loaded.|
None - assume the RDS database already has the table defined with the correct structure
Create if not exists - Only create the table if it doesn't already exist
Replace - Always drop and re-create the table. Use with care!
|Primary Key||Select||Specify a set of columns to be used as the primary key for the target table. This is optional, but if specified allows you to UPSERT existing data in the target table.|
Replace - Existing rows with the same Primary Key values will be replaced.
Ignore - Existing rows with the same Primary Key values will be ignored.
|Truncate Target Table||Select||Whether or not to truncate the target table before loading data.|
|On Warnings||Select||Choose whether to Continue with the load if an error is raised or to Fail the run.|
|Additional Copy Options||Text||Any additional options that you want to apply to the copy. Some of these may
conflict with the options the component already sets - in particular care is taken to
escape the data to ensure it loads into the target database even if the data
contains row and/or column delimiters, so you should never override the escape or
Options are documented here: MySQL, PostgreSQL MariaDB Aurora DB
|Batch Size||Integer||This is optional, and specifies the number of rows to load to the target between each COMMIT. On a very large export, this may be desirable to keep the size of the Amazon RDS log files from growing very large before the data is committed.|
A select query is issued against the source table. The output is formatted in an appropriate way to load into the target database, and data is streamed in - no temporary files are used and the load into the target table begins almost immediately.
In this example we load the airports data into a MySQL RDS table called "rds_airports".
We allow the component to create the target table (if it doesn't already exist), and then load the data into it in 1,000 row chunks.