Server Migration Tool

The server migration tool allows users to push all assets from current server to new server. This tool runs as a shell script when placed on the server you wish to move. Through the tool, users are able to filter what is migrated, although everything will be migrated by default.

This can be useful in a number of cases:

  • Setting up a new live environment by pushing all assets from a staging environment
  • Setting up a test environment by pushing a single project from live to a test environment to diagnose
  • Migrating a Single Node Matillion Setup to a Cluster by provisioning a new cluster using Cloudformation and then pushing everything to it from the existing environment.

To use this tool, simply copy the scripts to your server (the Matillion ETL instance), SSH onto the server and run the python script with the target hostname (including port if required) as an argument. It may be necessary to update your python installation before running the tool.

It is also required that API permissions are granted to the user for both source and target server.

The tool comes with help that can be accessed through the --help argument. A list of optional arguments is given below.

 
Argument Description
--source-server SOURCE_SERVER Source Server Hostname (Default: http://127.0.0.1)
--skip-drivers Skip JDBC Driver migration (true/false)
--skip-profiles Skip API Profile migration (true/false)
--skip-projects Skip Matillion Project migration (true/false)
--skip-schedules Skip Schedule migration (otherwise schedules are migrated but disabled) (true/false)
--skip-sqssetup Skip SQS Listener setup (even when included queues are initially disabled) (true/false)
--skip-oauth Skip OAuth Token migration (true/false)
--allow-deletes Allow deleting same-named resources. Projects are deleted and recreated. (true/false)
--project-filter PROJECT_FILTER Regular Expression to match against Group- name/Project-name/Version-name; default .* (migrate everything)

Their usage is as below:

migrate_server.py [-h] [--source-server SOURCE_SERVER] [--skip-drivers][--skip-profiles] [--skip-projects][--skip-schedules] [--skip-sqssetup] [--skip-oauth][--allow-deletes] [--project-filter PROJECT_FILTER] target_server


Example

Below is a generic example for running the migration tool from the EC2 Console with the necessary updates that may be required to run to script. Simply download migrate.py to /tmp and use the following 3 commands (replacing the source and target server IP with your own).

Below is a generic example for running the migration tool from an SSH session with the necessary updates that may be required to run to script. Simply download migrate.py to /tmp and use the following 3 commands (replacing the source and target server IP with your own).

Note: If running the Migration Tool from the source or the target server then use your private-ip or 127.0.0.1:8080 to identify localhost instead of using the IP address. Servers may not recognise their public ip and may throw an error.

  • sudo yum install python34-pip.noarch
  • sudo pip-3.4 install requests
  • sudo /tmp/migrate_server.py --allow-deletes --source-server http://source-server-ip http://target-server-ip
  • You will be asked to provide source credentials (ec2-user/instance-id) and target credentials (ec2-user/instance-id). The script will then proceed to migrate all available assets.

Note: Schedules on the new server will be disabled by default.

  • Disable SQS integration on the old server and shutdown the server.
     
  • On the new server, enable SQS Integration and Enable all schedules.