Administering Matillion ETL

Overview

From version 1.21.5 onwards, Matillion ETL offers an administration page which automates many server administration tasks that used to require editing configuration files by hand.

The /admin url will present a login page (similar to the login for Matillion) - the default user is ec2-user and the default password is set to the instance-id on first launch.

It is possible to perform basic administration tasks on Matillion ETL instances if you have admin permissions to that instance. GCP offers a simple SSH command line tied to each particular instance that can be found through the main GCP menu under 'Compute Engine'. Clicking the SSH button by the desired instance will launch the SSH command line window.

 

SSL

Matillion can listen for HTTP and/or HTTPS requests, depending on the configuration. By default, Matillion listens only for HTTP requests (port 80).

This can be changed to HTTPS (port 443) or both by selecting the appropriate button. If SSL is enabled, you may optionally upload SSL certificates to allow clients to validate the identity of the server.

Save Configuration saves any changes in this section but does not immediately apply it.

You may provide a certificate and key file using the Upload Certificates button once you have chosen the appropriate files. This is optional, but clients using HTTPS will get a certificate validation error until you provide valid certificates.

Finally, for the changes to take effect, you must restart the server - this will disconnect any connected client sessions and abort any currently running jobs.

In order to Enable and Disable SSL, please refer to the documentation on Securing Matillion ETL. Note that SSL is enabled by default with Matillion ETL.

 

Managing Users

There are 3 different login schemes.

NONE disables all security and lets anyone use the system. (This is not an option for the admin page itself, that always requires authentication).

INTERNAL uses an internal database of username and passwords.

You can add remove and modify users in the Security Configuration section.

To add a user:

  • In the Security Configuration -> Add User section enter a Username and click Add User
  • Enter a password for the user. If you wish the user to also be able to access the admin page tick the admin checkbox.

To change the user's password click the change password link

To Remove a user click the X

EXTERNAL is used for linking to an existing directory server (e.g. OpenLDAP or Microsoft Active Directory).

The sample values below are for an Active Directory server running for the realm EXAMPLE.COM.

Parameter Description
Connection Name The name of a user to make the initial bind to the directory.

For active directory, that will include a realm using the form "user@REALM"

exampleuser@EXAMPLE.COM
Connection Password The password for the user to make the initial bind to the directory.
Connection URL

The location of the directory server, using one of the forms below:

For non SSL - ldap://<hostname>:389
For SSL - ldaps://<hostname>:636
User Base The part of the directory tree to begin searching for users.
cn=Users,dc=example,dc=com
User Search The attribute to search for user names.
sAMAccountName={0}
Role Base The part of the directory tree to begin searching for groups/roles - often the same place as users.
cn=Users,dc=example,dc=com
Role Name The name of the attribute containing the role name.
cn
Role Search

How to find all the roles for a user.

member={0}
METL Role Name The role a user must be a member of to gain access to the Matillion ETL application.
METL Admin Role Name The role a user must be a member of to gain access to the Matillion ETL administration page - this can be different to the METL Role Name but is not required to be.


Once the configuration is Saved, you will need to restart the server to take effect - use the Restart Server button on the top-right of the screen.

Inside the SSH command prompt, browse to the tomcat directory using a command such as:

cd /etc/tomcat

Then open the file containing the user information using a command such as:

sudo vi tomcat-users.xml

Using your text editor of choice, browse to the bottom of the file and, between the admin's credentials and the closing tag, add a line for the new user, completing the username and password fields to your own liking.

<user username="<username>" password="<password"> roles="Emerald,API,Admin" />

Save and close the file.

 

Software Updates

You may check for software updates using the Check for Updates button.
 

If any updates are found, the Update button will become available:

This will download any updated packages and apply them - once applied the server will be restarted, which will disconnect any users and abort any running tasks.

Using the SSH command line, enter the below command to instigate an update:

sudo yum upgrade

This will attempt to update both the OS and Matillion ETL instance. Alternatively, to upgrade only Matillion ETL, use the below command:

sudo yum upgrade matillion-*

Either command will prompt the user for confirmation (y/n) for the update before continuing.

 

Access Server Logs

Tomcat Catalina server logs of your Matillion ETL instance are available through the 'More' menu at the upper-right of the admin page.

This will download a 'catalina.out' file that contains the servlet's logs in plain text. The download link for a given instance is always:

http://<ServerIP>/admin/FileDownload

Server logs record only for a finite time and size so it is likely your download will contain only recent history. For this reason, if an error is encountered, it is recommended that users attempt to reproduce the error then immediately download the log file.

Using the SSH command line, browse to the tomcat server log directory using a command such as below:

sudo ls -al /var/log/tomcat

Then use your preferred text editor to open the file for viewing:

sudo less /var/log/tomcat/catalina.2017-05-02.log

Use tail to view last few logged incidents. This will autorefresh the output can be useful to monitor a currently-running instance.

sudo tail /var/log/tomcat/catalina.2017-05-02.log