We have a large number of tables to connect to and are using environmental variables as much as possible to reduce the amount of work needed for config changes, however for the password to a database if we use an environmental variable the value is in plain text.
Would it be possible to add a new type to the environmental variables that doesn't show the value ie password field type?
14 Community Answers
Ed Thompson —
We hope to introduce a password manager which will allow you to manage passwords across a project, and then reference the password by name in the components. At that point, using a variable for the password’s name will be fine, and its value will be retrieved when the job runs.
Because of this plan, it is unlikely we will introduce a password-style variable type.
I’ll post again once we know when it will be available.
For Bash and Python scripts that’s the case, yes: you will need to embed them in the script, either as a string literal or using a Matillion Environment Variable. It’s better to use an environment variable because then it can automatically have different values in dev/test/production etc.
I’ll update this thread once additional options have become available.
We have a similar problem to this, we spoke on a demo last week so you may have answered us already although now when implementing it we are coming across a blocker.
Basically we have a number of users that will have a job run for each of them where we can pass in a variable and run a job but then need to dynamically extract a password for sftp upload via the password manager. We can do this all the way up to the last stage but we have to set the password hardcoded in the options for the sftp put object. The sftp is for each user therefore don't want to have a separate job for each user? We also want to be careful about where we store the passwords for these users.
The Matillion development work EMD-3076 on password management has not yet been completed, but it won’t include the possibility to have passwords vary by “external” (i.e. non-Matillion) user. The reason being that your list of users is actual data rather than metadata.
To avoid hardcoding, and keep the passwords secure, the solution is to implement some kind of data-driven password management protocol yourself. Examples would be:
Include the password in a data table containing your list of users, and use the Table Iterator
Same but encrypt or obfuscate the password
Write a “getPassword” operating system utility that behaves differently in the Live and Development environments, and which requires a “username” parameter
Similar but implemented as a REST web service, and maybe invoke via Python or an API Query
You would need to choose a technique that matches your needs both in terms of convenience and security.
We have reviewed the possibility of reading the passwords in bash and Jython however we have decided not to build this because it would lead to a potential security flaw as it would allow anyone to access any password in plain text from the password manager.
However we do have a possible workaround for you. My colleague has written a script which will uses KMS in jython to encrypt a password. If this is something you’re interested in, can you send me your email and I will share it with you?