Handling secrets in modules and plugins

Added in version 2.22.

ansible-core 2.22 adds a secret masking API that modules and plugins can use to register sensitive values. Any registered value is replaced with $REDACTED$ wherever Ansible writes output, without changing the value itself. This page describes the API, how registered secrets move between the controller, its worker processes, and the managed node, and what you must do in callback plugins and plugin configuration to take part in masking.

For the user-facing behavior, including which values Ansible registers automatically and how registered secrets are matched, see Masking secrets in Ansible output.

How masking works 

Ansible keeps a process-wide registry of secret strings. Every place that Ansible writes output, referred to as an egress boundary, runs the text through the registry and replaces any registered secret with the placeholder. The boundaries are:

The Display object on the controller, which covers the screen, the log_path log file, warnings, deprecation messages, errors, and tracebacks.
The result mapping of every task result handed to a callback plugin.
AnsibleModule.log() in Python modules and the Ansible.Basic logging in PowerShell modules, which write to syslog and the Windows Event Log on the managed node.

Registration is append-only and non-destructive. Registering a value does not change it, and code that already holds the value keeps working with it. Secrets are never removed from the registry for the life of the process.

Warning

Secret masking is best effort and does not guarantee that a secret never reaches output. Modules and plugins must still avoid writing sensitive data where they can, and use no_log where a whole result is sensitive. The public API on this page is stable, but the matching algorithm and the other behaviors described here are implementation details that may change in future releases.

Note

The log_path log file is a handler on the Python root logger. Only messages that Ansible writes through Display are masked before they reach it. Messages that a plugin, or a library it imports, emits with the standard logging module are written to the file unmasked. Pass such messages through mask_secrets() before logging them, or avoid logging sensitive data with logging at all.

Secrets registered in a worker process, such as an action, connection, lookup, or filter plugin running inside a forked worker, are sent back to the controller together with the worker’s results and registered there. Secrets registered inside a module on the managed node are returned to the controller as part of the module result and registered there as well. In both cases the secret is masked in every later task, not only in the task that registered it.

In the other direction, the controller passes any registered secrets that appear in a module’s arguments to the module, so that the module-side masking in AnsibleModule.log() and Ansible.Basic also covers them. Secrets that are not present in the module arguments are not sent to the managed node. A module that obtains a sensitive value another way, such as reading it from a file or receiving it from an API, must register the value itself for module-side masking to cover it, even if the controller already knows that value.

Secrets that a module registers during its run are returned inside the module’s raw JSON result and are only registered on the controller when that result is processed. Connection plugin authors must audit any code that displays the raw standard output or standard error of a module, since that text contains the new secrets in plain text before the controller knows about them. The connection plugins shipped with ansible-core only display raw module output when ANSIBLE_DEBUG is enabled. Gate such output the same way, or do not display it at all.

Propagation from a worker to the controller happens when the worker sends its results, and only workers forked after that point inherit the new secret. Workers that are already running, such as those executing the same task for other hosts, do not see it. If a plugin needs a value masked across every host in the current task, register it on the controller before the task runs, for example in a vars plugin or through the register_secret filter in an earlier task.

Note

Masking works by searching output for the exact registered string. A registered secret is masked wherever it appears, including inside a longer word or value, so avoid registering very short or common values. See How registered secrets are matched for the rules that apply to short, overlapping, and adjacent secrets. If you know a secret can also appear in output in another shape or format, register that form as well so it is redacted too. A base64 encoded or URL encoded copy of a registered secret is a different string and is not masked on its own. The JSON-escaped form of a secret is the one exception. Because most Ansible output and module data is JSON, the registry derives the escaped forms of each secret itself, with and without \uXXXX escapes for non-ASCII characters, and masks those as well.

The `ansible.module_utils.secrets` API 

The ansible.module_utils.secrets module is the public API for working with secrets. It is available to modules, module utilities, and all controller-side plugins.

from ansible.module_utils.secrets import register_secret, register_secrets, mask_secrets

register_secret(value): Register a single string as a secret and return it unchanged. The return value makes it convenient to wrap an expression in place, for example token = register_secret(response['token']).
register_secrets(values): Register every string in an iterable. Use this when you have several values to register at once.
mask_secrets(value, *, mask_placeholder='$REDACTED$'): Return a copy of value with every registered secret replaced by mask_placeholder. Use this when you write text to a destination that is not an Ansible egress boundary, such as a file, an external logging system, or a message queue.

All three functions expect Python str values. Convert bytes with to_text() before registering them. Leading and trailing spaces, tabs, carriage returns, and newlines are stripped from a value before it is registered, so there is no need to strip a value read from a file or a subprocess yourself. There is no minimum or maximum secret length but values that are empty after stripping are silently ignored by register_secret() and register_secrets(). In both cases register_secret() still returns the value exactly as it was given.

Registering secrets in a module 

Values for options declared with no_log: True in the argument_spec are registered automatically when AnsibleModule validates the arguments. Register any other sensitive value that your module discovers at run time, such as a token returned by an API or a password read from a file on the managed node, as soon as you have it:

from ansible.module_utils.basic import AnsibleModule
from ansible.module_utils.secrets import register_secret


def main():
    module = AnsibleModule(
        argument_spec=dict(
            username=dict(type='str', required=True),
            password=dict(type='str', required=True, no_log=True),  # registered automatically
        ),
    )

    session = login(module.params['username'], module.params['password'])

    # The token is a new secret discovered by the module. Register it so it is masked
    # in the module log, the callback output, and any later task on the controller.
    token = register_secret(session['token'])

    # The real value is still returned so it can be used by later tasks. It is only
    # masked when written to output.
    module.exit_json(changed=False, token=token)


if __name__ == '__main__':
    main()

There is no need to strip the value from the result. The result keeps the real value, and the controller masks it when a callback or Display writes it out.

Note

On ansible-core versions before 2.22, no_log values are removed from the module result and replaced with VALUE_SPECIFIED_IN_NO_LOG_PARAMETER, and are stripped from module logs. The secrets API is not available on those versions.

The AnsibleModule.log() method masks its message and the log_args values before writing to syslog. The module invocation log entry always replaces no_log options with $REDACTED$ , regardless of the value’s length, and masks any other registered secret it contains.

Modules that respawn under a different Python interpreter keep the secrets that were passed in from the controller, and any secret registered in the respawned process is returned to the controller as normal.

Registering secrets in a plugin 

The same functions work in every controller-side plugin type. This example is a lookup plugin that reads a credential from an external service and registers it before returning it:

from ansible.module_utils.secrets import register_secret
from ansible.plugins.lookup import LookupBase
from ansible.utils.display import Display

display = Display()


class LookupModule(LookupBase):

    def run(self, terms, variables=None, **kwargs):
        self.set_options(var_options=variables, direct=kwargs)

        results = []
        for term in terms:
            value = fetch_credential(term)
            results.append(register_secret(value))

            # Display output is a masked boundary, so this prints the placeholder.
            display.vvv(f"fetched credential for {term}: {value}")

        return results

For plugin options that are always sensitive, prefer marking the option with secret: true in the plugin documentation instead of registering the value by hand. See Marking plugin configuration options as secret.

Masking output you write yourself 

Anything you write outside of an egress boundary is not masked. Call mask_secrets() before writing:

from ansible.module_utils.secrets import mask_secrets

with open('/var/log/my_plugin.log', 'a') as fd:
    fd.write(mask_secrets(f"request completed: {response_text}\n"))

Deprecated helpers 

The following helpers are replaced by automatic masking and are deprecated:

ansible.module_utils.basic.heuristic_log_sanitize() is deprecated and will be removed in ansible-core 2.25.
ansible.module_utils.common.parameters.remove_values() and sanitize_keys() are deprecated and will be removed in ansible-core 2.25.
The live argument of ansible.utils.cmd_functions.run_cmd() is deprecated and will be removed in ansible-core 2.25.

Modules that called remove_values() or sanitize_keys() on their results, or that strip no_log values from a result before returning it, no longer need to. Remove those calls and return the real values.

The live argument of run_cmd() writes subprocess output directly to standard output, which bypasses masking. Run the subprocess yourself if you need to stream its output, and mask it with mask_secrets().

The heuristics that these helpers applied are gone as well. AnsibleModule.log() no longer rewrites password-like text such as user:password@host in URLs, and run_command() no longer replaces password-like arguments such as --password=... with ******** in the cmd value of a failure result. Only registered secrets are masked. If your module passes a secret on the command line or embeds one in a URL and that secret is not a no_log option, register it with register_secret() before use.

PowerShell and C# modules 

The Ansible.Secrets C# module utility provides the same API for Windows modules. Import it with #AnsibleRequires and use the static Ansible.Secrets.SecretMasker class:

#!powershell

#AnsibleRequires -CSharpUtil Ansible.Basic
#AnsibleRequires -CSharpUtil Ansible.Secrets

$spec = @{
    options = @{
        username = @{ type = 'str'; required = $true }
        password = @{ type = 'str'; required = $true; no_log = $true }  # registered automatically
    }
}
$module = [Ansible.Basic.AnsibleModule]::Create($args, $spec)

$session = Connect-MyService -Username $module.Params.username -Password $module.Params.password

# Register the discovered token so it is masked in the Event Log and on the controller.
[Ansible.Secrets.SecretMasker]::RegisterSecret($session.Token)

# Mask a string yourself before writing it somewhere that is not an Ansible boundary.
$summary = [Ansible.Secrets.SecretMasker]::MaskString("Connected with token $($session.Token)")
Set-Content -Path C:\logs\my_module.log -Value $summary

$module.Result.token = $session.Token
$module.ExitJson()

The SecretMasker class exposes:

RegisterSecret(string secret) and RegisterSecret(SecureString secret): Register a value. Prefer the SecureString overload when you already hold the value as a SecureString, so the plain text is not exposed to PowerShell AMSI notifications.
MaskString(string value) and MaskString(string value, string maskPlaceholder): Return a copy of value with registered secrets replaced by $REDACTED$ or the placeholder you supply.

The matching rules match the Python implementation. Ansible.Basic registers no_log option values automatically, masks messages written through its event logging, and returns any newly registered secrets to the controller when the module exits.

Marking plugin configuration options as secret 

Plugin configuration options can be marked with secret: true. When a marked option is resolved, its value is registered as a secret regardless of the source that set it: an environment variable, an ansible.cfg entry, an Ansible variable, a command line option, or a direct argument to the plugin.

options:
  api_token:
    description: Token used to authenticate to the service.
    type: str
    secret: true
    env:
      - name: MYCOLLECTION_API_TOKEN
    vars:
      - name: mycollection_api_token

secret is only supported for options of type str, string, and list. For a list option, each string element is registered and non-string elements are ignored. Declaring secret: true on an option of any other type is an error when the plugin’s configuration is loaded, so the plugin fails to load rather than silently leaving the value unmasked.

secret is not supported on suboptions. Sub-option definitions are documentation only and are not processed by the configuration manager, so there is no point at which their values could be registered. If a sub-option holds a sensitive value, register it in the plugin with register_secret() after reading the option.

The ansible-core connection and become plugins use this keyword for their password, private key, and passphrase options. When you write a plugin that accepts a password or token, mark the option as secret instead of relying on callers to add no_log or use register_secret.

Callback plugins 

The result mapping of every task result handed to a callback plugin is masked before the plugin sees it. Ansible walks CallbackTaskResult.result and replaces registered secrets at any depth, including inside loop results, so a callback never receives an unmasked secret value in the result. The walk covers:

Strings. Every str value and every str dictionary key is passed through mask_secrets(). When two keys mask to the same placeholder, a numeric suffix is appended to keep both entries, for example $REDACTED$ and $REDACTED$ (2).
Numbers. An int or float whose text form is a registered secret is replaced with the placeholder string, so {"pin": 1234567} becomes {"pin": "$REDACTED$"} and the result stays valid when serialized. A callback that expects a number in a result must handle receiving a string instead.
Command output lines. The stdout_lines and stderr_lines values are rebuilt from the masked stdout and stderr text, so a secret that spans several lines is masked in the lines as well. This only happens when the lines value still matches the split text. A module that sets stdout_lines itself has each line masked on its own like any other list.

Booleans, bytes, and None are returned unchanged. Only result.result is masked. Anything else the callback can reach is not:

The task and play names, and the task, host, warnings, deprecations, and exception attributes of the result.
The statistics passed to v2_playbook_on_stats() and the arguments of any other callback method.
Data the callback builds by combining or deriving values, for example a value it decodes, hashes, or re-encodes after receiving the result.

Output sent through Display() is masked automatically wherever the data came from. Output written anywhere else, such as a file, a socket, an HTTP request, or a database, should be passed through mask_secrets() first so that data from outside result.result is masked as well. A callback that only writes through Display() needs no changes. The junit and tree callbacks shipped with ansible-core write to files and call mask_secrets() on everything they write, which also covers the task and play names in their output:

import json

from ansible.module_utils.secrets import mask_secrets
from ansible.plugins.callback import CallbackBase


class CallbackModule(CallbackBase):
    CALLBACK_VERSION = 2.0
    CALLBACK_TYPE = 'notification'
    CALLBACK_NAME = 'namespace.collection_name.json_file'
    CALLBACK_NEEDS_ENABLED = True

    def v2_runner_on_ok(self, result):
        # result.result is already masked, but the task name is not, so mask the
        # serialized form before writing it.
        entry = {'task': result.task_name, 'host': result.host.name, 'result': result.result}
        entry_json = mask_secrets(json.dumps(entry, default=str))

        with open('/var/log/ansible-results.jsonl', 'a') as fd:
            fd.write(entry_json + '\n')

The mask_secrets() function masks a secret in its literal form and in its JSON-escaped form, so a callback can serialize data with json.dumps() first and mask the resulting text, as the example above does. A secret that contains a double quote, a backslash, a control character, or any other character that is written differently when encoded as JSON is still recognized as a secret:

import json

from ansible.module_utils.secrets import mask_secrets, register_secret

register_secret('pa"ss\\wörd!')

# The literal value is masked.
mask_secrets('pa"ss\\wörd!')      # $REDACTED$

# The JSON-escaped form is masked as well.
result = {'password': 'pa"ss\\wörd!'}
result_json = json.dumps(result)  # {"password": "pa\"ss\\w\u00f6rd!"}
mask_secrets(result_json)         # {"password": "$REDACTED$"}

Masking the serialized text is not always enough, or not what you want, for data that does not come from result.result. Mask each value before serializing it instead when:

The output must stay valid. A non-string value whose text form is a registered secret is replaced inside the serialized text. If 12345678 is registered, {"key": 12345678} becomes {"key": $REDACTED$}, which is not valid JSON.
The serializer escapes differently from JSON. XML writes My<Secret> as My<Secret>, and repr(), YAML, and other formats have their own escaping rules. Only the literal and JSON-escaped forms are registered, so a secret containing an escaped character is not found in that text.
The serializer transforms values. Base64, URL encoding, compression, or any other transform produces text that does not contain the registered form.
Only some fields are secret. Masking the whole text can also replace the same string where it is not sensitive, such as a key name or an unrelated field that happens to hold the same value.

All of the limitations of secret masking described in the user guide apply to the masked result and to any output a callback masks itself, so read them before designing how your callback handles results.