Skip to content

Regular Expressions in Shield

Wherever Shield asks you for a regular expression — a Custom Data Type definition, a Regex Application definition, a Mask Capture Regex, a JSON Search DSL Regex node, or a SAML allowed-URL pattern — that pattern is compiled with Go's regexp package, which implements the RE2 syntax. RE2 is fast and linear-time (it can't catastrophically backtrack), but in exchange it omits a few features you may know from PCRE, Perl, JavaScript, or Python regex.

This page is the single reference for that flavor. The field-level pages link here instead of repeating the rules.

TL;DR

  • Shield regex = Go regexp (RE2).
  • No look-ahead ((?=…), (?!…)), no look-behind ((?<=…), (?<!…)), and no back-references (\1, \k<name>). RE2 does not support them — a pattern that uses them will fail to compile and the field won't match anything.
  • Use inline flags at the start of the pattern: (?i) case-insensitive, (?s) dot matches newline, (?m) multi-line ^/$.
  • Capture groups ( … ) are supported, including named groups (?P<name>…). Several fields (RegexSubgroup, valueGroupIndex, Value Group Index) use a capture group to select the part of the match to obfuscate.
  • In JSON fields (the Search DSL, API payloads), backslashes are doubled: write \\d, \\w, \\..

What works

RE2 covers the everyday regex you need to describe sensitive data:

Feature Syntax Example
Character classes [A-Z], \d, \w, \s, . \d{3}-\d{2}-\d{4} (SSN)
Quantifiers * + ? {n} {n,} {n,m} [A-Z]{2}\d{6}
Anchors ^ $ \b \B \A \z ^https://api\.
Alternation a|b (?i)ssn|social
Grouping & capture (…), named (?P<name>…) Bearer\s+([\w.-]+)
Non-capturing group (?:…) (?:Mr|Ms)\.\s
Inline flags (?i) (?s) (?m) (?U) (?i)^c.+

Anchoring

Most Shield regex matches are contains-style (the pattern can match anywhere in the input). Anchor with ^…$ when you need the match to span the whole value — for example a Regex Application definition is tested against the entire URL, so ^https://api\..*$ pins it from the start.

What is not supported

These constructs exist in PCRE / Perl / JavaScript / Python but not in RE2. Using them is the single most common reason a pattern "doesn't work":

Not supported Looks like Do this instead
Look-ahead foo(?=bar), foo(?!bar) Match the trailing context explicitly, then capture the part you want with a group + RegexSubgroup/valueGroupIndex.
Look-behind (?<=foo)bar, (?<!foo)bar Match the leading context (e.g. foo(bar)) and select group 1.
Back-references (\w+)\s+\1 Not expressible in RE2 — restructure the rule, or detect the repeated value some other way.
Possessive / atomic a++, (?>…) Not needed — RE2 is already linear-time; use a normal quantifier.
Recursion / subroutines (?R), (?1) Not available; flatten the pattern.

Invalid patterns silently match nothing

A pattern that uses an unsupported feature fails to compile. Depending on the field, that means the definition is rejected on save or simply never matches at runtime — so a rule that "isn't catching anything" is often an unsupported-syntax problem, not a logic one. Test the pattern (see below) before relying on it.

Capturing only part of a match

Because there's no look-around, the RE2 way to "match in context but obfuscate only part" is a capture group plus a group-index field:

Bearer\s+([A-Za-z0-9._-]+)

With the group selector set to 1, Shield detects the token only and leaves the literal Bearer prefix untouched. The selector field is called:

The same 0 = whole match, 1 = first group convention applies everywhere.

Case sensitivity

Regex matching is case-sensitive by default. Add the (?i) flag at the start of the pattern to make it case-insensitive:

(?i)api.?key

(Note the asymmetry in the JSON Search DSL: a Value exact-match is case-insensitive, but a Regex match is case-sensitive unless you add (?i).)

Reference

The authoritative syntax for the engine Shield uses:

When in doubt about whether a construct is allowed, check the RE2 wiki — anything not listed there (notably look-around and back-references) is not available in Shield.

Testing your patterns

  • regex101.com is handy for building patterns, but select the Golang flavor — the default PCRE flavor will happily accept look-around and back-references that Shield rejects.
  • For an exact match to Shield's behavior, test against the Data Scanning API with representative content, or iterate in the Admin Console Test Page.
  • Remember the JSON double-escaping (\\d) when pasting a pattern into a JSON field or API body — but not when typing it into a plain Admin Console text field, where \d is correct.