> For the complete documentation index, see [llms.txt](https://gdplabs.gitbook.io/glchat/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://gdplabs.gitbook.io/glchat/developer-documentation/pii-aware-retriever.md). # PII Aware Retriever GLChat's **PII Aware Retrieval** is a privacy-preserving retrieval system that anonymizes personally identifiable information in user queries, performs hybrid search combining entity-filtered and semantic vector search, and de-anonymizes results before returning them downstream. It ensures that raw PII never reaches embedding models or third-party LLMs. > **When to use this:** PII Aware Retrieval should only be enabled when your **knowledge base content is masked** (anonymized at storage time with `pii_mapping` in chunk metadata). If your KB stores raw PII text, the de-anonymization step is a no-op and you gain no privacy benefit from the retriever itself — you would only get the semantic search behavior. *** ### Prerequisites Before enabling PII Aware Retrieval, the following must be configured:
PrerequisiteConfig FieldWhereDescription
Enable Anonymize EManonymize_em = truePipeline configAnonymizes the query before it reaches the embedding model and vector search. Without this, raw PII flows into third party embeddings provider.
Enable Anonymize LManonymize_lm = truePipeline configAnonymizes queries and chunks before they reach all the configured language model (including LMRP). Without this, the LM sees raw PII in context.
LMRP PII Safe flagpii_safe = trueLMRP catalog entryMarks self-hosted models as PII-safe so they receive raw (non-anonymized) input. Third-party LMs must NOT be marked pii_safe. This is a safety gate, not an enabler — it controls whether a specific LM bypasses anonymization.
DPO PII Enabledpii_enabled = trueDPO pipeline configEnables PII processing in data ingestion time with NER-based anonymization.
Enable PII Aware Retrievalenable_pii_aware_retrieval = truePipeline configActivates the PII-aware retriever (hybrid search with entity filtering + de-anonymization).
PII Entity Filter Modepii_entity_filter_modePipeline configChooses the matching strategy: "Exact Match", "Contains", "Fuzzy", or "Contains + Fuzzy". Default: "Contains".
#### How the Prerequisites Interact **Privacy Context** holds the runtime PII configuration: | Field | Controls | | -------------- | ----------------------------------------------------- | | `anonymize_em` | EM (embedding model) anonymization on/off | | `anonymize_lm` | LM (language model) anonymization on/off | | `pii_manager` | PIIManager instance (detect, anonymize, de-anonymize) | **Per-LMRP `pii_safe` flag** overrides the LM anonymization decision: | `pii_safe` | Behavior | | ---------- | ----------------------------------------------------------------- | | `true` | Skip anonymization — model is self-hosted in a private VPC | | `false` | Respect `anonymize_lm` — model is third-party, PII must be masked | The example LMRP's model kwargs that have `pii_safe` field: ```json { "pii_safe": true, "response_schema": { "properties": { ... }, ... }, "default_hyperparameters": { ... }, } ``` * `anonymize_em` gates whether the **embedding model** receives masked or plain queries. * `anonymize_lm` gates whether the **language model** receives masked or plain input. * `pii_safe` is a **per-LMRP override** that forces LM anonymization OFF for self-hosted models in a private VPC. It is set via the LMRP catalog entry and patched onto the LMRP instance at runtime. * `enable_pii_aware_retrieval` activates the hybrid entity-filtered + semantic retriever. * `pii_enabled` in DPO enables NER-based PII anonymization for DPO data generation. *** ### How It Works — End-to-End Flow This section walks through a complete retrieval cycle when PII Aware Retrieval is active. #### Step 1: User Query & PII Detection **Input:** `"What did Ashish Vaswani publish at Google Brain about attention?"` PII Manager detects entities: | Detected PII | Type | Token | | ------------------ | ------------------ | ------------ | | `"Ashish Vaswani"` | PERSON | `` | | `"Google Brain"` | ORGANIZATION\_NAME | `` | **Output:** * Anonymized query: `"What did publish at about attention?"` * Mappings: `AnonymizerMapping(pii_value="Ashish Vaswani", anonymized_value="", pii_type="PERSON")`, `AnonymizerMapping(pii_value="Google Brain", anonymized_value="", pii_type="ORGANIZATION_NAME")` #### Step 2: Enrich Retrieval Params The `enrich_retrieval_params_with_pii_map` function prepares the retrieval parameters: 1. **Simplifies the KB filter** — strips the conversation\_id filter so it can be cleanly merged with the entity filter. 2. **Converts mappings to pii\_map** — `{"Ashish Vaswani": "", "Google Brain": ""}`. 3. **Filters mappings to current turn** — Keeps only mappings whose `anonymized_value` token appears in the masked query text. #### Step 3: PII Resolution & Entity Filter Construction Extracts the entity names from `pii_map`: ``` pii_entities = ["Ashish Vaswani", "Google Brain"] ``` Builds the ES Query DSL filter based on the active `pii_entity_filter_mode` (see Entity Filter Modes). #### Step 4: Hybrid Search (Parallel Execution) Both KNN legs run in parallel: | Leg | Filter | Weight | Purpose | | ---------------- | ------------------------- | --------------------------------- | -------------------------------------------- | | **Filtered KNN** | Entity filter + KB filter | `1 - vector_weight` (default 0.2) | Guarantees entity-relevant docs are surfaced | | **Semantic KNN** | KB filter only | `vector_weight` (default 0.8) | Broad topical recall via vector similarity | **Filtered KNN leg:** ```json { "query": { "bool": { "must_not": [{"exists": {"field": "metadata.conversation_id"}}], "filter": [ {"wildcard": {"metadata.entities.keyword": {"value": "*Ashish*"}}} ] } } } ``` * KNN vector search → 5 chunks mentioning "Ashish Vaswani" or "Google Brain" **Semantic KNN leg:** ```json { "query": { "bool": { "must_not": [{"exists": {"field": "metadata.conversation_id"}}] } } } ``` * KNN vector search → 5 semantically similar chunks (no entity guarantee) #### Step 5: Weighted RRF Fusion Both result sets are fused by rank position: ``` score(doc) = Σ weight_i / (rank_i(doc) + k) weights = [0.2, 0.8], k = 60 ``` Documents appearing in both legs get a score boost. #### Step 6: De-anonymization Our resolver replaces tokens in chunk content using `chunk.metadata["pii_map"]`: ``` Before: " published the paper while at ..." After: "Ashish Vaswani published the paper while at Google Brain..." ``` If a chunk has no `pii_map` in its metadata, de-anonymization is a no-op (chunk returned unchanged). #### Step 7: Chunk Entity Normalization (when Anonymize LM is on) When both `anonymize_lm` and `enable_pii_aware_retrieval` are active, retrieved chunks go through **chunk entity normalization** before being re-anonymized for the LM. See Chunk Entity Normalization for details. *** ### Entity Filter Modes The entity filter constrains the **filtered KNN leg** to only documents that mention the queried entity. Four modes are available via the `pii_entity_filter_mode` config field. #### Exact Match Uses Elasticsearch `terms` query on `metadata.entities.keyword`. The document is included only if the entity appears **verbatim** as an array element. ```json { "terms": { "metadata.entities.keyword": ["Ashish Vaswani"] } } ``` | When it works | When it fails | | ---------------------------------------------------- | ----------------------------------------------------------------- | | Query entity and stored entity are identical strings | NER produces different surface forms at index time vs. query time | **Failure example:** Stored entity is `"Ashish Vaswani"`, but query-time NER extracted only `"Ashish"`. The `terms` query looks for an exact array element `"Ashish"` — it won't match `"Ashish Vaswani"`. #### Contains *(default)* Uses Elasticsearch `wildcard` query with `*entity*` pattern on `metadata.entities.keyword`. The document is included if the stored entity **contains** the query entity as a substring, case-insensitively. ```json { "wildcard": { "metadata.entities.keyword": { "value": "*Ashish*", "case_insensitive": true } } } ``` **Why this is the default:** It fixes the most common NER surface-form mismatch, where the user's query mentions a partial name or abbreviation that is a substring of the full name stored in the index. | Query entity | Stored entity | Exact Match | Contains | | -------------- | ------------------------- | ----------- | -------- | | `"Ashish"` | `"Ashish Vaswani"` | ❌ | ✅ | | `"Google"` | `"Google Brain"` | ❌ | ✅ | | `"University"` | `"University of Toronto"` | ❌ | ✅ | | `"OpenAI"` | `"OpenAI"` | ✅ | ✅ | **Limitation for PERSON entities:** Short person names can cause false positives via substring matching. For example, query entity `"John"` would match stored entity `"Johnson & Johnson"` — an unrelated organization. This is why **PERSON entities are excluded from Contains matching in chunk normalization** (see Chunk Entity Normalization — Contains mode). #### Fuzzy Uses Elasticsearch `fuzzy` query with `fuzziness: AUTO`. Matches stored entities within a Levenshtein edit distance of the query entity — useful for typos and minor misspellings. ```json { "fuzzy": { "metadata.entities.keyword": { "value": "Ashish", "fuzziness": "AUTO", "prefix_length": 2, "max_expansions": 50 } } } ``` **Critical limitation:** Fuzzy operates on the **full stored value**, not on substrings. The edit distance is computed between the query entity and the entire stored string. | Query entity | Stored entity | Edit distance | Fuzzy match? | | ---------------- | -------------------------- | ------------- | ------------ | | `"Asish"` (typo) | `"Ashish"` | 1 | ✅ | | `"Ashish"` | `"Ashish Vaswani"` | 11 | ❌ (too far) | | `"Vaswani"` | `"Vaswanis"` | 1 | ✅ | | `"Stanford"` | `"Standford"` (typo in KB) | 1 | ✅ | Fuzzy is best suited as a **complement to Contains** (see Contains + Fuzzy below), not as a standalone replacement. #### Contains + Fuzzy Combines both wildcard and fuzzy clauses in a `bool/should` query per entity. A document is included if it matches **either** condition. ```json { "bool": { "should": [ { "wildcard": { "metadata.entities.keyword": { "value": "*Ashish*", "case_insensitive": true } } }, { "fuzzy": { "metadata.entities.keyword": { "value": "Ashish", "fuzziness": "AUTO", "prefix_length": 2, "max_expansions": 50 } } } ], "minimum_should_match": 1 } } ``` This mode gives the **broadest recall**: it catches both partial-name documents (via wildcard) and typo-variant documents (via fuzzy). #### Mode Comparison Summary | Mode | Strategy | Catches partial names? | Catches typos? | Risk of false positives | Best for | | ------------------------ | --------------------- | ---------------------- | -------------- | ----------------------- | ------------------------------- | | **Exact Match** | `terms` on `.keyword` | ❌ | ❌ | Lowest | Strict matching, clean NER | | **Contains** *(default)* | `wildcard *entity*` | ✅ | ❌ | Medium | Most common case (NER mismatch) | | **Fuzzy** | `fuzzy` with AUTO | ❌ | ✅ | Medium | KB with OCR errors/typos | | **Contains + Fuzzy** | `wildcard + fuzzy` | ✅ | ✅ | High | Maximum recall | *** ### Chunk Entity Normalization #### The Problem: Anonymize LM Token Mismatch When **both** `anonymize_lm` and `enable_pii_aware_retrieval` are active, a critical problem arises. After retrieval, chunks are de-anonymized (tokens replaced with raw PII), then re-anonymized before being sent to the LM. The LM's PII re-anonymizer may mint **new tokens** for entity names in the chunk that don't exactly match the query entity names already in the session map. **Example of the mismatch:** ``` Session state: "Ashish" → (from user query "What did Ashish publish?") Retrieved chunk (after de-anonymization): "Ashish Vaswani published the Attention Is All You Need paper..." LM re-anonymization sees "Ashish Vaswani" as a NEW entity: "Ashish Vaswani" → (new token minted!) LM receives: Question: "What did publish?" Context: " published the Attention Is All You Need paper..." Result: LM cannot correlate with → reduced answer quality ``` #### The Solution: Normalize Before Re-anonymization The pipeline step have normalize kb strategy that runs **before** the LM re-anonymization step. It replaces KB entity surface forms in chunk **plain content** with the query entity form that already has a session token. When the LM re-anonymizer processes the normalized content, it reuses the existing token. **After normalization:** ``` Before: "Ashish Vaswani published the Attention Is All You Need paper..." After: "Ashish published the Attention Is All You Need paper..." Now the LM re-anonymizer sees "Ashish" and reuses : "Ashish" → (existing token reused!) LM receives: Question: "What did publish?" Context: " published the Attention Is All You Need paper..." Result: LM correctly correlates question and context ✅ ``` #### How Normalization Works The process follows these steps: 1. **Extract KB entities** from `chunk.metadata["entities"]`. 2. **Get current-turn mappings** — filter `anonymized_mappings` to only those whose `anonymized_value` token appears in the current query's masked text. 3. **Find replacement pairs** based on the active `filter_mode`. 4. **Apply replacements** with word-boundary regex, longest-KB-entity-first to prevent clobbering. 5. **Return normalized chunks.** #### Normalization by Filter Mode **Contains Mode** Applies **Case 1 only**: query entity is a substring of the KB entity (`query ⊂ KB`). The KB entity (longer form) is replaced with the query entity (shorter form). **Example:** * Query: `"Stanford"` → `` * KB entity: `"Stanford university"` * Result: `"Stanford university"` → `"Stanford"` (shorter, already-mapped form) **PERSON entities are EXCLUDED** from Contains matching to avoid false-positive attributions. A short person name like `"John"` could match unrelated KB entities like `"Johnson & Johnson"`, incorrectly replacing `"Johnson & Johnson"` with `"John"` in chunk content. **Case 2 is skipped** (KB is shorter, query is longer): Replacing a shorter KB string with a longer query form risks false-positive replacements of common substrings in chunk text. **Fuzzy Mode** Uses Levenshtein similarity matching. **All entity types** (including PERSON) are eligible. **Example:** * Query: `"Ashish"` → `` * KB entity: `"Asish"` (typo) * `Similarity("ashish", "asish")` = 91% ≥ 85% threshold * Result: `"Asish"` → `"Ashish"` **Guards:** * Only query entities with length ≥ **5** are eligible for fuzzy matching. * Length ratio between shorter and longer entity must be ≥ **0.25** to prevent extreme mismatches (e.g., a 2-char nickname against a 20-char full name). * Similarity threshold: **85%**. **Contains + Fuzzy Mode** Both strategies are applied. A KB entity is replaced if it matches **either** the Contains rule or the Fuzzy rule. **Exact Match Mode** **No normalization is performed.** Exact matches are handled natively which already finds identical strings in the session map. #### Normalization Guards | Guard | Value | Purpose | | ---------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------- | | `MIN ENTITY LENGTH RATIO` | 0.25 | Prevents replacing very short strings with very long ones or vice versa | | `FUZZY THRESHOLD` | 0.85 (85%) | Ensures fuzzy matches are close enough to be semantically the same | | `MIN FUZZY ENTITY LENGTH` | 5 | Prevents fuzzy matching on very short entities (high false positive risk) | | `MIN ENTITY LENGTH` (ES filter) | 3 | Skips overly short entities in ES wildcard/fuzzy queries | | PERSON exclusion | Contains mode only | Prevents `"John"` matching `"Johnson"` in chunk content | | Pre-masked Anonymizeable | Any mode | If chunks already have `masked` set (PrivacyGatewayStep ran), normalization is skipped to avoid double-processing | | Current-turn filtering | Any mode | Only entities from the current query turn are used for normalization (prevents stale session entities from leaking) | | Longest-first replacement ordering | Any mode | Replacements are sorted by KB entity length descending to prevent shorter patterns from clobbering longer ones | *** ### Fusion Strategy — Weighted RRF The two KNN legs are fused using **Weighted Reciprocal Rank Fusion (RRF)**: ``` score(doc) = Σ weight_i / (rank_i(doc) + k) ``` Weights are derived from the `vector_weight` config field: | Leg | Weight Formula | Default | | --------------------------------- | ------------------- | --------- | | Filtered KNN (entity-constrained) | `1 - vector_weight` | 0.2 (20%) | | Semantic KNN (unconstrained) | `vector_weight` | 0.8 (80%) | **Why this weighting:** The filtered leg has a lower weight because it is the narrower constraint — it surfaces entity-relevant documents. The semantic leg handles broader topical recall. The asymmetry means that a document appearing in both legs (entity-relevant AND semantically relevant) gets the highest combined score. **Tuning guidelines:** * Increase `1 - vector_weight` / Reducing `vector_weight` → Prioritize entity-match precision * Increase `vector_weight` → Prioritize semantic breadth **Changing `vector_weight`**: This single config field controls **both** the PII-aware retrieval weights and the standard hybrid search weights for the same chatbot. #### RRF Calculation Example ``` Input: filtered_results: [chunk_A, chunk_B, chunk_C] (ranks 1, 2, 3) semantic_results: [chunk_D, chunk_A, chunk_E] (ranks 1, 2, 3) weights: [0.2, 0.8], k = 60 chunk_A: filtered=1, semantic=2 → 0.2/(1+60) + 0.8/(2+60) = 0.0160 ← highest chunk_D: semantic=1 → 0 + 0.8/(1+60) = 0.0131 chunk_B: filtered=2 → 0.2/(2+60) + 0 = 0.0032 chunk_E: semantic=3 → 0 + 0.8/(3+60) = 0.0127 chunk_C: filtered=3 → 0.2/(3+60) + 0 = 0.0032 Final order: [chunk_A, chunk_D, chunk_E, chunk_B, chunk_C] ``` *** ### Configuration Reference | Config Field | Type | Default | Level | Description | | ---------------------------- | -------- | ------------ | -------------------- | -------------------------------------------------------------------------------------- | | `enable_pii_aware_retrieval` | bool | `false` | PRESET | Enables PII-aware retrieval with entity filtering + hybrid search | | `pii_entity_filter_mode` | dropdown | `"Contains"` | PRESET | Entity filter strategy: `"Exact Match"`, `"Contains"`, `"Fuzzy"`, `"Contains + Fuzzy"` | | `vector_weight` | float | `0.8` | PRESET | Weight for semantic leg; filtered leg = `1 - vector_weight`. Must be 0.0–1.0. | | `anonymize_em` | bool | `false` | PRESET | Anonymize queries for the embedding model | | `anonymize_lm` | bool | `false` | RUNTIME | Anonymize queries and chunks for the language model | | `pii_safe` | bool | `false` | LMRP catalog | Marks self-hosted models as PII-safe (bypasses anonymization) | | `pii_enabled` | bool | `false` | DPO Preset | Enables PII processing in DPO pipeline (data ingestion time) | | `FEATURE_ANONYMIZE_ENABLED` | bool | `false` | Environment Variable | Feature flag for the entire anonymization system | *** ### Trade-offs and Limitations #### 1. Performance Overhead PII-aware retrieval runs **two** KNN searches in parallel instead of one, adding query latency. The PII detection/anonymization steps in preprocessing also add latency. #### 2. Reduced Accuracy from Entity Normalization When chunks are normalized (KB entity → query entity), the chunk content is **modified**. This can change the meaning in some edge cases: * **Over-trimming:** `"Google Brain published..."` → `"Google published..."`. The chunk no longer mentions "Google Brain" specifically, potentially affecting the LM's answer specificity. * **PERSON entity limitation in Contains mode:** Because PERSON entities are excluded from Contains normalization, the token mismatch problem still exists for person names that differ in surface form between query and KB. For example, query `"Ashish"` but KB stores `"Ashish Vaswani"` → normalization is skipped → `` is minted → LM can't correlate. #### 3. Fuzzy Matching False Positives * The ES `fuzzy` query with `fuzziness: AUTO` can match slightly different but semantically unrelated entity names (e.g., `"Stalin"` matching `"Stallone"` if they're close enough in edit distance — unlikely at 85% threshold but possible with very short names). * The 85% fuzzy threshold in chunk normalization may occasionally normalize a different but similar-named entity. #### 4. Short Entity Exclusion Entities shorter than 3 characters are excluded from wildcard/fuzzy ES filters. If a user queries about `"Li"` (a common Chinese surname), the entity filter will fall back to Exact Match, which may fail if the KB stores `"Li Wei"`. #### 5. Non-ES Data Store Limitation Chroma and Redis vector stores **do not support** wildcard or fuzzy queries. The entity filter always falls back to Exact Match on these stores, regardless of `pii_entity_filter_mode` configuration. Users on Chroma/Redis only get the basic hybrid search behavior without the improved Contains/Fuzzy matching. #### 6. Shared `vector_weight` Config `vector_weight` controls both PII-aware retrieval weights and standard hybrid search weights. There is no way to independently tune the PII-aware weights vs. the non-PII-aware weights for the same chatbot. Fortunately, we won't use standard hybrid search if we enable PII-aware retrieval. #### 7. Filter Mappings to Current Turn Only Filter mapping restricts entity mappings to only those whose `anonymized_value` token appears in the current query's + transformed retrieval query masked text. This prevents stale session entities from prior turns from leaking into the entity filter, but it also means that entities mentioned in previous turns but not the current one will NOT contribute to the filtered KNN leg. The semantic KNN leg still provides recall for these entities. #### 8. De-anonymization Requires `pii_map` in Chunk Metadata If knowledge base chunks do not have `pii_map` in their metadata (common when KB content is not masked at storage time), the de-anonymization step is a no-op. The retriever returns chunks with tokens still in the content, meaning downstream components receive masked content instead of raw PII. *** ### Scenarios Where Accuracy Can Be Reduced #### Scenario 1: PERSON Name Token Mismatch (Contains Mode) ``` Config: pii_entity_filter_mode = "Contains", anonymize_lm = true User query: "What did Ashish publish?" → PII detection: "Ashish" → KB chunk: "Ashish Vaswani is the lead author of the Attention paper." → Stored entity: "Ashish Vaswani" Contains mode ES filter: *Ashish* → chunk IS found ✅ But chunk normalization SKIPS PERSON entities in Contains mode: → "Ashish Vaswani" is NOT normalized to "Ashish" → LM re-anonymizer sees "Ashish Vaswani" as a new entity → LM receives: Question: "What did publish?" Context: " is the lead author of the Attention paper." Result: LM may fail to correlate with and give an incomplete answer. ``` **Mitigation:** Use **Fuzzy** or **Contains + Fuzzy** mode, which normalizes PERSON entities via Levenshtein similarity. #### Scenario 2: Over-Broad Wildcard Matches Flooding Results ``` Config: pii_entity_filter_mode = "Contains" User query: "Tell me about the University policy" → PII detection: "University" → Contains mode ES filter: *University* → matches ALL stored entities containing "University": - "University of Toronto" ✅ - "Stanford University" ✅ - "University of Cambridge" ✅ - "University Hospital" ✅ (unrelated!) - "The University Press" ✅ (unrelated!) The filtered KNN leg returns documents from ALL these entities, diluting the relevance of the actual results the user wanted. The semantic leg (80% weight) compensates somewhat, but the filtered leg's results may still push more relevant documents down in the final RRF ranking. ``` #### Scenario 3: Fuzzy Match Misses Long Entity Names ``` Config: pii_entity_filter_mode = "Fuzzy" User query: "Tell me about OpenAI" → PII detection: "OpenAI" → KB stored entities: ["OpenAI Research Lab", "OpenAI LP"] Fuzzy ES filter with "OpenAI": • "OpenAI Research Lab": edit distance = 13 → ❌ (AUTO fuzziness limits to ~1-2 for 6-char term) • "OpenAI LP": edit distance = 3 → ❌ (still too far) Result: Fuzzy-only filter returns 0 documents in the filtered leg. The semantic leg (80% weight) still provides recall, but the entity-filtered guarantee is lost entirely. ``` **Mitigation:** Use **Contains + Fuzzy** to catch partial-name matches via wildcard. #### Scenario 4: Missed Search Due to Short Entity Names ``` Config: pii_entity_filter_mode = "Contains" User query: "What did Li contribute to the project?" → PII detection: "Li" → Entity length check: len("Li") = 2 < 3 → SKIPPED in wildcard filter Result: The filtered KNN leg falls back to Exact Match, looking for "Li" as an exact array element in metadata.entities.keyword. If the KB stores "Li Wei" instead, no match. The entity filter produces 0 results, and only the semantic leg contributes. This is especially impactful for users with very short names. ``` #### Scenario 5: Normalization Modifies Chunk Meaning ``` Config: pii_entity_filter_mode = "Contains", anonymize_lm = true User query: "What happened at Google?" → PII detection: "Google" → KB chunk stored entity: "Google Brain" Chunk content: "Google Brain developed the Transformer architecture at Google Brain's research lab." Normalization (Contains, non-PERSON): "Google Brain" → "Google" After normalization: "Google developed the Transformer architecture at Google's research lab." The LM receives a chunk that no longer mentions "Google Brain" specifically. The LM may give a less specific answer (referencing "Google" broadly instead of "Google Brain"). ```