Generative AI System Design Interview (2) -- Gmail Smart Compose

拿gmail 补全作的一个case study。 感觉此书知识结构真挺好的,非常系统,大脑褶皱看平了

Introduction

说白了这一章就是教我们LLM补全的设计架构

Gmail's Smart Compose feature [1] assists users by suggesting the next few words as they write an email.

Clarifying Requirements

seems 我们作为被面试人需要给面试官提一串需求来去定位我们的设计

function- suggest range? personalized?

dataset - size? range? language?

output -biased?

system - user number? real-time?

Specifying the system's input and output

Choosing a suitable ML approach

rnn vs transformer

Various techniques are introduced to reduce the complexity of attention.

Group Attention [6] and FlashAttention [7]

Data Preparation

data source: general data/email data

Text cleaning

Remove non-English text/ confidential information(脱敏)/ irrelevant characters or symbols/ duplicated data

Text normalization

时间,日期,电话等归一化

Text tokenization

Tokenization level

目前最好用的是subword-level

Tokenizer choose

Byte-Pair Encoding (BPE)

SentencePiece .

Tiktoken

token indexing

converting textual tokens into integer numbers

Model Development

Transformer Architecture

这个我们这里也不展开了,平时算法笔记也做的比较详细了

Encoder-only -- understanding the overallmeaning of a text.

Decoder-only -- processes the input sequence and generates a newsequence iteratively

Encoder-decoder -- encoder component processes the input sequence and a decoder uses that processed information to generate the output sequence ( e.g 翻译)

在这个续写任务中,我们当然就用decoder啦

A decoder-only Transformer consists of the following components:

Text embedding

把每个 token ID 映射成高维向量,作为神经网络真正处理的输入。

Positional encoding

positional encoding provides the Transformer with position information for each token in the input sequence 一般情况下选 fixed 准没错

Fixed positional encoding

pros: Efficiency, Support for long sequences

cons: Predefined limits(就是字数不能超), Suboptimal performance

Learned positional encoding

pros:Optimal performance

cons: Inefficiency, Lack of generalization

Transformer

Transformer architecture consists of a stack of blocks. Each block contains the

  • Multi-head attention: This layer updates each embedding by using the attention mechanism. The attention mechanism captures the relationships in the sequence by allowing each embedding to attend to its preceding embeddings. Due to the nature of its mechanism, multi-head attention is commonly known as self-attention, a term we'll use throughout the rest of this book.
  • Feed forward: This layer applies two linear transformations, with a ReLU activation in between, to each embedding in the sequence independently.

Prediction head

the final component in a decoder-only Transformer, translates the Transformer's output into probabilities for every token in the vocabulary.

These probabilities are used to choose the most likely next token.

Training

two-stage training

why two stage?

Adaptability, Improved generalization, Fast finetuning, Handling data scarcity, Mitigating overfitting, Resource optimization

pretraining

一般工程中我们都拿这个阶段已经训练好的模型

The purpose of pretraining is to develop a model capable of understanding natural language, including syntax, common knowledge, and language structures.

ML objective and loss function

In the case of text generation, the most commonly used ML objective is "next-token prediction."

btw 这里注意区分一下masked prediction

训练方式典型模型输入能看什么训练目标更适合
Next-token predictionGPT / LLaMA / Qwen 这类生成式 LLM只能看前文预测下一个 token文本生成、对话、续写
Masked prediction / MLMBERT可以同时看左右文预测被 [MASK] 掉的词文本理解、分类、检索、匹配

Cross-entropy loss

这个也不多解释了,就是奖励应该出现的那个token

In practice, the model processes all token lengths within a sequence in parallel. This allows it to compute the loss for each token position simultaneously. Parallelizing this step speeds up training by handling multiple tokens at once, instead of sequentially.

Transformer 会同时为每个位置输出一个 next-token 概率分布, 然后每个位置都和正确答案算一次 cross-entropy loss,最后加起来或取平均

causal mask = 防止模型在预测下一个 token 时偷看后面的 token。

  • BOS(beginning of sequence) 和 EOS(end of sequence) 是 序列起始符/结束符号

finetuning

这里就用到专项(task- specified)数据集了(邮件)

ML objective and loss function

同 pretrain, next-token prediction + cross-entropy loss

缺失信息处理

这个比较工程,比如你打了一个 dear, 模型怎么知道要写谁的名字呢,于是他去更深的信息去翻

Combining various inputs

总之放到工程就可以理解为把各种信息拼到一块吧,比如一个json 可以包含各个类别的信息

Sampling

process of using a trained generative model to generate new data

The process continues untill the model predicts the <EOS> token

一般情况下选 deterministic,Consistency, Better handling of common phrases, Reduced risk of inappropriate suggestions

deterministic

generate text in a deterministic way, that is, without randomness or variability in the output.

At each step of token generation, the model selects the token with the highest probability from the predicted distribution.

pros:Consistency, Predictable outputs

cons: Lack of diversity, Repetitive text

我草原来这种重复的罪魁祸首是greedy search哇

always selects the token with the highest probability as the next token

保留多条高分路径

track multiple potential sequences of tokens simultaneously. At each step, the model calculates the probabilities for the next possible tokens for each sequence and selects the "top-k" most probable sequences.

  1. Initialization: Start with the user's partial email as the input to the trained model. The model predicts the probability distribution for the next token. Beam search selects the top three tokens with the highest probabilities.
  2. Expansion: For each top three sequence, pass it to the model and obtain the probabilities of the next token.
  3. Pruning: Select the top three sequences based on their cumulative probabilities.

Once the beam search algorithm has stopped, we select the sequence with the highest cumulative probability as the output.

cons:Limited diversity, Struggle with long sequences

Stochastic 随机的

Stochastic sampling methods introduce randomness into the generation process.

For example, at each step of token generation, the model samples from the predicted distribution based on the probabilities assigned to each token.

pros Diversity Novelty

cons Inconsistency Unexpected outputs

注意: stochastic sampling 不是不会重复,而是相比 greedy / beam 这种 deterministic decoding,它更不容易被固定的最高概率路径锁死。

在邮件补全的场景下,补全的内容通常比较短,也不期望多样性,没有必要上beam search

Deterministic vs Stochastic

CharacteristicDeterministic methodsStochastic methods
ApproachFollow a predictable process to generate outputGenerate output based on probability distribution
EfficiencyTypically less efficient due to tracking multiple pathsMore efficient since randomness allows for quicker selections
QualityCoherent and predictableDiverse and creative
RiskUsually lead to repetitive output for longer sequencesMight produce inappropriate output due to their creativeness
Use caseSuitable for tasks requiring consistency, such as language translationSuitable for tasks requiring creativity, such as open-ended text generation
MethodsGreedy search, beam searchMultinomial, top-k, top-p

Evaluation

Offline evaluation metrics

ensure the model's performance is acceptable before deploying it to production

Offline evaluation uses pre-collected and historical data to evaluate a model's performance

除了 Perplexity,Ecact match 还有 BLEU score and ROUGE-N, 后边讲

Perplexity 困惑度

This metric measures how accurately the model predicts the exact sequence of tokens present in text data.

越低越好

ExactMatch@N

percentage of generated phrases that are exactly N words long and that match the first N words of the ground-truth text.

Online evaluation metrics

how a model performs in real time as users interact with the system

defined based on specific requirements and needs

User engagement metrics

Acceptance rate

Usage rate

Effectiveness metrics

Average completion time

Latency metrics

System response time

Quality metrics

Feedback rate

Human evaluation

ML System Design

这个好啊,有一说一感觉平时工作都是在前面的部分

针对Smart Compose,整个流程主要是

  1. Monitoring: The triggering service monitors the user's activity as they type.
  2. Triggerring: The service triggers the phrase generator once it identifies specific patterns.
  3. Beam search: The phrase generator employs beam search to get top-k potential completions from the trained model.
  4. Filtering: The phrase generator interacts with the filtering component to remove long suggestions and those with low confidence scores.
  5. Post-processing: The completion with the highest score is picked and passed to the post-processing service. The service replaces gender-specific pronouns and adjusts sensitive terms.
  6. Display suggestion: The suggestion is displayed to the user for their consideration.

Triggering service

triggering service activates the Smart Compose feature by monitoring user activity such as keystrokes.

判断需要的时候再触发

Phrase generator

这个就是我们刚刚讨论了老半天的算法部分

It generates the most likely completion based on the partial text the user has already typed.

Removing long suggestions

Removing low-confidence suggestions

好简单粗暴....

Post-processing service

addresses potential biases before suggestions are presented to the user

Pronoun replacement

Gender-neutral word replacement

Lexical ( 词汇) analysis for sensitive terms

NSFW (Not Safe For Work) content filtering

Other Question to prepare

下回再说,现在主要是给书过一遍

  • Supporting Smart Compose in multiple languages [28].
  • Personalizing suggestions [28].
  • Incorporating additional context for better predictions [28].
  • Understanding how different tokenization algorithms work, such as BPE [11], SentencePiece [12], and WordPiece [29].
  • Understanding different ML objectives such as masked language modeling (MLM) and its variations [18].
  • The multi-token prediction objective and its pros and cons [30].
  • Balancing quality and inference latency [28].