Generative AI System Design Interview (2) -- Gmail Smart Compose
拿gmail 补全作的一个case study。 感觉此书知识结构真挺好的,非常系统,大脑褶皱看平了
Introduction
说白了这一章就是教我们LLM补全的设计架构
Gmail's Smart Compose feature [1] assists users by suggesting the next few words as they write an email.

Clarifying Requirements
seems 我们作为被面试人需要给面试官提一串需求来去定位我们的设计
function- suggest range? personalized?
dataset - size? range? language?
output -biased?
system - user number? real-time?
Specifying the system's input and output

Choosing a suitable ML approach
rnn vs transformer

Various techniques are introduced to reduce the complexity of attention.
Group Attention [6] and FlashAttention [7]
Data Preparation
data source: general data/email data
Text cleaning
Remove non-English text/ confidential information(脱敏)/ irrelevant characters or symbols/ duplicated data
Text normalization
时间,日期,电话等归一化
Text tokenization

Tokenization level
目前最好用的是subword-level




Tokenizer choose
Byte-Pair Encoding (BPE)
SentencePiece .
Tiktoken
token indexing
converting textual tokens into integer numbers


Model Development
Transformer Architecture
这个我们这里也不展开了,平时算法笔记也做的比较详细了
Encoder-only -- understanding the overallmeaning of a text.
Decoder-only -- processes the input sequence and generates a newsequence iteratively
Encoder-decoder -- encoder component processes the input sequence and a decoder uses that processed information to generate the output sequence ( e.g 翻译)

在这个续写任务中,我们当然就用decoder啦
A decoder-only Transformer consists of the following components:
Text embedding
把每个 token ID 映射成高维向量,作为神经网络真正处理的输入。


Positional encoding
positional encoding provides the Transformer with position information for each token in the input sequence 一般情况下选 fixed 准没错

Fixed positional encoding


pros: Efficiency, Support for long sequences
cons: Predefined limits(就是字数不能超), Suboptimal performance
Learned positional encoding

pros:Optimal performance
cons: Inefficiency, Lack of generalization
Transformer

Transformer architecture consists of a stack of blocks. Each block contains the
- Multi-head attention: This layer updates each embedding by using the attention mechanism. The attention mechanism captures the relationships in the sequence by allowing each embedding to attend to its preceding embeddings. Due to the nature of its mechanism, multi-head attention is commonly known as self-attention, a term we'll use throughout the rest of this book.
- Feed forward: This layer applies two linear transformations, with a ReLU activation in between, to each embedding in the sequence independently.
Prediction head
the final component in a decoder-only Transformer, translates the Transformer's output into probabilities for every token in the vocabulary.
These probabilities are used to choose the most likely next token.

Training
two-stage training

why two stage?
Adaptability, Improved generalization, Fast finetuning, Handling data scarcity, Mitigating overfitting, Resource optimization
pretraining
一般工程中我们都拿这个阶段已经训练好的模型
The purpose of pretraining is to develop a model capable of understanding natural language, including syntax, common knowledge, and language structures.
ML objective and loss function
In the case of text generation, the most commonly used ML objective is "next-token prediction."

btw 这里注意区分一下masked prediction
| 训练方式 | 典型模型 | 输入能看什么 | 训练目标 | 更适合 |
|---|---|---|---|---|
| Next-token prediction | GPT / LLaMA / Qwen 这类生成式 LLM | 只能看前文 | 预测下一个 token | 文本生成、对话、续写 |
| Masked prediction / MLM | BERT | 可以同时看左右文 | 预测被 [MASK] 掉的词 | 文本理解、分类、检索、匹配 |
Cross-entropy loss
这个也不多解释了,就是奖励应该出现的那个token

In practice, the model processes all token lengths within a sequence in parallel. This allows it to compute the loss for each token position simultaneously. Parallelizing this step speeds up training by handling multiple tokens at once, instead of sequentially.

Transformer 会同时为每个位置输出一个 next-token 概率分布, 然后每个位置都和正确答案算一次 cross-entropy loss,最后加起来或取平均
causal mask = 防止模型在预测下一个 token 时偷看后面的 token。
- BOS(beginning of sequence) 和 EOS(end of sequence) 是 序列起始符/结束符号
finetuning
这里就用到专项(task- specified)数据集了(邮件)
ML objective and loss function
同 pretrain, next-token prediction + cross-entropy loss
缺失信息处理
这个比较工程,比如你打了一个 dear, 模型怎么知道要写谁的名字呢,于是他去更深的信息去翻

Combining various inputs

总之放到工程就可以理解为把各种信息拼到一块吧,比如一个json 可以包含各个类别的信息

Sampling
process of using a trained generative model to generate new data
The process continues untill the model predicts the <EOS> token

一般情况下选 deterministic,Consistency, Better handling of common phrases, Reduced risk of inappropriate suggestions
deterministic
generate text in a deterministic way, that is, without randomness or variability in the output.
At each step of token generation, the model selects the token with the highest probability from the predicted distribution.
pros:Consistency, Predictable outputs
cons: Lack of diversity, Repetitive text

我草原来这种重复的罪魁祸首是greedy search哇
Greedy search
always selects the token with the highest probability as the next token

Beam search
保留多条高分路径
track multiple potential sequences of tokens simultaneously. At each step, the model calculates the probabilities for the next possible tokens for each sequence and selects the "top-k" most probable sequences.

- Initialization: Start with the user's partial email as the input to the trained model. The model predicts the probability distribution for the next token. Beam search selects the top three tokens with the highest probabilities.
- Expansion: For each top three sequence, pass it to the model and obtain the probabilities of the next token.
- Pruning: Select the top three sequences based on their cumulative probabilities.
Once the beam search algorithm has stopped, we select the sequence with the highest cumulative probability as the output.

cons:Limited diversity, Struggle with long sequences
Stochastic 随机的
Stochastic sampling methods introduce randomness into the generation process.
For example, at each step of token generation, the model samples from the predicted distribution based on the probabilities assigned to each token.

pros Diversity Novelty
cons Inconsistency Unexpected outputs
注意: stochastic sampling 不是不会重复,而是相比 greedy / beam 这种 deterministic decoding,它更不容易被固定的最高概率路径锁死。
在邮件补全的场景下,补全的内容通常比较短,也不期望多样性,没有必要上beam search
Deterministic vs Stochastic
| Characteristic | Deterministic methods | Stochastic methods |
|---|---|---|
| Approach | Follow a predictable process to generate output | Generate output based on probability distribution |
| Efficiency | Typically less efficient due to tracking multiple paths | More efficient since randomness allows for quicker selections |
| Quality | Coherent and predictable | Diverse and creative |
| Risk | Usually lead to repetitive output for longer sequences | Might produce inappropriate output due to their creativeness |
| Use case | Suitable for tasks requiring consistency, such as language translation | Suitable for tasks requiring creativity, such as open-ended text generation |
| Methods | Greedy search, beam search | Multinomial, top-k, top-p |
Evaluation
Offline evaluation metrics
ensure the model's performance is acceptable before deploying it to production
Offline evaluation uses pre-collected and historical data to evaluate a model's performance
除了 Perplexity,Ecact match 还有 BLEU score and ROUGE-N, 后边讲
Perplexity 困惑度
This metric measures how accurately the model predicts the exact sequence of tokens present in text data.
越低越好


ExactMatch@N
percentage of generated phrases that are exactly N words long and that match the first N words of the ground-truth text.

Online evaluation metrics
how a model performs in real time as users interact with the system
defined based on specific requirements and needs
User engagement metrics
Acceptance rate
Usage rate
Effectiveness metrics
Average completion time
Latency metrics
System response time
Quality metrics
Feedback rate
Human evaluation
ML System Design
这个好啊,有一说一感觉平时工作都是在前面的部分
针对Smart Compose,整个流程主要是
- Monitoring: The triggering service monitors the user's activity as they type.
- Triggerring: The service triggers the phrase generator once it identifies specific patterns.
- Beam search: The phrase generator employs beam search to get top-k potential completions from the trained model.
- Filtering: The phrase generator interacts with the filtering component to remove long suggestions and those with low confidence scores.
- Post-processing: The completion with the highest score is picked and passed to the post-processing service. The service replaces gender-specific pronouns and adjusts sensitive terms.
- Display suggestion: The suggestion is displayed to the user for their consideration.

Triggering service
triggering service activates the Smart Compose feature by monitoring user activity such as keystrokes.
判断需要的时候再触发
Phrase generator
这个就是我们刚刚讨论了老半天的算法部分
It generates the most likely completion based on the partial text the user has already typed.

Removing long suggestions

Removing low-confidence suggestions

好简单粗暴....
Post-processing service
addresses potential biases before suggestions are presented to the user
Pronoun replacement
Gender-neutral word replacement
Lexical ( 词汇) analysis for sensitive terms
NSFW (Not Safe For Work) content filtering
Other Question to prepare
下回再说,现在主要是给书过一遍
- Supporting Smart Compose in multiple languages [28].
- Personalizing suggestions [28].
- Incorporating additional context for better predictions [28].
- Understanding how different tokenization algorithms work, such as BPE [11], SentencePiece [12], and WordPiece [29].
- Understanding different ML objectives such as masked language modeling (MLM) and its variations [18].
- The multi-token prediction objective and its pros and cons [30].
- Balancing quality and inference latency [28].