The chatbot I’ve been building answers questions about company policies, most of which are specific to one country. That makes it a multilingual problem twice over. The questions come in whatever language people like to write in. The knowledge base is multilingual too, and each country’s mix is different.
Every answer depends on retrieval. If the right document isn’t among the few the language model gets to read, no prompt will fix the answer. It will either say it doesn’t know or, worse, write something plausible from the wrong source. So before touching prompts I spent a few weeks on the search step: about two dozen configurations, each scored against the same golden set of questions.
This post covers three countries, each with its own knowledge base:
So there are six combinations of knowledge base and question language, and I’ll call each one a cell. Thinking in cells turned out to matter more than any single setting. In short: English questions were fine all along, the overall average hid that questions in Swedish, Polish and Indonesian were quietly losing their documents, and translating each question into the knowledge base’s languages fixed it.
Every question in the golden set is annotated with the page or pages a good answer needs, and tagged with its language and the country it’s about. I wrote about building that set in Golden datasets without the gold rush.
Two metrics did most of the work:
I also tracked how many documents each configuration passed on to the answer model, plus latency and cost, which I’ll give as ratios. Each experiment changed one thing from the configuration before it, so every gain or loss has a single cause. The previous post explains why I read these numbers as evidence rather than as a pass/fail gate.
I started by trying every query mode the search engine offers, from plain keyword matching to hybrid search with a reranker.
Keyword search alone found the right pages about half the time. Vector search, which matches meaning rather than words, jumped to 84%, and hybrid search, which runs both and fuses the results, landed in the same place.
The reranker is a model that reads the question and each candidate page together and scores how well they match. On keyword search it added 20 points of recall. On hybrid search, recall barely moved (84.0% to 83.5%), but MRR jumped from 0.593 to 0.711: the same pages, with the right one much nearer the top.
I kept it, for two reasons. The answer model reads the top results first, so ranking matters. And the reranker’s score is what makes it possible to cut the list with a threshold later. But that flat recall number was hiding something, and it came back in the next step.
Ten documents per question is a lot for the answer model to read, so the next changes were about passing on less.
Together they cost half a point (83.5% to 83.0%) and cut the average number of documents passed on from 10 to 7.9. Up to this point, every configuration had also appended the country name to the query text (“… in Sweden”). With the filter in place that looked redundant, and it hurt other kinds of questions, so the next change removed it. Recall dropped to 81.0%.
Two and a half points down from the base, for a cleaner query and 6.8 documents. Then I split the results into cells.
English questions held up in every knowledge base. Every country has English pages, so an English question always shares a language with something it can find. Sweden’s English questions stayed around 94%, Poland’s rose 4 points, and the United States’ dipped by 2.5. Nothing dramatic.
The other three cells told a different story:
I don’t have a complete explanation for the reranker step. The cell that lost most to it, Swedish questions, is one where the question does share a language with some of the pages, while the cell with no shared language gained. So it isn’t simply “non-English questions do worse”. My guess is that the reranker is weaker on Swedish text in particular. The last step is easier to read: removing the appended country name cost all three non-English cells 5 to 15 points, while the English cells as a whole didn’t change. An English word like “Sweden” seems to have been giving those questions an anchor in the English pages.
These cells are small, so one question moves the number by several points. Even so, a 35-point drop for Swedish questions is not something an overall average should be allowed to hide.
The fix was to stop searching with the question as written, and search in the knowledge base’s languages instead. The walkthrough below goes through it step by step. Steps 1 and 2 recap the single search path, 3 to 5 cover translation and merging, and 6 covers a third path I’ll come back to.
Every question now becomes two queries, one per language the knowledge base is written in:
The language the person wrote in is ignored. An Indonesian question about Sweden gets English and Swedish queries, not an Indonesian one, because no pages are written in Indonesian and searching for Indonesian text would find nothing. Picking languages from the knowledge base’s side, not the user’s, is the part of this design that matters most.
Both queries run in parallel through the same hybrid search and reranker. The two result lists are merged by keeping each page’s higher score, and the same threshold applies to the merged list.
Overall recall went from 81.0% to 85.5%. The gain landed exactly where the problem was:
Fifteen questions that used to miss now found their page, and fourteen of them were asked in Swedish, Indonesian or Polish. One English question got worse. The model reads 6.4 documents on average, slightly fewer than before. Compared with the version that still had the country name in the query, recall is 2.5 points higher: a proper translation does what the appended word was doing by accident.
What I can’t tell you yet is how the work splits between the two queries. I didn’t run each one on its own, so I don’t know how much of the gain comes from the English query and how much from the local one. That’s the next experiment.
The cost is one translation call per question, which is the only language-model call in retrieval. Retrieval latency a little more than doubled, but it’s still a small part of the time a full answer takes, and cost per question rose by about a quarter.
The usual way to merge ranked lists is reciprocal rank fusion (RRF). Each document gets 1/(60 + rank) for every list it appears in, and the totals are summed. It ignores the scores completely, which is its strength when the lists come from systems whose scores aren’t comparable.
Here the scores are comparable, because both paths are scored by the same reranker. Taking the maximum keeps that information: a page one path is confident about stays near the top even if the other path missed it. RRF instead rewards pages that turn up everywhere, so a page that was mediocre on both paths can overtake one that a single path was sure of. Step 5 of the walkthrough shows this happening.
When I tested both, recall was the same (84.0% against 83.7%), but max score ranked the right page higher: MRR 0.706 against 0.637. There’s a practical reason too. A threshold only makes sense on a score with a fixed meaning, and RRF totals don’t have one.
The condition matters, though. If your paths are different retrievers with different score scales, such as raw keyword scores next to vector similarities, the max is meaningless and RRF or score normalisation is the right choice.
HyDE (Hypothetical Document Embeddings) asks a language model to write a plausible answer, then searches with that passage instead of the question. The idea is that an answer-shaped passage looks more like the documents than a question does.
Used on its own instead of the question, it changed nothing overall.
Overall recall stayed at 83.0%: eight questions gained, seven lost, and retrieval was four times slower. Underneath, Swedish questions gained 5 points and Indonesian questions lost 5. That mix is what made it worth one more try. A technique that finds different pages can be useful as an extra path even if it doesn’t beat the original, so I added it as a third parallel path with the same max-score merge. A path that only adds candidates can’t push good results out, because the merge keeps whichever score is highest.
Next to the two translated queries, it added 0.6 points: 86.1% against 85.5%, three questions gained (two of them in the United States) and one lost. That cost 2.4 times the retrieval latency and 4.6 times the language-model tokens. For a chatbot, where people are waiting for an answer, that isn’t worth it, so the chat setup uses two paths.
The lesson is to judge an extra path against your best setup, not a weak one. Once the translations were in place, most of what HyDE could have caught was already being found.
The matrix below puts the whole story in one place. Rows are knowledge bases, columns are question languages, and each cell shows how that combination did. Step through the configurations, or press play, and watch which cells move. Switch to MRR to see the ranking side.
Rows are knowledge bases, columns are the language the question was asked in. Darker cells score higher.
Configuration used for chat: English and country-language translations, merged by max score.
| Knowledge base | English | Swedish | Polish | Indonesian |
|---|---|---|---|---|
| SwedenSwedish + English pages | 93.5% | 100.0% | not asked | 100.0% |
| PolandPolish + English pages | 83.3% | not asked | 85.0% | not asked |
| United StatesEnglish pages | 65.3% | not asked | not asked | not asked |
Compared with the hybrid-plus-reranker base, the configuration I use for chat finds the right pages more often (85.5% against 83.5%), ranks them about as well (MRR 0.705 against 0.711), and passes on 6 documents instead of 10. The overall gain looks modest because most questions are in English, and English questions had nothing to gain. Swedish and Indonesian questions went from the worst cells to perfect ones, and Polish questions won back what they had lost.
I also checked top-k. Keeping only 5 results cost over 3 points, and keeping 20 added under a point for twice the reading, so 10 stayed.
That sounds like a small requirement. It rules out the most obvious implementation: toss a fresh coin on every request. You could save every user’s choice in a database, but then every server needs access to that record. PlanOut offers another way: make the coin toss repeatable from the inputs.
This is the part of experimentation systems I want to take apart here. One shopper, one button, and the actual numbers from the archived Python implementation. No production traffic or company internals: the example below uses invented IDs.
The product owns the checkout page. The experiment owns the rule for choosing the button. Those should not become one tangled piece of code.
PlanOut was developed at Facebook as a language and framework for defining online experiments. Its 2014 paper describes how an experiment maps a unit, such as a user ID, to parameter values. A parameter might be a button colour, a piece of text, or a discount. Product code asks for that value and uses it.
In other words, the experiment says “blue”. It does not draw the button, count purchases, or tell us whether blue is better.
Here is a small definition using PlanOut’s Python API:
from planout.experiment import SimpleExperiment
from planout.ops.random import UniformChoice
class CheckoutButton(SimpleExperiment):
def setup(self):
self.salt = "checkout-v1"
def assign(self, params, userid):
params.button = UniformChoice(
choices=["blue", "pink"], unit=userid
)
experiment = CheckoutButton(userid="user-42")
colour = experiment.get("button") # "blue"
Andi’s invented ID is user-42. It is the unit: the thing we randomise. If we used a session ID instead, we’d be randomising sessions, and Andi could get a different button on his next visit. Choosing the unit is a design decision, not a detail to leave to the SDK.
A hash function turns an input string into a fixed-length output. The same input always gives the same output. Small changes to the input usually produce very different outputs.
PlanOut uses SHA-1 as a mixing function for assignment. This is not a password scheme or a way to hide someone’s group. We want repeatable, well-spread values, not secrecy.
For this example, PlanOut joins three pieces with dots:
checkout-v1.button, supplied from the assigned variable name.user-42.The complete key is checkout-v1.button.user-42. The default separator is a dot; explicit salt overrides and multi-part units have their own paths in the source.
Change the user ID and follow the actual calculation. The salt stays checkout-v1, the parameter stays button. Nothing is sent to a server.
Computing the assignment...
SHA-1 produces 40 hexadecimal digits. The Python implementation keeps the first 15. Each hex digit holds four bits, so that prefix gives a 60-bit integer. For Andi:
key: checkout-v1.button.user-42
SHA-1: 3441f9fc514ea2ac8f04b1c44a64d90aa350f4fb
first 15: 3441f9fc514ea2a
integer h: 235347851596851754
h % 2: 0
choices[0]: blue
% means remainder. Dividing an even number by 2 leaves remainder 0; an odd number leaves 1. With two choices, that is the whole UniformChoice decision: choices[h % 2].
A thousand servers can repeat it. None needs a stored row saying “Andi got blue”, as long as they agree on the input and implementation.
It is tempting to explain every assignment as “put the hash on a line from zero to one, then split the line in half”. That is a useful picture for weighted choices. It is not the exact code path for UniformChoice.
The archived Python implementation has two different operations:
UniformChoice: use the integer’s remainder to select a list position.WeightedChoice: divide the integer by a scale, then compare it with cumulative weights.For the second operation, the scale is 2**60 - 1, or 1,152,921,504,606,846,975. Andi’s normalized value is about 0.2041317216. With weights [0.5, 0.5], it lands in blue’s half of the line.
Both operations happen to give Andi blue here. That does not mean they assign every user identically. Replacing one with the other can move users even if both advertise a 50/50 split.
A small source-level detail matters if you want compatibility: PlanOut converts that scale to a floating-point value, and uses <= at the weighted boundaries. The mathematical normalization includes 1 at the maximum hash. Replacing the denominator with 2**60, changing the comparison, or switching to another port without checking its hash behaviour is not a harmless cleanup.
Think of the salt as the label on a shuffled deck. With the same deck label and the same user ID, draw the same card again. Change the label and you get a new shuffle.
Here are computed results for 10,000 invented users, user-0 through user-9999, using the same key construction and UniformChoice modulo rule:
| Change | Users whose button changed |
|---|---|
Repeat checkout-v1, same choices |
0 / 10,000 |
Change salt to checkout-v2 |
4,984 / 10,000 |
Keep salt, reverse [blue, pink] |
10,000 / 10,000 |
The original split was 5,004 blue and 4,996 pink. These are reproducible results for this toy population, not a promise that every experiment produces those exact counts.
These are the same 200 IDs, user-0 through user-199. Each dot uses SHA-1 and the first 15 hex digits, then h % 2. Change only the salt and see which users cross over.
Salt: checkout-v1 · parameter: button
Computing 200 assignments...
For Andi, the new salt produces prefix 5c87192d9dc8829, integer 416707841066043433, and remainder 1. He moves to pink. Reversing the original choices also gives him pink, but for a different reason: his original index is still 0; index 0 now means pink.
That last row is the trap. Keeping the hash stable is not enough. Keep the meaning of its output stable too.
A new experiment can use a new salt deliberately. A live experiment should not casually rename its salt, change ID formatting, reorder choices, edit weights, or switch hashing algorithms. Any of those can change the experience for people already enrolled. Stateless repeatability is not the same thing as a saved sticky assignment that survives definition changes.
The parameter name acts as a salt too. button and hint get different hash keys for the same user, unless you intentionally share an explicit salt. This keeps two parameters from being perfectly tied to each other by accident.
Now suppose two teams both want to test the checkout page. One tests the button; the other tests the layout. We do not want Andi in both at the same time.
A namespace is a shared allocation space. In the Python implementation, a user hashes into an integer segment. Experiments reserve disjoint sets of segment IDs. The user’s segment decides which experiment, if any, runs. Only then does that experiment choose its parameter values.
The numbered tiles above are an illustration, not Andi’s computed upstream namespace segment. The real add_experiment() samples from the available segments, so an experiment’s allocation need not be a neat contiguous range.
Mutual exclusion applies inside that namespace: a segment cannot belong to both checkout experiments at once. A separate namespace for the homepage can also enrol Andi. Namespaces do not automatically prevent interactions between experiments on different surfaces.
The namespace name, unit, segment count and allocation history are part of the routing contract. Preserving only the button’s salt will not preserve routing if those change.
We have computed a button colour. Has Andi seen the button? Not necessarily. The page might fail before rendering it. The application might ask for the colour on a route where checkout is never shown.
PlanOut gives you a logging hook. In the default Python experiment behaviour, the first eligible get() on an instance triggers exposure logging before returning the parameter. The instance remembers that it logged, so repeated get() calls on that same instance do not each emit a new automatic exposure. A new instance is a new situation; there is no global per-user deduplication promise.
That hook records parameter access, not human viewability. Automatic logging can be disabled, and an experiment marked out of experiment does not log an exposure. The base experiment delegates the log to your implementation; SimpleExperiment writes JSON to a file. Getting a production event pipeline right is still your job.
A useful record connects the unit, experiment, parameter values and time to the definition that produced them. Upstream also records a checksum when available. In a production system I would make the definition version, override, surface and event ID explicit, then decide where the application should emit the event that analysis treats as exposure.
A hash can reconstruct an intended choice if the original inputs and definition survive. It cannot reconstruct a missing render event or a fallback that nobody recorded.
Assign user-0 through user-9999 with checkout-v1.button and h % 2. A 50/50 rule does not promise exactly 5,000 in each group. Run it again: the same inputs give the same counts.
Ready. All computation stays in your browser.
Suppose the button experiment expects an even split, but the logged population contains 700 blue and 300 pink users. That is a sample ratio mismatch, or SRM: the observed group counts are unexpectedly far from the planned allocation.
It is a warning to investigate the experiment, not evidence that pink loses. Maybe assignment changed. Maybe one group has missing events. Maybe the eligibility filter is different on the two paths. Check the population you count, the planned allocation, and the logging before reading an effect estimate. A fixed 50/50 check also does not fit a bandit that intentionally moves traffic.
PlanOut does not perform that investigation or estimate lift for you. Repeatable assignment is one part of trustworthy experimentation. Exposure data, outcome definitions and statistical analysis are separate parts.
The public repository was archived on June 1, 2021 and is read-only. That is verifiable. An exact public explanation from the maintainers for why it was archived is not established in the sources linked below.
So I would not turn the archive badge into a story about Facebook abandoning experimentation, maintenance costs, or a single successor replacing PlanOut. Its own documentation described the interpreter as one way to define experiments at Facebook, running on top of QuickExperiment. The public library was never the whole internal platform.
What has changed for someone choosing a tool today is the available scope. A language gives you definitions and operators. Modern experimentation products can also provide configuration delivery, allocation controls, exposure handling, metric definitions and analysis. Those are reasons to evaluate a broader platform, not proof of the archive’s motive.
There is no official replacement established by the PlanOut sources. The replacement depends on which job you need done.
If I were moving an existing experiment, I would first make a test set of IDs, salts, definitions, overrides and expected outputs. Run old and new systems against it. Explain every disagreement before moving live traffic. A new SDK producing “roughly half in each group” is not enough if it puts different people in those groups.
The lesson I keep from PlanOut is smaller than a whole platform: separate the experiment definition from product code, make assignment repeatable, and make the boundary between a decision and an observation explicit. Andi getting blue twice is easy to demonstrate. Knowing whether blue helped him is a different system.
In the first weeks of building an agent you can iterate by feel. You try a handful of questions, read the answers, change a prompt, try again. That’s a perfectly good way to start.
It breaks down once the system has some reach. The assistant I work on serves several markets, takes questions in any language and mixes global and region-specific knowledge. A change that improves one language can quietly degrade another. Nobody can read enough answers by hand to notice. And because the output is fluent, a worse answer doesn’t look worse.
So I treat evaluation as engineering evidence. Compare a change with a baseline, look at the failures one by one, and only then decide what to change next. This is what people mean by evaluation-driven development: the eval isn’t a check at the end, it’s what steers the design.
I measure four things: answer relevance, retrieval recall, groundedness and latency. I want to be specific about why those four, because each one exists to catch something the others miss.
Two rows in that matrix deserve a comment. “Faithful to sources, but off-topic” shows why groundedness alone is not enough: an answer can quote its sources perfectly and still not answer the question. And the last row shows the limit of all of them. If the source document is out of date, the assistant can be relevant, well retrieved and faithfully grounded in something that is no longer true. No automated metric flags that. A person reading the failures does, or a content review does.
This is why I don’t collapse the four into one number. An average lets a good latency score hide a poor recall score. Reading them side by side tells you which part of the system to look at.
There are two evaluations in the loop, and they answer different questions.
Offline evaluation runs on demand against a fixed set of questions. In my case it runs whenever there’s an architecture change or a significant code change, and it’s also what I use to check that a new market or language is ready. Because the test set is fixed, two runs are comparable. It answers: is this change better than what we have?
Online evaluation runs against live traffic. Its job is to detect regressions and bugs in production, the things nobody thought to put in the test set. It answers: what did we miss?
The loop closes when a failure found online becomes a new offline case. Over time the test set stops being what you thought users would ask and becomes a record of what actually broke. Neither kind is a substitute for the other, and neither is a substitute for understanding the task: someone has to read the failures.
Readiness reviews for a new market combine the offline results with something else: the history of issues that came through support tickets. The eval tells you how the system does on the test questions, and the ticket history tells you what people have actually run into. In practice, the findings from that combination have mostly led to improvements in the source content, not in the code.
The default instinct, and it’s a reasonable one, is to wire the eval into CI and block a release if the score falls below a threshold. It’s what we do with unit tests. I chose not to, and the pipeline I built produces a comprehensive report that compares the current run with a baseline or with agreed criteria, and leaves the decision to an engineer.
My reasoning comes down to three points.
The scores are noisy. Metrics like relevance and groundedness are commonly scored by a language model acting as a judge, and the agent itself is not deterministic. Run the same system twice and you’ll get slightly different numbers. A hard threshold turns that noise into a coin flip:
A single threshold hides the trade-offs. A change that lifts recall while costing a little relevance on a narrow slice might be exactly the change you want. A gate sees one number and says no. A report shows the per-metric differences, the slices that moved, and the actual examples that got worse, so a person can judge whether the trade is worth it.
A red cross ends the conversation. When a gate fails, the natural response is to make it pass: tune until the number moves, or lower the bar. When a report arrives, the natural response is to read it. I’d like the team to argue about specific failing examples, not about the threshold.
I’m not against gates in general. A hard gate suits checks that are cheap, deterministic and clearly binary. A latency budget is a good example: either the response time is inside the budget or it isn’t, and there’s little to interpret. The same goes for a safety filter that must never be bypassed, or a schema check on tool output. Those I’d happily block on.
The metrics I’d keep as reports are the ones that need reading: relevance, and the finer judgements about groundedness. The rule of thumb I use is that if a person would want to look at the case before deciding, it shouldn’t be an automatic block.
Making the evaluation a report changes how people use it. It becomes something you read before a review, alongside the diff, and not a gate that you try to get past. Reports that people actually want to read need to be legible: baseline next to current, differences by metric and slice, and the worst examples one click away. If nobody reads the report, you’ve built a gate with extra steps, so it’s worth spending effort on the reading experience.
The practical lesson is to make evaluation part of the development loop, keep the reports readable by the people making the next decision, and keep a human in the seat where the decision is a judgement call.
Working out the what and the who is a small investigation every time. You find the conversation, read the question, look at what the assistant retrieved and what it said, check whether the right information exists anywhere, and decide whether to fix a document or fix the software. Then you find the person who owns that thing and explain it to them. Done by hand, on every flagged item, it eats the time of the people who should be improving the assistant.
I built and run a workflow that does the first pass of that investigation. This post describes how it’s put together and, more to the point, where I chose not to let the agent decide anything.
The workflow runs weekly, or on demand when someone wants an answer sooner. It reads the production traces for flagged conversations, and for each one it works through the same steps.
Replay. The trace records what the user asked and what the assistant did. The workflow asks the same question again, in the language it was originally asked. This matters more than it sounds. Some complaints describe a problem that has since been fixed, some can’t be reproduced, and some reproduce every time. A flag you can’t reproduce is a different kind of case from one you can.
Check the evidence. This is the step I’d insist on in any version of this system. The workflow looks at what the assistant retrieved and asks whether the information needed for a good answer was actually there. Was the relevant document in the knowledge base at all? Was it retrieved? Does the answer follow from what was retrieved? Everything after this step depends on the answer, so it comes before any conclusion, not after.
Classify. From the evidence, the workflow picks a likely cause from a fixed list. The most important split on that list is between two families of problem:
Route. Each cause on the list maps to an owner, and that mapping is a plain lookup table. More on that below.
The workflow handles feedback, traces and conversations in whatever language they arrived in, since the assistant is used in many. The output is an investigation report per case, and it can open a follow-up ticket for the owner.
It’s easy to treat all bad feedback as “the AI got it wrong” and send it to the engineers. That’s a waste, and in my experience it’s also the wrong diagnosis for a lot of cases. If the answer was wrong because a policy document was ambiguous, the engineer can do nothing useful with the ticket. The person who maintains that document can.
Separating content gaps from application faults sends work to the people who can act on it. It also gives you a quiet second benefit. Counting causes over a few weeks tells you where the assistant’s weak spots really are. If most of the problems turn out to be content gaps, the most useful improvement isn’t a cleverer prompt but better source material. Which way it falls will differ from system to system, but you can’t know until the causes are counted.
A made-up example shows the difference. Suppose a user asks how a policy applies in their region and marks the answer as wrong. The replay reproduces the same answer. The evidence check finds that the regional document exists and was retrieved, but that the answer quotes the global one. That points at how the application chose between sources, so the cause is configuration, and the ticket goes to the engineers. Change one fact: the regional document doesn’t exist, so the assistant fell back to the global one, which was the best it had. Now the cause is a content gap, and the ticket goes to whoever maintains that content. The complaint was identical both times, and the right owner was different.
An agent that always produces an answer is dangerous in a triage role, because the answer is often a confident guess. So the workflow has a rule: missing or conflicting evidence goes to a human.
Missing evidence means the replay couldn’t be run or the trace was incomplete. Conflicting evidence means the signals disagree, for example the right document was retrieved but the answer contradicts it in a way that could point at either the content or the model. In both cases the workflow doesn’t pick a side. It says what it found, says what it couldn’t establish, and hands the case over with the material already gathered.
I think this is the difference between a triage tool people trust and one they route around. The first time it confidently blames the wrong team, the reports stop being read. A tool that says “I can’t tell, here is what I’ve got” loses very little, and it keeps its credibility for the cases where it does commit.
The last design choice is about who does what. The model is good at the fuzzy parts: reading a trace, comparing an answer with retrieved passages, deciding which cause on the list fits. It’s a poor choice for deciding who gets paged, because you want that to be predictable and easy to change.
So the cause-to-owner step is deterministic. Something like this:
OWNER_BY_CAUSE = {
"content_gap": "content-owners",
"configuration": "application-team",
"architecture": "engineering",
}
def route(cause):
# An unknown or missing cause never guesses; it goes to a person.
return OWNER_BY_CAUSE.get(cause, "human-review")
This is a sketch, not the real table, but the principle is what I’m after. If a team reorganises, you edit a mapping. If someone asks why a case went where it did, the answer is a line of code and not “the model felt like it”. And because the classifier has to choose from a fixed list, its output is something the rest of the system can check.
The workflow has helped the team deal with recurring issues and content gaps. I haven’t formally measured how often its root-cause call and its owner assignment are right, and I’d rather say so than quote a number. The comparison I’d make at this stage isn’t accuracy but effort: how much of the manual investigation is already done when a case reaches a person. Because it produces a report with the replay and the evidence attached, a reviewer starts from a worked lead instead of a blank page. My own estimate is that it removes most of the digging per item, based on comparing the manual and the automated routes. It’s an estimate, not a controlled measurement.
If you’re building something like this, the parts I’d carry over are these: check the evidence before concluding anything, make the cause list small and explicit, keep routing deterministic, and give the agent a clear way to say it doesn’t know. The goal is to shorten the investigation, not to automate certainty.
The usual advice for building one is to sit down and write fifty question and answer pairs by hand. That advice is fine for a prototype. It stops working when the assistant serves several markets, several languages and a knowledge base that keeps changing, because nobody has the time to write, and re-write, enough examples to cover all of that.
I work on a production assistant of this kind, and I’m building a pipeline that generates the candidates instead. This post describes the design: simulated personas propose evaluation records, and a person approves each one before it counts. I’ll keep it about the mechanism, not the system it runs in, and I won’t give sizes or rates because those aren’t mine to share.
An evaluation record is more than a question and an answer. In the pipeline I’m describing, each one has:
The source URL matters most. It anchors the record to something checkable. An expected answer with no source is an opinion. An expected answer that points at a specific document can be verified by opening the document, and it gives you a second thing to measure: whether the assistant’s retrieval found that document at all. I’ll come back to that in the last section.
The metadata is what lets you slice results later. An average score across everything hides a lot. If the same assistant does well in one language and badly in another, you want to see that, which means every record needs to say which language it belongs to.
The obvious way to generate questions with a language model is to hand it a document and ask for questions about it. The result is predictable. The questions echo the document’s own vocabulary, they’re all polite and well-formed, and they all have a tidy answer in the text. Real users don’t ask like that.
A persona is a short description of a kind of user: who they are, where they are, what they’re trying to get done, how they tend to phrase things. The generator is asked to write queries as that person would, about a document sampled from the knowledge index. Different personas produce different questions from the same source:
Personas also give you a coverage lever. If a language or an intent is thin in the set, you ask for more of it by changing the persona mix, instead of asking a colleague to write more by hand. The pipeline is configurable, and the persona mix is the natural knob for that: you change the request, not the code.
The design decision I care most about is where the human sits. Generation is cheap, so I want it to over-produce and be varied. Approval is the expensive part, so it needs to be strict. Nothing a model generates becomes golden on its own, whatever score it gets from another model.
The reason is simple. A golden set is the ruler you measure everything else with. If the ruler is wrong, every comparison you make with it is wrong in a way that’s hard to see, because the numbers still look like numbers. A model checking a model’s homework tends to agree with itself.
What does the reviewer actually check? Three things, together, for each record:
Checking all three at once is faster than it sounds, because the record puts the three side by side. The reviewer is reading, comparing and deciding, not researching from scratch. That’s the whole trade: the model does the tedious drafting and the person keeps the judgement.
Generated evaluation data fails in a handful of predictable ways, and it helps to know which check catches which.
The last row is the one people skip. Each record can be individually fine and the set can still be lopsided: twenty near-identical questions about the same document, and nothing about the one that users actually struggle with. Reviewing records one by one won’t show you that. You need to look at the distribution by document, language, geography and intent, and then adjust the personas or the sampling.
It’s tempting to report the size of the golden set as an achievement. I’d avoid that. A set padded with near-duplicates is worse than a smaller one that covers the real spread of what people ask, because it gives you a confident number about a narrow slice. The measures I’d care about are coverage, how much reviewers agree with each other, and how many records they had to edit. A headline count tells you none of that.
The same goes for keeping the set alive. The knowledge base changes, so a record that was correct last quarter can point at a document that has since been rewritten. Because each record carries its source URL, you can find those records mechanically and send them back for review, instead of discovering the problem when a score drops for no visible reason.
Because each record names its source, the evaluation can ask two separate questions. Did the assistant give a good answer? And did it retrieve the document that answer should come from? When the first is bad and the second is good, the fault is in how the answer was written. When both are bad, look at retrieval. That split is much harder to make when the golden set is only questions and answers, and it’s the main reason I insist on the URL.
If you’re building something similar, the parts I’d keep are these: anchor every record to a source, use personas to widen what gets asked, keep a person as the only route into the golden set, and review the set as a whole as well as record by record. The rest is tooling.
A multi-armed bandit handles that by moving traffic toward what’s working while the test is still running. On the experimentation platform I worked on, we ran bandits for exactly this kind of problem, coupon optimisation and adaptive traffic allocation, using Thompson sampling. This post explains the algorithm with coupons, then shows how we fitted it into a production system.
Every bandit balances two needs. Exploiting means showing the coupon that looks best so far, to earn conversions now. Exploring means still showing the others sometimes, because “looks best” is based on limited data and might be wrong. Pure exploitation locks in an early lucky guess; pure exploration is just an A/B test. Thompson sampling gets the balance from how uncertain it is about each coupon.
For each coupon, keep two numbers: how many people saw it and how many converted. Those two numbers define a Beta distribution, the bandit’s belief about that coupon’s true conversion rate, here the share of people who buy after seeing it. Little data gives a wide curve; lots of data gives a narrow one.
The whole algorithm is: for each user, draw one random value from each coupon’s curve, and show the coupon with the highest draw.
import random
def choose(stats):
# stats: {"A": (conversions, views), ...}
draws = {
coupon: random.betavariate(1 + conv, 1 + views - conv)
for coupon, (conv, views) in stats.items()
}
return max(draws, key=draws.get)
That’s enough to get the balance right. A coupon with a high, narrow curve wins most draws, so it gets most of the traffic. A coupon with a wide curve sometimes draws high, so it still gets explored until the data settles what it’s worth. A coupon with a low, narrow curve almost never wins. Nobody has to tune an exploration rate; uncertainty does the work.
The textbook version draws per user, updates the counts after every conversion, and draws again. Ours ran differently: the bandit was a batch job that edits a traffic split. Batching like this has real advantages over per-request sampling:
Every few hours, the job recomputed each coupon’s posterior from logged exposures and conversions, estimated the probability that each coupon is the best, and wrote those probabilities back as the experiment’s new traffic split.
def split_from_posteriors(stats, draws=10_000):
wins = {coupon: 0 for coupon in stats}
for _ in range(draws):
wins[choose(stats)] += 1
return {coupon: n / draws for coupon, n in wins.items()}
Allocating traffic in proportion to “probability of being best” is exactly what Thompson sampling does on average, just applied in batches. The new split then travels to the assignment service like any other config change, which is why the config delivery work mattered: bandits were among the most frequent writers.
Here is a simulated run, with the true conversion rates set to 6%, 10% and 7.5%, and a few hundred users between updates:
The early updates are the interesting part. After the first batch, B and C look about equally good and split the traffic between them. As C collects more data its curve narrows below B’s, and traffic moves to B. A drops out almost immediately.
The payoff is measured as regret: conversions lost compared with already knowing the best coupon.
Both lines start the same, because both begin with an even split. After that, the fixed split keeps losing conversions at the same rate until the test ends, while the bandit’s losses flatten once it’s confident.
The algorithm is ten lines. Most of the work is everything around it:
Use an A/B test when you need to learn: a product decision, an effect size, something you’ll report. Use a bandit when you need to earn while choosing among options with fast, measurable feedback: coupons, banners, notification copy, ranking tweaks. If the options change often, or the test would otherwise run for weeks while showing users a clearly worse option, a bandit usually pays for itself.
I designed a metric semantic layer for an experimentation platform that served dozens of product teams: a centralised metadata schema, plus a query engine on BigQuery that computed metrics from it. The catalogue held more than 100 reusable metrics, and every product team’s experiments used them. This post is about what that work taught me. Most of it was about people, and the schema was the smaller part.
A semantic layer is a place where the meaning of a metric is written down once, in a form that both people and programs can read. Instead of each dashboard or experiment analysis carrying its own SQL for “conversion”, they all refer to one definition, and a query engine turns that definition into the query that runs against the warehouse.
Vendors have plenty to say about this. I’m going to skip the theory and describe the problem it solves.
Nobody sets out to define a metric wrongly. Drift happens because every team has a reasonable local reason:
Each choice is defensible. The trouble is that all of them carry the same name, and a name is what people compare.
For an experimentation platform this costs more than for a dashboard. An experiment reports the effect of a change on a metric. If the metric can mean different things in different teams, results are not comparable, and a headline like “this change lifted conversion” cannot be checked against the last one.
A definition that both a person and a program can rely on has to answer a few questions explicitly. This is the anatomy I’d expect from any semantic layer:
The schema had to be precise enough for a program to compute from it. That was the job of the query engine: it took a definition and produced the BigQuery query for it, so the same definition gave the same number wherever it was used.
You can build the schema and engine and still end up with two versions of conversion, one governed and one not. The technical work leaves open the harder question: who decides what “conversion” means?
In my experience the answer has to be a named owner for each metric. Not a committee, and not “the data team” in general. The person who owns a definition can decide when a request to change it is right, and can say no when a team wants a local variant that would fragment the meaning. Without that, the catalogue turns into a list of opinions.
A few practices follow, and they are the ones I’d insist on:
A catalogue that nobody uses is documentation. What made the difference to me was that the governed definition had to be the easiest option, not just the correct one. If a team can write its own SQL in ten minutes but must file a request and wait a week to use a shared metric, the local copy wins every time, and it wins for good reasons.
That pointed the design at a few things. Some of these I did and some are what I’d emphasise now, and I’ll keep to the general principle rather than the detail of any one system:
Every product team ended up using the catalogue in its experiments. I’d credit that to the definitions being useful and reachable, and I don’t think a mandate would have produced the same result. A rule can make people register a metric. Only a metric that saves them work makes them keep using it.
An owner field is a start. What it needs behind it is a path for changing a definition that people can follow without heroics. A common shape looks like this:
None of this is complicated. The point is that changing what “conversion” means is a decision that affects other people’s results, and the process makes it visible. Otherwise the definition changes silently, and the next dashboard comparison is a debate again.
If I were starting again, I’d check for these early:
The schema and the engine mattered, but a shared definition only becomes a working practice through reuse, and reuse depends on people. Someone has to own each definition, changes have to be visible, and the shared version has to be easier to reach than a copy. Get those right and a dashboard comparison goes back to being a conversation about the product.
Photographers call this the exposure triangle. Neural-network training has its own version of that problem, with three knobs that also pull against each other: learning rate, batch size and warmup. This post lays the two side by side, shows what the training knobs do in a small simulation, and spends a fair amount of time on where the comparison stops being useful.
A camera sensor needs a certain amount of light to produce a well-exposed picture. Three controls set how much it gets, and each one changes something else about the image:
Brightness is measured in stops, and each stop is a doubling or halving of the light. That gives the triangle its useful property: the controls trade against each other one stop at a time. Open the aperture by one stop and you can halve the exposure time for the same brightness. The picture is equally bright, but you’ve swapped a shallower focus for less motion blur.
The point isn’t that any corner is better. The right setting depends on what you care about in the scene, and a good photographer decides that first.
Training a neural network has a comparable shape. You have a budget for how much progress each step can make safely, and three settings that spend it:
The pairing I’m using is aperture with learning rate, shutter with batch size, and ISO with warmup. I’ll explain each pair in turn, and mark how strong I think it is.
Aperture and learning rate (strong). Both are the main lever. They decide how big a step you take per unit of effort, and both have a sweet spot between too little and too much.
Shutter and batch size (fairly strong). A long exposure collects more photons, which averages out random fluctuation. A big batch collects more examples, which averages out sampling noise. In both, more collection reduces noise and costs you something: time on the camera side, compute on the training side.
ISO and warmup (loose). This is the pair I’m least sure about. ISO is a late-stage amplifier: you reach for it when the other two are already stretched. Warmup plays a similar supporting role, because it’s a modest, cheap adjustment that lets the other two be pushed further. But ISO amplifies noise, and warmup does the opposite: it protects you from instability. I’d call this a pairing of roles, not of mechanisms.
Words are easy here, so I ran the experiment. The setup is deliberately tiny: gradient descent on a two-dimensional quadratic bowl, with random noise added to each gradient to stand in for sampling a mini-batch. It’s pure Python, and it’s illustrative rather than a real model.
On the left, batch size is fixed and only the learning rate changes:
On the right, learning rate is fixed and the batch size changes. The averaging effect is visible. A batch of one gives a noisy trace that settles about two orders of magnitude above where a batch of 64 does. Averaging over more examples shrinks the noise floor, roughly in proportion to one over the batch size.
That last sentence hides the coupling. The height of the noise floor depends on both the learning rate and the batch size, in this simple setting roughly as their ratio. Double the learning rate and you raise the floor. Double the batch and you lower it. That’s the same kind of trade as opening the aperture by a stop and shortening the exposure by a stop, and it’s where the analogy earns its keep.
There is a well-known heuristic from large-batch training that has the same shape as the photographer’s reciprocity. If you multiply the batch size by k, multiply the learning rate by about k as well, and add a warmup phase so that the early steps don’t blow up. It’s often called the linear scaling rule, and it comes from work on training image models with very large batches. It isn’t a law. It works up to some batch size and then stops working, and for adaptive optimisers such as Adam, people often find a square-root scaling fits better.
I still find it worth remembering, because it shows why treating the knobs as independent is a mistake. If you double the batch size to “make training more stable” and leave the learning rate alone, you have changed the operating point. The run may get smoother and slower. If you double the learning rate to “make it faster” and leave everything else alone, you have moved towards the edge in the left panel.
A habit I’d suggest, borrowed from the photographers: when you change one control, name the other one you’re spending. “I’m raising the learning rate, so I’m accepting more noise and less margin before divergence.” “I’m shrinking the batch to fit on the device, so I’ll expect a noisier curve and probably a lower learning rate.” Saying it out loud makes the trade visible.
I’ve pushed the comparison as far as I can. These are the places where it stops helping.
Photography has a target; training doesn’t. An exposure is either about right or it isn’t, and the camera can even show you a light meter. Nothing like that exists for training. There is no learning rate that gives the “correct” amount of progress, only better and worse outcomes on a metric you picked.
The knobs aren’t interchangeable. In a camera, a stop of aperture and a stop of shutter are worth exactly the same amount of light, so the trade is precise. In training, batch size and learning rate coupling is a rough heuristic, and it depends on the model, the optimiser and the data. I’d treat any specific scaling rule as a starting guess to test.
Warmup isn’t a peer of the other two. ISO is a number you set once per shot. Warmup is a schedule, with a length and a shape, that runs for the first part of training and then disappears. It matters most for large models and adaptive optimisers, and it does very little in my toy problem, where the loss is a smooth bowl and the only limit is curvature. Also, real training has more knobs than three: weight decay, the schedule after warmup, gradient clipping, the optimiser choice. A triangle undercounts.
The costs are different in kind. In photography every trade lands on a single image, and you see the result at once. In training the cost shows up later, in a curve you read after minutes or days, and sometimes only in an evaluation you weren’t watching. That delay is the reason I think “name what you’re spending” is such a useful habit for training and less necessary in a camera.
Noise means different things. Grain in a photo is nearly always unwanted. Noise in gradient descent is sometimes a nuisance and sometimes useful, because it can help a run escape sharp regions. A smaller batch isn’t just a worse estimate.
I don’t think the analogy predicts anything. It gives me a way to explain to a colleague, or to myself, why “just change the learning rate” is rarely a single-variable question. It also reminds me to start from what I care about, the way you would with a subject that moves or a scene that’s dim.
When I set up or review a training run, I try to ask three questions in that order. What’s the failure I’m most worried about: divergence, slow progress, or noisy results? Which knob buys me the most protection against it? And which cost am I accepting in exchange? Then I look at the curve, the way I’d look at the back of the camera, and adjust.
I built a serving engine and SDK for an ML platform. It supported approximately 40 model deployments, and the teams behind them did not have to configure the underlying infrastructure. This post is about the design thinking behind it. I’ll describe the shape of the interface and the decisions I’d defend. Where I’m talking about general practice rather than my own system, I’ll say so.
A trained model in a notebook is a function: features in, prediction out. A model in production is a service, and the gap between the two is mostly not machine learning. Someone has to:
None of these are hard individually. Together they are a lot of unfamiliar surface area for someone whose expertise is modelling. A data scientist can learn them, but each one who does spends days on work that isn’t their job, and each ends up with slightly different choices. Ten teams shipping ten models by hand gives you ten ways of doing authentication and ten different sets of dashboards.
The engine took over the parts that are the same for every model:
The design test I used for each item was simple: would two different data scientists ever have a good reason to make a different choice here? For authentication and tracing the answer is almost always no, so they belong in the platform. For anything where the answer is yes, they belong in the interface, as an option with a default.
An SDK is easy to get wrong by hiding too much. If the only way to use it is a magic decorator that reads the notebook’s global state, nobody can reason about what it does. I wanted the author’s side to be a few explicit things:
Here is the shape of that, as a sketch rather than the real API:
from serving_sdk import Model, Schema, Config
class ChurnModel(Model):
input = Schema(user_id=str, features=list[float])
output = Schema(score=float)
def load(self, path):
self.model = load_my_model(path) # any framework
def predict(self, request):
return {"score": self.model.score(request.features)}
config = Config(cpu="1", memory="2Gi", replicas=2)
Two things make this a good interface rather than just a short one. First, the class has exactly the three methods and attributes a model needs, so there is little to misuse. Second, nothing in it refers to the framework. The engine calls load and predict, and what happens inside is the author’s business. That is how a serving engine stays framework-agnostic: the contract is about requests and responses, not about scikit-learn or PyTorch or anything else.
Make the right thing the default. Settings that matter for safety or operability, like authentication and tracing, should need no configuration to be on. Turning something off should be deliberate. A default the author must remember to enable will be missed by some teams.
Narrow beats flexible. It is tempting to offer hooks for every possible need. Each hook is a promise you have to keep as the platform changes, and each one lets teams drift back into bespoke setups. I would rather have a smaller interface that covers most models cleanly, and add an option only when several teams have asked for the same thing.
Errors are part of the interface. A stack trace from deep inside the engine tells a data scientist nothing. Validation before deployment, with messages such as “the output schema says score is a float but predict returned a string”, turns a support conversation into a fix the author makes alone. When people say a platform is “easy to use”, a lot of that is error messages.
Examples are documentation. A working example for each common model type does more than a reference page. Most people will copy the closest example and edit it, so the examples define the real interface.
Keep the escape hatch honest. Some models will not fit. The choice is between a documented way out and a quiet fork of the platform. A documented way out, even a clumsy one, keeps those teams visible and lets you learn from them.
Every abstraction leaks somewhere, and the useful question is where you want it to. A common place is performance: a model that needs a lot of memory or a slow start-up behaves differently under the platform’s scheduling from how it did in a notebook. When the layer hides Kubernetes completely, the author has no vocabulary for the problem.
The general fix I prefer is to expose the effects rather than the machinery. Let the author declare what the model needs (memory, warm-up time, concurrency) in the config, and surface what happened (a start-up that took too long, a request that timed out) in terms of those settings. They shouldn’t need to know what a pod is to fix it.
Building one serving path had a second benefit that I underrated at the start. When every deployment is produced the same way, the platform can also say something consistent about all of them. Authentication, tracing and metrics come out in one form, so a single set of dashboards and alerts can cover every model. Governance questions, such as which models are running and who owns them, have one place to be answered.
Nobody had to be asked to follow a standard. It came with the easiest way to deploy.
Existing tools cover this ground, and the build-versus-adopt question is a fair one. My view is a general one and depends on your situation. If an existing serving framework already matches your company’s authentication, tracing and deployment conventions, adopt it and spend the time elsewhere. If most of the work is fitting those conventions, then the real product is the thin layer that connects a model to them, and that layer is where an in-house SDK earns its keep. Either way, the design questions above still apply: the interface your data scientists see is yours to shape.
The measure of success was not the SDK’s feature list. It was that teams could take a model from a notebook to a running endpoint without becoming infrastructure engineers, and that there was one path to keep secure and observable.
That isn’t how it works today. The services that assign users to variants pull their config: they read it through a cache that events and a TTL keep fresh, and apps ask on every request. Most of those reads get the same answer as last time.
When I wrote about getting experiment config to every service, push was the most advanced option, and I held back from it. Push means keeping a connection open to every client for hours, and I didn’t know how well that works inside Kubernetes. So I went and learned. This post is what I found, in the order I wish I’d learned it:
SSE is a normal HTTP request whose answer never finishes. The client asks once. The server replies, keeps the connection open, and writes a new message into it whenever something happens. Think of a phone call where only one side talks, and only when there’s news.
This is what the client receives over time:
id: 42
data: {"experiment":"checkout-cta","status":"paused"}
: ping
id: 43
data: {"experiment":"search-rank","traffic":0.5}
Three things in there matter later:
id. If the connection drops, the client reconnects and says “the last one I got was 42”. The server then sends whatever came after it. Browsers do this automatically, through a header called Last-Event-ID.SSE only goes one way, from server to client. That’s all config delivery needs. If you need both directions, WebSockets are the usual choice.
Most web infrastructure is built for short requests: a question comes in, an answer goes out, done in under a second. An SSE connection can stay open for hours. Between the client and the server there are several middlemen, such as load balancers and proxies, and each one was tuned with short requests in mind.
Four things can go wrong:
There’s also a fifth, smaller one: every open connection takes some memory on every machine it passes through.
With many services, each one needs the same networking features: timeouts, retries, encrypted traffic, and metrics on every call. Without help, every team builds these into its own code, in its own way.
A service mesh takes those features out of the application and puts them into a small proxy next to every copy of every service. An analogy that helped me: each service gets a personal assistant that handles its calls. The service just says “call the push service”. The assistant finds a healthy copy, encrypts the call, retries if it fails, and writes down how long it took. A manager gives every assistant the same rulebook.
Istio is the most common service mesh on Kubernetes. In Istio’s terms:
You write the rules as Kubernetes objects, for example “send 10% of requests to version 2” or “time out after 3 seconds”.
There’s a nice irony here. istiod sends its rules to every Envoy over long-lived connections that stay open all the time. The mesh itself runs on the pattern I was nervous about.
With Istio, an SSE connection from an app to our push service passes through three middlemen: the cloud load balancer, the ingress gateway, and the Envoy sidecar next to the push service. Each has its own timers.
Does the mesh fix it? No, it adds more timers. But the fix is easy.
Every hop has an idle timer. For example, AWS’s load balancer closes connections that have been quiet for 60 seconds, and Envoy closes a stream after five quiet minutes. Envoy also has a 15-second limit on how long a whole response may take, which would cut every SSE connection. Istio switches that limit off by default, but it comes back if you set a timeout on the route yourself.
Fix: send a heartbeat comment every 15 to 30 seconds. That’s shorter than every idle timer on the path, so none of them fire.
Does the mesh fix it? Yes, mostly. Envoy passes messages through as they arrive.
Buffering usually comes from other places. NGINX-based ingress controllers buffer responses unless the server sends the header X-Accel-Buffering: no. Compression can also hold small messages back until it has enough to compress.
Fix: turn buffering off for the stream, and don’t compress text/event-stream responses.
Does the mesh fix it? No. This one surprised me.
Envoy is smarter than plain Kubernetes about spreading load: it balances every request, instead of every connection. But an SSE connection is one request that never ends. Once it lands on a server, it stays there, mesh or no mesh.
Fix: have the server close each connection after a while, say every 10 minutes, with a little randomness so they don’t all close together. The client reconnects, and that time it may land on the new server.
Does the mesh fix it? Partly.
During a deploy, Istio gives the old server a short grace period, five seconds by default, and then closes its connections. The mesh won’t reconnect for the client, so every client does it on its own. What the mesh does add is protection for the servers: it can limit connections and stop sending traffic to a server that is struggling.
Fix: clients wait a random few seconds before reconnecting, and send the last id they saw so they don’t miss anything.
sequenceDiagram
participant C as App
participant G as Gateway
participant P1 as Old push pod
participant P2 as New push pod
C->>G: open stream
G->>P1: forward
P1-->>C: message 41
Note over P1: deploy starts
P1-->>C: goodbye, stream closes
Note over C: wait a random 0–5 s
C->>G: open stream, last id was 41
G->>P2: forward
P2-->>C: messages 42 and 43, then live
One trap is specific to meshes. In older setups, the sidecar could shut down before the application, cutting connections before the server could say goodbye. Newer Kubernetes versions let sidecars start first and stop last, and Istio can use that.
Does the mesh fix it? No, it makes it a bit worse. Each open connection is now also held by the gateway and the sidecar. It’s small per connection, but worth measuring before you have hundreds of thousands.
Some things come for free with a mesh: encrypted traffic between services without code changes, metrics on how many streams are open and how long they last, and the option to send a new version of the push service only a small share of new connections at first.
| Problem | Does a mesh fix it? | What you do |
|---|---|---|
| Middlemen hang up on quiet lines | No, adds more timers | heartbeat every 15–30 s |
| Middlemen hold messages back | Mostly | buffering off, no compression |
| New servers get no clients | No | close connections every ~10 min, with randomness |
| Everyone calls back at once | Partly | random wait, resume from last id |
| Memory per connection | No, a bit worse | measure it |
For our config system, with around ten changes a day, not yet. Polling cheaply with ETags, or Firestore listeners, gives us enough freshness with far less to run. And adopting a whole service mesh for one push service would be a lot.
If we needed changes to reach every client within a second, I’d build SSE with the fixes above: heartbeats, connections that close every few minutes, message ids for resuming, random waits on reconnect, buffering off, and polling as a fallback. I’d only use a mesh if the cluster already had one. None of the fixes depend on it.
My worry was right about which problems exist. I overestimated how hard they are: each one has a known fix, and most of the fixes are a few lines of server and client code.
Last-Event-ID and comment heartbeatsproxy_buffering and X-Accel-Buffering