Rejecting text after embedding was too late for my budget

Introduction

On November 1, Trend Seeker reached its shared $0.10 daily embedding cap around 01:15 UTC. Podcasts accounted for more than 70% of the recorded tokens. Of the 19,343 podcast evaluations updated that day, 17,410 were rejected after embedding.

The cap was doing its job, but most of the day was still ahead. Rejecting a paragraph after producing its vector could save storage and later processing. The embedding request had already been paid for.

Trend Seeker finds product problems in conversations, reviews, job ads and podcasts. Embeddings help connect that material to related ideas. I wanted to spend more of the limited budget on passages that could contribute evidence, so I added filters before the API call. Then I started training a small local model to make the same decision from examples.

The rules removed a substantial amount of input in an offline audit. The first learned model added much less and missed useful material, so it is running in shadow mode. It records decisions while the existing pipeline continues to process the inputs it would reject.

Move the decision before the paid call

Some podcast text is easy to exclude: an outro, repeated caption fragments, or an explicit sponsor pitch. Other passages need more care. A discussion about a customer struggling with invoices can be useful even when nobody says they want an app.

The first admission rule keeps demand cues and work context, including references to customers, tools, workflows and industries. Other discussion gets a stable 5% exploration sample based on the text. That leaves a route for passages outside the vocabulary to enter the pipeline.

The follow-up added repetition and promotional-text checks, cleaned overlapping captions, and tightened sampling for feeds with enough history to show low useful yield. Concrete demand can still bypass feed sampling. Full transcripts remain in source storage.

These checks have to cover work already waiting in the queue. Filtering only newly produced events would leave the old backlog consuming the same budget. The shared policy runs at the transcript producer, at raw ingestion, and again in the embedding worker before a new paragraph or context request. Cached evaluations retain their existing routing.

Boundary	Why the check belongs there
Producer	Avoid publishing work that the rules already exclude
Raw ingestion	Apply the policy before creating embedding work
Embedding worker	Cover queued records created before the change

A queue can outlive the policy that filled it. The final check before the API call makes the current policy apply to that older work too.

Count the evidence lost alongside the input removed

I audited the follow-up rules against 41,699 podcast evaluations from October 31 and November 1. The audit used the exact Rust admission code and a snapshot of current feed statistics. It admitted 17,387 rows and skipped 56.83% of paragraph input characters.

Of 55 stored matches in the sample, 52 survived. The three excluded matches included two promotional passages and one passage of general product strategy. The rules also excluded some paragraphs that the semantic evaluator had accepted.

The two measures in the figure answer different questions. Character reduction estimates how much paragraph input the rules avoid. Retained matches show a cost of that reduction in evidence the system had already found useful enough to attach to an idea.

Neither is a measured reduction in the embedding bill. Tokenization differs from character counting, and context comparisons have their own cost. Matches can also appear later. I use this audit to judge a proposed filter; actual usage and useful outcomes still need observation after deployment.

Train a local classifier on the decisions we already have

Handwritten rules depend on the vocabulary I thought to include. The next step was to learn from existing decisions about podcast passages.

The first training snapshot contained 10,000 current semantic decisions, including 1,519 reviewed by an LLM. Replaying the existing paragraph and feed filters left 8,239 observations. This model is meant to find additional low-value input after those filters have done their work.

It is a linear text classifier trained on CPU with scikit-learn's SGDClassifier and log_loss, which implements logistic regression. Words, adjacent word pairs and short character sequences become hashed features. The paragraph and preceding context have separate feature prefixes. Python fits the coefficients; Rust computes the live score from the saved weights. The feature extractor is shared between training and inference.

This gives the worker a local decision before a paid embedding request. It does not need another hosted model call to decide whether to make the first one.

The labels need care too. An unmatched paragraph may simply be waiting for the right idea, so unmatched and pending items are never treated as negative examples. Existing matches and evidence used in successfully synthesized or merged idea clusters are protected as positives. Reviewed labels receive more training weight.

Keep related passages out of both training and testing

Paragraphs from the same episode can resemble one another. Identical text can also arrive more than once. Splitting those rows independently would make the test easier than the future work the classifier will see.

The trainer groups parent episodes and identical normalized text before making chronological splits. It also withholds 20% of feed identities from fitting. Thresholds are chosen using validation data, then checked on later observations and unseen feeds. This follows the broader rule of keeping test data out of model and threshold selection, described in scikit-learn's data leakage guidance.

The first audit produced these results:

Cohort	Useful labels retained	Additional paragraph bytes avoided
Validation	318 / 319	1.83%
Later observations	258 / 260	3.97%
Unseen feeds	294 / 298	3.62%

These are incremental results on inputs that survived the existing filters. They cannot be added to the earlier 56.83% figure: that audit counted characters in a different sample, while this one counts bytes after the rules.

The missed passages included a legitimate Shopify operations match. All successful seed evidence survived, but the stronger reviewed and outcome-backed positive subsets were too small to establish the required retention confidence.

I left the held-out labels and threshold unchanged after inspecting the misses. Adjusting them until this test looked good would make the next result harder to trust.

Let the model observe before it can reject

The qualification target is a one-sided 95% confidence lower bound of at least 99% useful-input retention, plus at least 10% additional input reduction. The later-observation and unseen-feed cohorts failed those requirements. The first artifact is marked unqualified.

As the diagram shows, a deployed classifier can collect evidence without controlling admission. A November 3 live check confirmed shadow mode and recorded proposed rejections, with no enforced decisions.

Each decision stores the input hash, model version, score and pipeline stage. Outcomes can later be joined back to those decisions. A new model starts a fresh shadow window. Enforcement requires offline qualification, at least seven days of observation, sufficient live retention evidence and the additional input reduction.

A uniform 5% audit sample remains admitted even if a future model qualifies. Existing podcast exploration samples remain admitted too. The implementation can return to shadow after a retention regression, and a replay command can send rejected inputs back through the embedding pipeline while preserving the deterministic rules.

I now have rule-based filters before the paid call and a local classifier gathering evidence for a possible next step. The first model stays in shadow. Its next version needs to avoid more input while retaining enough useful passages to justify letting it reject anything.