A few months ago, we took an honest look at publications on Medium and asked if they were succeeding. Were users incentivized to create quality publications on Medium?

We had the Boost Nomination Program, which was helpful for the cohort that was selected, but we didn’t have any signal that the BNP was motivating for other users. Were we getting higher quality articles as a result of BNP?

So we asked ourselves “How can we reward good editors for their editorial work and make it clear how others can earn that reward?”

That’s when we came up with the Editor Partner Program (EPP).

The first thing we introduced for EPP was the ability to assign an editor to a given submission. Before, there was no way to tell if a publication editor was actively working on a post. And since we wanted to start compensating editors directly, we needed a way (in the data) to know who worked on a given submission.

So we introduced the ability for editors to assign themselves to submissions in the inbox.

Even for those not in the EPP, this was a useful organizational tool.

I’m not going to go in depth on EPP, but I wanted to flag it as the catalyst for a lot of the recent changes we’ve made and the ones we’re making in the near future.

We knew we wanted to encourage publications to be “open” because we wanted more writers to find a good community of readers. But we knew what being “open” could mean for editors. Lots of slop submissions.

Open submissions doesn’t feel fair … unless we give them something that makes their lives better.— Direct quote from a meeting

Scott, just solve AI slop… done

I wish I had a magic wand to remove all AI slop. We’ve looked into a lot of tools and they just aren’t reliable. They flag human writers as AI a lot.

Some publications are also fine with AI-written articles. Who am I to tell them no?

Rather than throwing up our hands and saying “this is unsolvable,” we thought about mitigation tools. The first was obvious: bulk blocking and bulk declining.

That was something I could write and ship in a day. It was out the door so fast that we thought “what else can we do?” And “how do we decide what to do?”

Workshop on how we might improve editors’ lives

I’m a sucker for a good workshop. Coworkers coming together and throwing ideas at the wall to see what sticks.

We had an hour-long workshop with people from all over the organization. It was important that it wasn’t just engineers. We pulled people from our community team, trust and safety, design, content, and more.

None of us know everything. But we all see the product at different altitudes. And it was important that we had people there who actually talk to users regularly.

Manage and reduce spam

The first thing we focused on was to build on top of the bulk block and remove feature, and focus on other ways to reduce and mitigate slop.

We came out of the workshop with some easy wins. The first was a way to more easily see what submissions were related to your publications’ topics.

We had ways to quickly sift out the spam, but could we introduce more tools to cut down on the spam in the first place?

The next thing we wanted to give editors was the ability to set stricter submission rules for their publication. These would be hard stops that prevent people from submitting until all the boxes are checked. We gave editors tools to (hopefully) spot and remove slop, next let’s give them a tool to keep the slop out of the inbox in the first place.

This also helped writers. Some would have their stories rejected immediately by editors because they didn’t include a subtitle or an image.

Now editors were only getting stories that more closely followed their rules. Of course, that doesn’t mean we removed all spam and it certainly didn’t mean that every submission was high quality (even if well-intentioned).

Continuing down our list of mitigation tools, one point of frustration editors run into is a user who publishes too much. When you see a submission, you only see that post in a vacuum. There’s no way to see if that person publishes constantly (i.e. they are trying to spam).

So we added more to the inbox by giving you a quick glimpse at their recent publishing history:

Writers who earnestly submitted something and got declined kept telling us the same thing: they didn’t know why. Why was my story rejected? Unless the editor left a private note, there was no indication whatsoever.

We introduced preset “reasons” for declining. It made the ecosystem feel less anonymous and gave writers something actionable to work on.

Make running a publication easier

With some of those light, mitigation tools in place, we also wanted to focus on ways we could improve managing a publication (that had nothing to do with removing spam).

We had feedback from editors that poor images and the inability to easily set how images are displayed made their publications look unprofessional.

That’s what inspired us to introduce a tool that easily allows you to set the focal point of an article’s preview image. This is a change that made it to every user, not just editors.

Listening to more feedback, we heard from editors that scheduling could be a pain. It required clicking into each post to make changes, and there wasn’t an easy way to see your schedule other than a list.

So we built out a content calendar. You could see the full schedule for your publication and drag-and-drop items between days to reschedule. No more submenus or guesswork.

We also made it easier for you to edit and manage the writers for your publication. The new “Editors & Writers” page lets you quickly search your list of editors and writers, add new ones, and even download a CSV of everyone who contributes to your publication.

Misses Things we learned

The catch-22 of listening to user feedback is that sometimes you get feedback on the solution to the prior feedback.

We shipped the ability to assign editors, which we used to mark payouts for the editor partner program. But some publications only have 1 editor. A few users left comments saying it didn’t make sense to have to assign themselves, given they were the only editors.

So we shipped “auto assignment” for publications with only 1 editor.

When we showed off the “off topics” flag, most people liked the idea, but the biggest piece of feedback we heard was “5 tags isn’t enough.”

Why could publications only set 5 tags? I dug into it and… no one knows. That decision was made 10 years ago 🤷. Nobody who worked on it still works at Medium, and the ticket linked in the comments was for an old Jira instance I couldn’t access. So we upped it to 10 to give publications more tags to choose from.

The “decline reasons” were a pretty well-received feature. Only problem was, I completely broke submissions for an hour. Oops.

For about an hour, nobody could submit a story to a publication because a migration hadn’t run when I deployed that feature, so no one could write to the submissions table.

It was an easy fix, but it’s the price of moving fast sometimes…

So, what did we actually learn? Sometimes fixes need fixes, and we have to create bandwidth for that. Our users have a great sense of what is working, so we try our best to listen and think through the impact. That could mean changing something we literally just built, and that’s okay.

What’s next?

People want a customizable “decline reason.” i.e. an open text box to describe why something was declined. I do too.

The problem is that as soon as we introduce an open text field, we introduce a vehicle for harassment. So we’d have to build reporting and Trust & Safety tools into it. That turns a small quality-of-life feature into a much bigger project.

We’re going to work on a Publications Directory. A better way for writers to find publications that resonate with their writing. We’re still early in planning, but we want to help people find your publication!

We’re working on editor tools that better help you communicate with authors in the draft. This would let you have a faster back-and-forth with authors and show them the changes you’ve made to their draft before publishing.

Where can you send feedback?

Use Medium’s support form to send feedback or report an issue. For tips on writing a useful ticket, see Contact Medium Support.

Or you can leave a comment here! I swear, I do read them. Just be forewarned, I might use them in a blog post in the future.

What We Built for Medium Editors This Year (and What I Broke) was originally published in Medium Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Context

At Medium, we have many Machine Learning models that we use to label stories automatically. These affect what stories we recommend to readers.

Here’s some examples:

Some Clarifications on our Machine Learning policy

Before we go deep on this project, I just wanted to clarify a few things about how we stand regarding AI in general.

Medium has been training internal models with user and post data for a long time now. We train models with specific tasks. For example, models that power our recommendations algorithm, or text classification models like the ones presented in this story. All in the goal to improve our product. With the LLM approach I describe in this story, we ARE NOT sharing these models with other companies. And we ARE NOT allowing anyone to train on our users’ data and content. Here we used Snowflake LLM tools for inference only (no LLM training was done here) and they are actually hosting all of the models inside their own infrastructure and guarantee that they are not using any of this for training. Shoutout to the Snowflake team for making it so easy and safe to use LLMs on our data!

If you want to read more about Medium’s stance on AI, I definitely recommend giving these a read:

• Default No to AI Training on Your Stories
• Finally, an internet standard for writers’ rights vs. AI companies
• We want your feedback: How can writers use AI to tell human stories?

Problem

During our roadmap planning we decided that our NSFW model was out of date and it was time to revamp it. This model labels stories as “Not Safe for Work” if they have sexually explicit content, lots of profanity, or basically anything you wouldn’t want to read on your big monitor in the middle of an open space!

As you can imagine it’s a pretty important model. We really need it to make sure our most “interesting” content only reaches our most “interested” users and ONLY them!

There’s actually a funny anecdote from 2021 that was shared internally back then. One of our employee was onboarding the WHITE HOUSE staff onto Medium so that they could start using the platform with the POTUS account. And the first thing that showed up on the homepage was a big picture of well… a butt… So that became of whole thing “Our algorithm is serving erotica to President Joe Biden!!”. That was a brand new account so that was our top story “by default”.

That’s embarrassing for us and for our users! This anecdote is actually what prompted the recommendations team to create the first version of our NSFW model (so, thank you Joe!).

We’re now in 2026 and this model is now pretty old. It’s not getting retrained regularly, the code runs on python versions that are extremely old. With turnover and changes in tools and file organization, we lost track of how it was trained and what kind of performance was measured at the time. So this is now an obscure part of the ML stack, in need of a good makeover.

LLM based approach

The typical way we’d build a binary classification model like this is to build a pipeline where:

• new data is regularly added to a training set by a human labeling team
• and the model is regularly retrained

So now, what if we replace all that with a really simple LLM based approach?

With LLMs it’s pretty easy to create whatever text classifier you want. You just need the proper prompt and have some sort of expected output that you can parse from the LLM’s response. And there’s no need for training at all.

We thought this NSFW model revamp was a great opportunity to give this a try. Are we able to build a NSFW classifier this way, that matches our performance expectations?

How we built it

The process was pretty simple

First we built an evaluation dataset:

• we sampled a random sample of stories published on Medium
• then we had our curation team label the dataset (is it NSFW? yes/no)
• as they did that, we also asked them to refine their definition of “NSFW”, and give concrete examples (this was very useful to build the prompt)

Side note: when we picked the random sampling of stories, we used other signals to make sure we oversample NSFW stories. If we just sample at random, we’d have to sample thousands of stories just to get a handful of NSFW examples. Thanks to this, our evaluation had a nice 50/50 split on NSFW/SFW labels.

Next step was to build the prompt for the LLM:

• we leveraged all the clarifications that the curators added while labeling the dataset and summarized that into “NSFW guidelines”
• we pass in the story’s contents
• we ask the LLM to output a score from 0 → 100
• and we also ask the LLM to give a short explanation for the score. For example: “graphic description of sexual intercourse”

Finally we need to evaluate the LLM approach and compare the different model:

• have a few LLMs make their predictions on the evaluation set
• have the legacy model make it’s predictions on the evaluation set too
• and then compare the metrics

Great news, after some testing, we found some LLM models that match or outperform the legacy model! We got parity with the legacy model on both false positives and false negatives. Some of the bigger LLMs even outperform the legacy model on both metrics. The mistral models performed really well for us, as well as some of the larger Claude models.

Performance is key here because there’s big downsides with both false negatives and false positives:

Now, we need to get an idea of the costs: is that LLM approach going to cost us an arm and leg?

Something nice with LLMs is that the costs are easy to estimate. Each model has a cost per input token and a cost per output token. Using that and a little bit of back of the enveloppe math you can quickly get a really good cost estimate.

We then picked the model that was the best fit for us in terms of costs and performance. For us that was Mixtral 8×7b (I promise that was a fair trial - nothing to do with the fact that I’m French!). The main way we kept the costs under control is by picking a relatively cheap model, and also by only scoring the stories that are eligible to get distributed in the first place (for example, no need to waste time on stories that have been flagged as spam).

For offline experimentation and for production classification, we simply used Snowflake. All of our data ends up in Snowflake and they make it super simple to run LLM inference directly inside a SQL query. There’s a complete function available that lets you call an LLM and pick your model as well as the expected response format.

Here’s the query I used during the evaluation process as an example:

set modelName = 'claude-haiku-4-5';
set promptName = 'raph_test';
set promptVersion = '1.3';

set responseFormat = '{
  "type": "json",
  "schema": {
    "type": "object",
    "properties": {
      "score":  {"type": "integer"},
      "reason": {"type": "string"}
    },
    "required": ["score", "reason"]
  }
}';

with raw_llm_responses AS (SELECT post_id,
                                  title,
                                  text,
                                  SNOWFLAKE.CORTEX.COMPLETE(
                                    $modelName,
                                    -- build prompt: instructions and post information
                                    ARRAY_CONSTRUCT(
                                      OBJECT_CONSTRUCT(
                                        'role',
                                        'system',
                                        'content',
                                        prompts.prompt
                                      ),
                                      OBJECT_CONSTRUCT(
                                        'role',
                                        'user',
                                        'content',
                                        '<title>' || COALESCE(title, 'N/A') || '</title>' || '\n' ||
                                        '<text>' || COALESCE(text, 'N/A') || '</text>'
                                      )
                                    ),
                                    OBJECT_CONSTRUCT(
                                      'temperature', 0,
                                      'response_format', PARSE_JSON($responseFormat)
                                    )
                                  ) AS llm_raw_response
                           FROM posts_to_evaluate
                                  join medium.ml.nsfw_classifier_prompt as prompts
                           where prompts.name = $promptName
                             and prompts.version = $promptVersion),

     -- parse results from the returned JSON
     llm_responses_parsed AS (SELECT post_id,
                                     title,
                                     text,
                                     llm_raw_response,
                                     llm_raw_response:structured_output[0]:raw_message AS parsed_json,
                                     TRY_CAST(parsed_json:score::STRING AS INT)        AS score,
                                     parsed_json:reason::STRING                        AS score_reason,
                                     parsed_json::STRING                               AS response_as_string
                              FROM raw_llm_responses)

SELECT post_id,
       score              AS prediction,
       score_reason,
       response_as_string as raw_response,
       $modelName         AS model_name,
       $promptName        AS prompt_name,
       $promptVersion     AS prompt_version
FROM llm_responses_parsed;

In our tech stack, it’s really easy to setup a simple job that runs a Snowflake query on a schedule. We just leveraged that and we were able to successfully ship that model very quickly.

And now repeat?

Once that new model was up and running, we were really interested in applying that same formula to our other models. We first sat down to weigh the pros and cons of that new LLM based approach, here’s how we broke it down:

Generally the LLM approach is:

• easier to setup
• more expensive
• less adaptive

One thing to keep in mind too is that LLMs are “stuck at a point in time”. They have been trained at a certain date and they don’t know anything about what happened after that date. So they will not be adapting to new trends and performance is likely to drop over time. Depending on the use case, you might want to upgrade your model every now and then.

The NSFW model was the perfect use case:

• we don’t have a proper training set for NSFW content. And it would be costly to create and maintain one
• we don’t feel like we need to be super reactive to new trends. I don’t really have data to support that, this is more a judgment call: we think that NSFW stories 6 months from now will be similar to the ones that are published today.
• costs are low as long as we only score a portion of the stories published on Medium. Here we just need to score the stories that are eligible for distribution in the first place

All in all, that model revamp was a great success:

• we have a brand new NSFW model
• it performs really well
• we don’t need to maintain a training set
• it’s cheap to run
• and it’s really simple in terms of engineering

What about our other models then?

It turns out that the NSFW model is maybe our only model that’s a good use case for an LLM based approach:

• for our topic model, we get our dataset for free (we just train on the topics that users are adding to their stories). This means that we automatically adapt to new trends, which is really nice. Switching to an LLM based approach would feel like a downgrade

• for the spam model, we also want to adapt to new trends quickly. Spammers are always trying new things and they are always trying to trick the platforms to get visibility. So in this case we are willing to pay the cost of maintaining a training set, augmented with new data every day. Also since we want to score ALL the stories posted on Medium, it becomes more complex and costly to use an LLM based approach

What about new use cases?

Even though that doesn’t work really well for our other existing models. It is still very exciting because it makes it really easy to spin up new text classification models. What if we wanted to give users more control on their content, like “allow erotica” but do not allow “true crime”? We’re a text-based platform, so the possibilities are endless!

Some side notes

Why ask for scores and not just a yes/no value?

With scores, you are able to rank the predictions and it means that you can use metrics that quantify how well you separate the NSFW stuff from the SFW stuff (metrics like ROC-AUC or PR-AUC). It also means that you can set the threshold wherever you like and you can change that in the future too - letting you choose the balance of false positives and false negatives that suits you best.

Why ask for a short description?

Our end goal is just to know if a story is SFW or not, we don’t actually need the LLM to explain it’s decision. And it does add a little to the inference costs (the output becomes longer). But in our use-case it was negligible in the overall costs.

I had to do a lot of spot checking and debugging on individual stories for this project. (I learned a lot of things 😳). And having the descriptions alongside the score was super useful for that. It’s also a great thing to have now that this is production, and it can also be used to break down NSFW stories into categories. That can be useful for the curation team or the trust and safety team

To wrap this up, it was really fun to build this model and we’re excited to build new features with this new tool in our toolbox! We’re hoping this opens up new possibilities for where our product teams can take Medium next and helps us make sure readers find more stories they love, and writers get more readers that love their work!

How we think about text classification in the LLM era was originally published in Medium Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

For years, there’s been this really popular Chrome extension for adding a table of contents to your Medium stories. Props to Vinicius De Antoni for writing a popular chrome extension all those years go.

The extension was so popular, we even wrote about it in our Medium Handbook.

How to make a table of contents for your Medium story

The fact that it was so popular was a signal to me. There was a decent size of our user base that wanted this feature. So, at first I set out to just copy it.

The Chrome extension extends the popup menu we have in the editor to add a new option. Just a styled button on the existing list.

const inject = () => {
  const tooltipToggleMenu = document.querySelector(
    "[data-action=inline-menu]",
  );
  const tooltip = document.querySelector(".inlineTooltip");
  const tooltipMenu = document.querySelector(".inlineTooltip-menu");
  const addEmbedButton = document.querySelector(
    ".inlineTooltip-menu [title='Add an embed']",
  );

  if (!tooltip || !tooltipToggleMenu || !tooltipMenu || !addEmbedButton) {
    return;
  }

  tooltip.style.width = "auto";

  const toCButton = document.querySelector("[data-action='inline-menu-toc']");
  if (toCButton) {
    return;
  }

  const handleAddToCClick = (e) => {
    const container = document.querySelector(".is-selected");
    tooltipToggleMenu.click();

    container.innerHTML = generate().join("<br/>");

    setTimeout(() => {
      simulateKeydown(container, 13);
    }, 0);
  };

  const addToCButton = addEmbedButton.cloneNode(true);

  const title = "Add a Table of Contents";
  addToCButton.setAttribute("title", title);
  addToCButton.setAttribute("aria-label", title);
  addToCButton.setAttribute("data-action", "inline-menu-toc");
  addToCButton.setAttribute(
    "data-action-value",
    "Generate a table of contents",
  );
  addToCButton.setAttribute("data-default-value", "Table of contents");
  addToCButton.innerHTML = "⋮";
  addToCButton.style.color = "rgb(26, 137, 23)";
  addToCButton.style.border = "1px solid";
  addToCButton.style.fontWeight = "bold";
  addToCButton.addEventListener("click", handleAddToCClick);

  tooltipMenu.appendChild(addToCButton);
};

(Code taken from the open source Chrome extension)

The button has an event handler that runs some JavaScript to look at the content of the post, find all the headings and creates an unordered list of links for each.

const generate = () => {
  const headingList = [...document.querySelectorAll("h3[name], h4[name]")];

  return headingList
    .filter(
      (headingTag) =>
        !headingTag.classList.contains("graf--title") &&
        !headingTag.classList.contains("graf--subtitle"),
    )
    .map((headingTag) => {
      let bullet;
      switch (headingTag.tagName.toLowerCase()) {
        case "h1":
        case "h2":
        case "h3":
          bullet = `·`;
          break;
        case "h4":
          bullet = `  ∘`;
          break;
      }

      return `${bullet} <a href="#${headingTag.getAttribute(
        "name",
      )}" title="${headingTag.textContent}">${headingTag.textContent}</a>`;
    });
};

(Code taken from the open source Chrome extension)

It inserts that list into the body of the post itself.

For me, this is the first downside. Mutating the post directly feels… not great. It doesn’t play well with our revision model and from writer’s perspective, adding stuff into the post I spent so much time writing can skew the reading experience.

Still, most people don’t mind the look and feel of it. The real issue, however, is that it isn’t dynamic. You click the button and slaps that list in your post. But what if you revise your article later? Add a new section with a new header and suddenly that table of contents isn’t accurate.

Then there was the usability of it. It was always inserted at the top of the post. Which meant that once you used it to jump to the bottom section, you’d have to scroll back to the top to use it again.

The list also just jumped you straight to the section. That’s how anchor links work by default, but it could feel abrupt if you didn’t quite know what was happening.

I thought we could do better. A dynamic solution that writers don’t have to maintain.

The principals are still the same though: the list should be built off of content of the post. That’s our source of truth. If the author revises it, they shouldn’t have to worry about maintaining some index of all their headings.

So, we build the table of contents based on the post’s body at render time. Why at render time? Well, if you didn’t know our post model is… complicated. We don’t store the full, finished post in our backend. We store a series of revisions that we play over one another to land on the final version. The end user doesn’t notice the difference, but it allows authors to step back through different versions of their article.

Rather than trying to traverse this complicated model, it was best to just honor the end result of the post and create a table of contents at run time.

Much like that Chrome Extension, we build a list based off the headings.

/**
 * Derives a nested table of contents from a post body model.
 *
 * In Medium's model, in-body H2 and H3 both render as the larger section heading
 * while H4 renders as the smaller sub-heading. So H2/H3 become top-level entries
 * and H4 sub-headings nest beneath the most recent top-level heading. The post's
 * lead title/subtitle/kicker are excluded because getParagraphStyles remaps them
 * to 'Title'/'Subtitle'/'Kicker'. An H4 that appears before any top-level heading
 * is promoted to top level so nothing is dropped.
 */
export function buildTableOfContents(bodyModel: TocBodyModel | null | undefined): TocEntry[] {
  if (!bodyModel) return []
  const styles = getParagraphStyles(bodyModel as Parameters<typeof getParagraphStyles>[0])
  const entries: TocEntry[] = []
  let currentTopLevel: TocEntry | undefined

  bodyModel.paragraphs.forEach((paragraph, index) => {
    const style = styles[index]
    const name = paragraph.name?.trim()
    const text = paragraph.text?.trim()
    if (!name || !text) return

    if (style === ParagraphType.H2 || style === ParagraphType.H3) {
      const entry: TocEntry = {name, text, children: []}
      entries.push(entry)
      currentTopLevel = entry
    } else if (style === ParagraphType.H4) {
      if (currentTopLevel) {
        currentTopLevel.children.push({name, text, children: []})
      } else {
        entries.push({name, text, children: []})
      }
    }
  })

  return entries
}

Then, it’s a matter of how we display it. My first attempt was to just add it to the 3-dot-menu at the top of the post:

I’m not a designer and I don’t have the best UX sense, but even I could tell this wasn’t quite right. As some people pointed out, the biggest downside was that you couldn’t access the Table of Contents from anywhere. You had to be at the top of the post.

Luckily, I work with some great designers and we were able to noodle out some ideas. There’s a lot of prior art on the internet for breaking up sections on a page. Look at Notion, Codex, Graphite, and many other apps.

Rather than adding another item to that popover menu, what if we had something that lived in the right margin and scrolled with you?

Better yet, what if it could tell you where you were within the body of the text?

Our next solution did just that. Long lines for main headings, short lines for subheadings. The “current” heading would highlight in green and update as you scrolled.

Edge cases

Good enough to ship, right?

Well, almost. The more we played around with it on production, the more edge cases we could see.

We had to account for images that took up the whole width of the page. We did that by adding a translucent background. This way, it’s always visible without being overly abrupt.

What about posts that have only one heading? Well, we just hide it entirely. There’s kind of no point.

What about posts that have one hundred headings? When we tested it, the table of contents went right off the page! Not a typical case, but there are plenty of long papers published on Medium, and they are arguably the ones that could benefit the most from a table of contents.

So we went with a sliding window that only shows a few at a time. Hover over it and you’ll see the full list.

Thanks Marco Capriz for writing this post that was great for testing the table of contents!

How to Build a Dyson Sphere

At this point, we felt good about the desktop solution. How did it feel on mobile?

Pretty bad…

We have a much smaller right margin on mobile, so the indicators were overlapping with the post content. We could squish them down, but then they were barely visible and the touch box was too small.

How do other apps handle it? Notion completely hides their table of contents on mobile. Most likely, they ran into the same issue.

We decided that mobile table of contents needed to have their own solution. What makes sense for a desktop user doesn’t always work for someone on their phone. More to come on that.

Overall, I think we landed on a great solution.

• Dynamic list users don’t have to maintain
• Clickable from anywhere in the post
• Smooth scrolling
• It doesn’t add any content to the post itself

Unlike the Chrome Extension, we can also add metrics to this, so we here at Medium can see if users are actually using the feature. The old Chrome Extension required writers to create and maintain these lists, but they are ultimately a tool for readers not writers.

Will this hurt my earnings?

Before we shipped, I had the question of “will this hurt scrolling metrics that affect reading time?”

The short answer is: no. We don’t calculate valid reads based on scrolling behavior. We might have in the past, but that was before my time 🤷. Plus, there was nothing stopping a user from quickly scrolling a post before, we just allowed them to do it a more usable way.

So don’t worry about this feature affecting earnings. We weight earnings based on engagement (claps, highlights, reading time). If a user uses the table of contents to quickly scroll your stuff and then bounce, they didn’t actually read anything and they probably wouldn’t have read it without this tool.

Overall, we think quality of life changes like this help with user retention. Which means more readers stick around because they have a more positive experience.

How long did it take to build?

All-in-all, this feature took a couple of days to finish. A lot of that was UX testing and iterating on the design to make it feel right. The code is pretty straightforward once we landed on a solution.

Expect us to ship more quality of life improvements like this in the near future!

How we built the new Table of Contents feature was originally published in Medium Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Desired outcomes (what “good” looks like)

🙏🏻 These are the kinds of changes we want to see if AI is going well at Medium.

Less drudgery, more leverage

Engineers use AI to remove repetitive, low‑leverage work (e.g. tedious refactors, boilerplate, large test suites), not just to ship more tickets.

We see examples like:

• “We took a part of the codebase that was basically untestable and, with AI, made it testable and added meaningful coverage in days instead of months.”
• “We cleaned up a legacy area we had been afraid to touch for years, using AI for the mechanical work.”

Faster learning and broader technical range

Engineers use AI to ramp on new languages, frameworks, and parts of the stack much more quickly.

Engineers can explore unfamiliar areas safely by asking AI to explain tradeoffs, pitfalls, risks, and how the system uses this areas of the code.

We see fewer cases where work stalls or we avoid implementing things because “nobody knows this code anymore”.

Frontend/mobile engineers are able to make safe, reviewable backend changes with AI’s help (plus a review from a backend engineer).

Teams can ship more end‑to‑end features without long handoffs.

Higher technical ambition, not just more speed

Teams take on work that used to be “too expensive”:

• Meaningful test coverage in critical areas.
• Pieces of M2 migration or other large refactors.
• Structural cost savings.

Incident/action items evolve from “add an alert” to “Let’s improve tests here”.

Engineers use AI pre-work to explore edge cases and tradeoffs (design rigor).

Better planning, reasoning, and risk reduction:

• Engineers use AI to think through complex scenarios before building, exploring edge cases, stress-testing assumptions, and comparing multiple approaches, and better understand the pros and cons of each approach
• Design docs become more rigorous as AI is used to challenge thinking and look for gaps, risks, and alternate approaches that the team might not be aware of.

AI fluency as mindset, not just tool usage

Engineers can explain:

• When they choose a heavier model vs a cheaper one, and why.
• How they use AI to reason about a system they don’t know (finding root causes, understanding hidden costs).
• Where they don’t trust AI (judgment calls, ambiguous product decisions, sensitive code paths).

In reviews, incidents, and 1:1s, people naturally talk about how AI helped them reduce risk or effort, not just “AI wrote this file.”

Better quality and maintainability

Code quality is improved through AI-assisted code reviews, security audits, and other code analysis tooling.

Engineers use AI to identify anti-patterns in the code, performance and cost bottlenecks, and security gaps that would otherwise be difficult to find by hand.

The code in areas we touch becomes clearer and more consistent over time, not messier.

Engineers use AI to apply established coding and architecture patterns, so touched code stays consistent and easy to navigate and reason about, regardless of which model or harness is used.

Engineers use AI to simplify and standardize, especially in refactors and tests, and avoid clever one-off patterns.

Outcomes to avoid (what we do not want)

⛔ These are anti‑goals and bad incentives we should explicitly avoid in performance, ladders, and messaging.

Velocity metrics for AI

No performance metrics as success metrics for AI adoption like:

• Number of PRs/commits.
• Lines of code added/changed.
• Tokens used or dollars spent on models.

*It’s okay to use them as debugging signals to identify if folks need support, are overloaded, or to ensure onboarding is going smoothly.

No framing like “burn $X of tokens per month or you’re not using AI enough.”

We don’t want engineers to optimize for “more output” at the expense of quality, safety, and thoughtful judgment.

Replacing human judgment with blind trust

Engineers should not feel pressured to:

• Ship AI‑generated code they don’t understand.
• Skip thinking about architecture, correctness, or ethics because “AI said so.”

For high‑risk or complex systems, we should increase the expectation of human understanding, even if AI writes the first draft.

We want to avoid the world where nobody can be held accountable for the code they committed because “AI wrote it”. Engineers are still accountable for their code clarity, code correctness, test coverage, and ability to reason about the code. The human judgment of engineers remains what matters most.

Drowning in code reviews

We want to avoid the case where teams are just cranking out an untenable amount of code, so much so that humans can’t reasonably keep up with reviewing them (even with AI assistance).

Eroding codebase understanding

We want to avoid:

• Critical areas where nobody can explain how things work because “AI wrote it.”
• Removing code review and shared understanding entirely, especially on core systems.

The goal is to move people from “I write every line myself” to “I orchestrate the system and can explain it” and not to “I don’t know what this does.”

Security and privacy regressions

We want to avoid cases where code is quickly written, without realizing what security or privacy implications it might cause. It’s easy to ship stuff with insane security gaps when there is no human looking at it.

Ignoring ethical and environmental concerns

We should not:

• Force teams to use vendors they see as misaligned with our values.
• Hand‑wave away questions about environmental impact and model energy usage.

Where possible, we should prefer vendors whose values align better with ours and be explicit about that.

Tool theater and cargo-cult adoption

Engineers should not feel pressured to mention AI in updates to look modern, even when it didn’t help.

Teams should not adopt a tool because leadership is excited about it, only when it actually improves results, quality, speed of learning, or safety.

Since this was published on February 20, we’ve rallied around these as outcomes we want to see from our AI adoption. There’s a lot of hype around AI. Our north star related to AI adoption is that we will see outcomes from our inputs, and that the humans at Medium can move with the speed and responsiveness of a much larger team, while staying true to what makes the platform worth building.

Brittany Jezouit (Chief of Staff, Medium) wrote and published a related story about how Medium uses AI internally.

Outcomes we want to see from AI at Medium Engineering was originally published in Medium Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

I know what you’re thinking.

Scott, I landed on Medium on a small but not mobile-small display and the download chunk related to images in my home feed was like 7% smaller.

First of all, thank you for noticing.

Second, what if I told you that you too can improve your total asset download size by adding just four characters to your code base?

Sounds too good to be true, right? Well, I’m here to tell you, it’s true… after I explain how we got here.

The history of images on the web

As a web developer, you’ve probably spent a lot of time thinking about and fighting with images.

According to the HTTP Archive, even in today’s modern landscape of “JavaScript everywhere” the majority of most website’s transfer size is images. We’ve moved away from the “hero image on every page” ethos, but still we’re an image-heavy internet.

When I was getting my start in web development in 2010ish, we were still adapting to a “mobile-friendly” world. The majority of the web was built on WordPress.

We didn’t have a way back then to dynamically load an optimized image. 3G connections were the primary speed in the US for viewing the internet, so the practice was images “need to work on mobile” regardless of the user’s device.

For those of us writing Angular and Ember (because React wasn’t a thing yet), you had to legitimately render the correct src based on context you could get from JavaScript.

{{#if browserSmal}}
  <img src=”image-high-res.jpg” />
{{else}}
  <img src=”image-med-res.jpg” />
{{/if}}

This is back when we wrote handlebars for our markup and the web was filling up with SPAs with no server-side-rendering.

We did it this way because HTML lacked a way for specifying how we wanted to optimize images.

Then in 2012, RICG introduced 2 new specs for image optimization, with 2 different use cases: <picture> elements and srcset.

Picture Elements

I love the <picture> element. It gives us a lot of control on how we want to display an image based on the information we know with a smart fallback if compatibility fails.

<picture>
  <source srcset=”logo-dark.png” media=”(prefers-color-scheme: dark)” />
  <source srcset=”logo-light.png” media=”(prefers-color-scheme: light)” />
  <img src=”logo-light.png” alt=”Product logo” />
</picture>

In this example, we can specify which image to display if the user prefers dark mode or light mode. If their browser doesn’t support that check, we specify our default.

We can add logic for screen dimensions, file types (i.e. support for avif or webp), and more. This can be really useful when we want to display different images based on this information.

For example, if we want a landscape banner image on desktop but on mobile, we want a narrow portrait. This isn’t “the same image at different resolutions” but “different images based on our layout.”

<picture> is great when we want a lot of control and have opinions on an image.

Srcset

While the <picture> element was a prescriptive syntax (lets us tell the browser what to do), srcset and is descriptive. We provide the browser with lots of different dimensions of an identical image (ideally) and it decides when to render them.

This new element/attribute is for providing more control, versus making images faster (it can, but most devs find using it that way to be overkill given you have to define every breakpoint).

As web developers, we get no say in what the browser decides here. The algorithms used are intentionally opaque as part of the spec:

In an implementation-defined manner, choose one image source from sourceSet. Let selectedSource be this choice.

— HTML Standards, 4.8.4.3.7 Selecting an image source

Let’s take a quick aside and talk about why the browser owns this implementation because I think it’s important and interesting.

First of all, as JavaScript developers, we don’t have any meaningful tooling around connection speed. A browser might have that level of clarity (it varies from browser to browser and OS to OS). So already, we can’t necessarily make the right call for which image to use based on that speed.

But even if we could we’d be making a vague judgement on what we prefer and not what the user prefers (and not all 3G connections are the same). Then there are additional things the browser knows, like which images have already been cached. If we have the 2x version of an image cached, but our page’s srcset requests the 1x version based on this new layout, which is faster to load? The cached 2x or the smaller 1x that requires a new request?

So the spec says “let the browser handle this.” It has a lot more context than we do and it would be really annoying to write out crazy conditions like srcset=”foo.jpg 480w unless cache is available?, foo.jpg 720w unless bigger cache available?”

So we provide the sizes we ideally want and the browser takes care of selecting which one to render based on context. Great. Let the computer do computer things.

What’s the problem then? Have we handled every optimization case possible?

Not really…

Sizes

In addition to srcset we also got the size attribute in 2012ish. It let us define our expected dimension for given images.

When I first read a blog post about this, it made no sense to me. “Isn’t that what srcset does??”

Again, I’m going to take a step back and look at the problem. When our browser receives HTML from our server, it looks at the img tags and starts fetching those images before the layout is rendered. That’s a really important detail. That means there is no reasonable way that the browser can know how much space an image will take up until after it’s been fetched.

It begins downloading assets, then it paints, renders, and inserts those assets. If it waited to download until after rendering, you’d get empty images and risk layout shift.

So, to help the browser make an even more informed decision, you can specify sizes on your images: sizes=”(width <= 600px) 480px, 800px”

Conceptually, it’s so simple. It makes sense. But when you start getting into practical usages (where you are trying to optimize for lots of scenarios) you start writing these out-of-control syntaxes. Mat Marquis, who helped write the spec for srcset and sizes puts it best:

Describing the sizes of a flexible image will require far too much calculation across breakpoints. (min-width: 1340px) 257px, (min-width: 1040px) calc(24.64vw — 68px), (min-width: 360px) calc(28.64vw — 17px), 80px is an example from a relatively simple layout, and there’s no way anyone could be expected to write this. I mean, how — from, what, resizing your browser and squinting? Guessing? sizes is one of the few markup patterns that all but require the use of tooling, which the furthest possibly cry from the web’s “open any text editor and you can build a website” ethos

— Mat Marquis, from “The End of responsive images”

And that’s just for images we know about? What about user avatars? Blog post images? Random assets we have in sidebars?

Just go to the Medium home feed and see how many images load there.

Almost 70 images, and we have a relatively light image-to-text ratio compared to most websites. (We’re a reading and writing company, after all.)

How can we help the browser make the best decision for what image to load? Well, the solution for that has partially been around for a while: loading=”lazy”

Most images on your website aren’t necessary for first paint. Many of them aren’t even in the viewport at first. For those, we can skip the “fetch asset before rendering” step by saying “this image should only be loaded when we’re confident the user will need it.”

For lazy loaded images, the browser should already have the information it needs for the layout. It should know the optimal version of the image it needs, right?

Well, until a few weeks ago, we had no way of communicating that. Before, we had to write endless breakpoint rules with endless image variants.

But on March 24th, 2026, we got support in WebKit (a similar one to Gecko which powers Firefox) for adding support of “auto” to sizes on lazy-loaded images.

Auto is our motto

Essentially, this says to the browser: “For this image, only load it when we need it and based on the layout, connectivity, dimensions, etc., load the most optimized version.”

We can once again let the browser handle all the heavy lifting of “load the most optimized” image and we actually get to write less code than before.

The best part? The existing sizes spec tells it to ignore any rule it can’t parse. So you can add “auto” to the beginning of all your lazy loaded images, and if the user has a browser that doesn’t yet support it, the rule will just be ignored.

This is a rule that is so performant and backwards compatible that WordPress (remember them?) already updated every instance of lazy loaded images in their codebase to support this.

So if you want an easy-to-implement win that has zero risk of breaking your website, slap an “auto” onto your sizes attributes for lazy loaded images.

That’s how I improved our image download sizes with just four characters. Okay, technically it was 6 characters since I had to add a comma and a space… but still!

How we improved image download sizes on Medium with just four characters was originally published in Medium Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Turning knowledge into reusable AI agent instructions for a small, fast-moving team.

We're a small Android team at Medium, just a handful of engineers maintaining and evolving the Medium Android app. Our codebase follows Clean Architecture with Kotlin, Jetpack Compose, Hilt, Apollo GraphQL, and a growing number of feature modules. Like most Android teams, we have strong opinions about how code should be structured: where ViewModels get their data, how analytics events flow, how feature flags are checked, what a "new screen" looks like from Fragment to preview function.

The problem? Those opinions lived in PR review comments, Slack threads, and the heads of engineers who'd been around long enough to know the patterns. When AI coding assistants arrived, they could generate Kotlin code but not our Kotlin code. The output was generic. It missed our conventions, our component library, our testing style.

Six months ago we started using Cursor as our companion IDE. What changed the game wasn't Cursor itself, it was skills and AGENTS.md: a way to encode our team's playbook so the AI follows it every time.

This post walks through what we built, how we structured it, and what impact it's had.

The Foundation: AGENTS.md as Project Context

Before skills, we wrote an AGENTS.md file at the root of our Android project. Think of it as a README for the AI, a document that's automatically loaded into context whenever any Agents works on our code.

Our AGENTS.md covers:

• Architecture overview: Module structure (data, domain, design, feature modules), layer responsibilities
• Key patterns: How we do dependency injection (Hilt), state management (StateFlow + SharedFlow), navigation (centralized Router), repository pattern (Apollo + Result<T>)
• Conventions: Compose best practices, ViewModel patterns, testing strategy
• Common commands: Gradle tasks for building, testing, and running Detekt

This gives Agent baseline awareness of our project. When it generates a ViewModel, it already knows to use @HiltViewModel, StateFlow, and @Immutable sealed interfaces. When it creates a test, it reaches for MockK and Turbine instead of Mockito and LiveData.

But AGENTS.md is passive context. For multi-step, opinionated workflows, we needed something more structured.

Skills: Step-by-Step Playbooks for the AI

An Agent skill is a Markdown file (stored in .agents/skills/) that teaches the AI a specific workflow. It's not a template, it's closer to a runbook: "here are the files involved, here's the order of operations, here are the patterns to follow, here's the checklist to verify."

We've built 13 skills so far. They fall into four categories.

Scaffolding Skills - "Create This From Scratch"

These are the highest-leverage skills. They replace the 30-60 minutes an engineer spends setting up boilerplate for a new screen, module, or layer component.

create-compose-screen: Our most detailed skill. It walks through creating a ViewModel with assisted injection, listener interfaces (in separate files), a Screen composable with a @VisibleForTesting overload, previews for every state, and test tags. A single prompt like "create a new screen for user notifications" produces 6-8 files that follow our exact patterns.

The skill specifies structure like two versions of every screen composable:

// ViewModel-injecting version
@Composable
internal fun MyFeatureScreen(
    itemId: String,
    referrerSource: String,
    listener: MyFeatureListener,
    viewModel: MyFeatureViewModel = hiltViewModel { factory: MyFeatureViewModel.Factory ->
        factory.create(itemId = itemId, referrerSource = referrerSource)
    },
)

// @VisibleForTesting version (for previews and tests - no ViewModel dependency)
@VisibleForTesting
@Composable
internal fun MyFeatureScreen(
    viewState: MyFeatureViewModel.ViewState,
    dialogState: MyFeatureViewModel.DialogState?,
    snackbarHostState: SnackbarHostState,
    listener: MyFeatureInternalListener,
)

Without the skill, the AI consistently generates a single composable tightly coupled to the ViewModel, which makes previews and UI tests painful.

create-feature-module: Handles directory structure, build.gradle.kts with the correct plugins and base script, settings.gradle.kts registration, and app-level dependency wiring.

create-use-case and create-repository Enforce our Clean Architecture layers. Use cases always use operator fun invoke(), return Result<T>, log with our Logger, and track analytics on success. Repositories use @Singleton, safeExecuteNotNull (our Apollo wrapper), and support FetchPolicy.

Migration Skills - "Modernize This Code"

We're in the middle of two long-running migrations, and skills let the AI do the mechanical work.

material3-migration: Contains an exhaustive mapping table of Material 2 to Material 3 component replacements (60+ components). It covers scaffold changes, LocalMinimumInteractiveComponentSize, theme references, and the subtle naming conventions in our design system (MediumScaffold becomes MediumScaffold3, imports shift from component to component3). Without this skill, the AI would have no way to know that MediumPullRefreshIndicator becomes MediumPullToRefreshBox that's not in any public documentation.

compose-viewmodel-migration: Guides migrating screens from the old pattern (Fragment creates ViewModel, passes streams to composable) to the new pattern (composable creates ViewModel via hiltViewModel with assisted injection). It covers the BundleInfo pattern, listener splitting, and the @VisibleForTesting overload.

Pattern Enforcement Skills - "Do It the Right Way"

Some patterns are subtle enough that even experienced engineers occasionally get them wrong. These skills exist to prevent specific categories of bugs and review comments.

viewmodel-flags-usage: Our most opinionated skill. Feature flags must be checked once at ViewModel initialization and saved as a private val. The result is passed through ViewState as a boolean. Never check flags in composables. Never recompute in a Flow.

// Check once at init, save - screen won't change during use
private val isAddressBookEnabled: Boolean = flags.isEnabled(Flag.ENABLE_ADDRESS_BOOK)

source-referrer-tracking: Defines the chain: a screen's source becomes the next screen's referrerSource. The skill explains SourceParameter serialization, the convention that source should be the last parameter in ViewState data classes, and the anti-pattern of accidentally passing referrerSource forward instead of source.

implement-analytics-event: Covers the full lifecycle: proto registration in Wire config, tracker interface in core, default implementation in app, Hilt binding, SourceNames constants, and the reportScreenViewed() pattern with deduplication.

Workflow Skills - "Handle This Repetitive Task"

add-deeplink: Our deeplink handler is a 1000+ line first-match-wins dispatcher. The skill explains the ordering rules (narrow before wide, fragment matches before path matches), provides five patterns (simple path, path + fragment + auth, regex-based dynamic segments), and specifies the test template including both logged-in and logged-out variants.

add-medium-uri: Three files must be updated in a specific order (interface, NoOp, default implementation) with naming conventions that vary by URL domain. Small task, but easy to get wrong without the skill.

check-and-add-translations: Finds missing translations across all modules by diffing values/ against values-X/, then adds them with the correct typography conventions (typographic apostrophe, never escaped).

write-unit-tests: Defines our testing conventions: backtick test names as human-readable sentences, Given/When/Then structure, MockK annotations, MainDispatcherRule for coroutine testing, Turbine for Flow assertions, Robolectric for Compose UI tests, and always wrapping screens in MediumTheme3.

What We Learned

Skills are living documents. We’ve iterated on most skills 3–5 times. The first version of create-compose-screen didn't mention the listener splitting pattern. The add-deeplink skill originally lacked the SUSI destination rule. Each time we caught a pattern break, we updated the skill.

Specificity beats generality. The skills that work best are hyper-specific to our codebase. material3-migration is essentially a lookup table. add-deeplink describes the exact ordering of our handler. These aren't portable to other projects — and that's the point.

Skills compound. A single feature request might trigger create-feature-module, then create-compose-screen, then create-use-case, then create-repository, then implement-analytics-event, then write-unit-tests. Each skill handles its slice correctly. The AI chains them based on what you ask for.

Consistency is the real win. With a small team, the risk isn't that code is bad it's that it's inconsistent. One engineer checks flags in a Flow, another checks at init. Skills eliminate that drift. Every new screen looks structurally identical, regardless of who (or what) wrote it.

Speed is the visible win. Setting up a new screen with ViewModel, listeners, composable, previews, test tags, and tests used to take most of a morning. Now it takes a prompt and a review pass.

Skills can also be written by an Agent, not just Developers

You don't have to write skills yourself. An Agent can write them for you.

The process works by having the Agent ask you the right questions, then observe your existing code to draft the skill. Here's how a typical session looks:

"I want to create a skill for adding a new analytics event. Can you walk me through how you usually do it?"

The Agent asks:

• Which files are involved, and in what order?
• Are there naming conventions to follow?
• What's the checklist you mentally run before opening a PR?
• Can you point me to a recent PR where you did this correctly?

That last question is key. PR examples are the fastest way to ground a skill. When you share a PR link (or paste the diff), the Agent can reverse-engineer the pattern: what changed, in which files, in what order, and what the code structure looks like. It then drafts the skill as a Markdown runbook, which you review and refine.

A good prompt to get started:

"Look at this PR: [link]. I want to write an Agent skill that teaches an AI to reproduce this pattern from scratch. Ask me any clarifying questions you need, then write the skill file."

The bar for a useful skill isn't perfection on the first try. It's one less PR review comment next week.

How to Start

If you're on an Android team (or any team with strong conventions), here's how we'd recommend starting:

1. Write your AGENTS.md first. Document your architecture, patterns, and conventions. This is the foundation.
2. Start with scaffolding skills. Pick your most boilerplate-heavy task (for us: new screens) and write a skill for it. Include the checklist, the file structure, and code patterns.
3. Add migration skills for active migrations. If you're migrating from Material 2 to Material 3, from RxJava to Coroutines, or from XML to Compose encode the mapping.
4. Encode your review feedback. Every time you leave the same PR comment twice, consider writing a skill for it.
5. Keep skills in your repo. Ours live in .agents/skills/ and are version-controlled. When patterns change, the skills change with them.

What's Next

Our AGENTS.md currently carries a lot of weight. It describes our architecture, patterns, conventions, testing strategy, and common commands all in one file. That worked as a starting point, but it has limits: everything is loaded into context all the time, even when only a fraction is relevant to the task at hand.

Our next step is breaking AGENTS.md into dedicated Agent rules scoped, file-aware instructions that activate only when relevant. For example:

• A rule for Compose conventions that activates when editing *.kt files under ui/ packages
• A rule for repository patterns that activates when working in data/ directories
• A rule for testing conventions that activates when editing files under src/test/
• A rule for ViewModel patterns (state management, SavedStateHandle, error handling) scoped to ViewModel files

This is the natural evolution: AGENTS.md gives the AI everything upfront, rules give it the right knowledge at the right time. Smaller context windows, more precise output.

Skills teach the AI how to write code. Commands go further — they automate workflows. One we're actively working on: a release diff command that compares the current release branch to the previous one, summarizes the changelog (new features, bug fixes, migrations), and creates a Linear ticket with the formatted release notes. Today that's a manual process: someone digs through git log, writes up the changes, copies them into Linear. A command could do it in seconds. We see potential for other commands too, generating weekly team reports from merged PRs, auditing a feature module's dependency graph, or preparing QA checklists from the diff.

Other explorations:

• Skills for more complex workflows
• Skills for functional testing
• Skills for Compose screen testing with more sophisticated interaction patterns

The “bet” we're making is simple: the value of an AI coding assistant scales with how much of your team's knowledge you can encode into its context. Skills are how we're doing that.

Making AI Write Android Code Our Way: A Practical Guide to Agent Skills was originally published in Medium Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Medium’s mission is to deepen understanding and spread ideas that matter. We’re building the best place for reading and writing online: A space that rewards clarity, authenticity, and craft over clickbait and engagement hacks.

That vision only works if the technology beneath it is as thoughtful as the writing above it. That’s where you come in. Yes, you!

The Role

We are starting the search for a Principal Backend Engineer at Medium, our most senior IC level and a leadership position at the company. In this role, you will lead the evolution of Medium’s core backend, the systems that power reading, writing, and discovery for millions of users every day.

This is a deeply hands-on, high-impact role. You’ll help craft the strategy for where we want to be as an engineering organization on the backend, and be accountable for the quality, standards, and evolution of the Medium backend platform.

You will:

• Partner with engineering, design, data, and executive leadership to define and deliver the future of Medium.
• Make our platform not just reliable, but genuinely delightful to build on.
• Modernize a large, living codebase so our teams can move faster and ship with confidence.
• Lead through influence by setting standards, reviewing critical code and designs, and mentoring engineers across teams.
• Shape how we build, ensuring our architecture, practices, and systems are strong, scalable, and aligned with Medium’s long-term goals.
• Balance vision with execution by building, debugging, and refining systems yourself to model technical excellence and thoughtful engineering craft.

And you can do this work wherever you do your best thinking, whether that’s a home office, a beachside Airbnb, a rainy coastal cabin in the Pacific Northwest, or your favorite café with great Wi-Fi and even better croissants. This role is fully remote in the US.

Why This Role Matters

I’ve been lucky enough to work with some truly great Principal Engineers in my career, the kind of people who bend the trajectory of a company. They don’t just write brilliant code, they change how everyone around them thinks about building. They spot patterns before they calcify into problems, simplify where others complicate, and bring calm, clear reasoning to complex systems and high-stakes decisions. When they get it right, everyone moves faster, builds better, and understands the system (and the mission) more deeply.

That’s what makes this role special. It’s not just about scale, it’s about stewardship.

About You

You find joy in making complex systems feel simple, in building things that help others move faster and think deeper. You leave codebases better than you found them, teams more confident than before you joined them, and products more human in how they behave.

You’ve built and scaled systems that millions rely on, and you’ve learned that reliability isn’t just uptime, it’s trust. You’ve wrestled with technical debt, fan-outs, and edge cases galore, and found ways to turn them into leverage, not legacy.

You lead with curiosity, not control. You ask great questions, create shared understanding, and help others see the system as clearly as you do.

And through it all, you never forget who it’s for: the writers and readers who come to Medium to make sense of the world.

Why Medium

To me, Medium has always been a rare and wonderful corner of the internet. A place where people come to slow down, think deeply, reflect, and share real human experiences.

In an era where misinformation, AI, and bots are reshaping public discourse, it’s more important than ever to elevate real human voices. That’s only possible if the technology underneath is fast, stable, and built for longevity so we can keep Medium human-centered, sustainable, and ready for what’s next.

If you’re a builder who loves the elegance of distributed systems and the beauty of human stories, we’d love to meet you.

Let’s build a better internet together. One that rewards ideas, not division.

👉 Apply here

• Work at Medium
• Following - Medium Engineering

We’re Hiring a Principal Backend Engineer to Shape the Future of Medium was originally published in Medium Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Medium Android App — Migrating from Apollo Kotlin 3 to 4: Lessons Learned

In this post, I’ll share my experience migrating the Medium Android app from Apollo Kotlin version 3 to version 4, including the challenges I encountered and how I solved them to improve our GraphQL implementation.

Understanding Our Apollo Cache Implementation

Before diving into the migration, it’s important to understand how we use Apollo’s cache in the Medium Android app. Our app relies heavily on Apollo’s normalized cache for several critical purposes:

Performance Optimization: We use FetchPolicy.CacheFirst as our default strategy, which means we always try to serve data from the cache first before making network requests. This significantly reduces loading times and provides a smooth user experience, especially when users navigate between screens that display similar content.

Real-time Updates: We use Apollo’s watch() functionality extensively to observe cache changes and automatically update our UI when data changes. This is particularly useful for features like:

• Live clap counts on posts
• Real-time follower updates
• Post viewed updates
• and more…

Starting the Migration

The initial plan was straightforward: update Apollo Kotlin from version 3 to 4. The IntelliJ plugin made this process seem simple at first glance.

Key Changes in Apollo Kotlin 4

• Group id / plugin id / package name: Apollo Kotlin 4 uses a new identifier (com.apollographql.apollo) for its maven group id, Gradle plugin id, and package name. This change from com.apollographql.apollo3 allows running version 4 alongside version 3 if needed. Source: Apollo Kotlin Migration Guide - Group id / plugin id / package name
• Exception handling: Apollo 4 has a new way of handling exceptions. Instead of throwing exceptions directly, they’re now passed through ApolloResponse. Source: Apollo Kotlin Migration Guide — Fetch errors do not throw
• ApolloCompositeException: In Apollo Kotlin 3, when both cache and network operations failed with a CacheFirst policy, you’d get an ApolloCompositeException containing both errors. Apollo Kotlin 4 simplifies this by throwing only the primary exception while adding any secondary failures as suppressed exceptions, making error handling more straightforward. Source: Apollo Kotlin Migration Guide — ApolloCompositeException is not thrown

The Challenge: Cache Miss Exceptions

After making the initial changes, I encountered a major issue: CacheMissException errors were appearing throughout our UI wherever we used watch(). This was happening because Apollo 4 passes all exceptions to ApolloResponse instead of silently ignoring them when using FetchPolicy.CacheFirst.

This exposed an underlying issue: our cache configuration wasn’t optimal.

Fixing the Cache Implementation

Our app was using Programmatic cache IDs, but the implementation had issues:

• Some IDs were missing from our CacheKeyGenerator
• Some cache key generation logic was incorrect
• These problems led to frequent cache misses

Solution: Declarative Cache IDs & __typename on all Operations

I decided to switch from Programmatic to Declarative cache IDs, which made it significantly easier to match IDs to types and fields. Here’s how I implemented it:

# Types
extend type Catalog @typePolicy(keyFields: "id")
extend type CatalogViewerEdge @typePolicy(keyFields: "id")
extend type Collection @typePolicy(keyFields: "id")
extend type CollectionViewerEdge @typePolicy(keyFields: "id")
extend type Post @typePolicy(keyFields: "id")
extend type PostViewerEdge @typePolicy(keyFields: "id")
extend type User @typePolicy(keyFields: "id")
extend type UserViewerEdge @typePolicy(keyFields: "id")
# etc

# Fields
extend type Query @fieldPolicy(forField: "collection", keyArgs: "id")
extend type Query @fieldPolicy(forField: "post", keyArgs: "id")
extend type Query @fieldPolicy(forField: "publication", keyArgs: "id")
extend type Query @fieldPolicy(forField: "user", keyArgs: "id")
# etc

To further improve cache hit rates, I added the __typename to all operations by configuring it in the Gradle build file:

apollo {
    service("service") {
        addTypename.set("always")
    }
}

Cache Update Extension Functions

To further improve our caching logic, I created extension functions for updating cache fragments that make both reading and writing more intuitive:

internal suspend inline fun <D : Fragment.Data> ApolloStore.updateCache(
    fragment: Fragment<D>,
    cacheKey: CacheKey,
    customScalarAdapters: CustomScalarAdapters = CustomScalarAdapters.Empty,
    cacheHeaders: CacheHeaders = CacheHeaders.NONE,
    publish: Boolean = true,
    crossinline block: (cachedData: D) -> D,
): Set<String> {
    val cachedFragment = getCachedFragment(
        fragment = fragment,
        cacheKey = cacheKey,
        customScalarAdapters = customScalarAdapters,
        cacheHeaders = cacheHeaders,
    ) ?: return emptySet()

    return writeFragment(
        fragment = fragment,
        cacheKey = cacheKey,
        fragmentData = block(cachedFragment),
        customScalarAdapters = customScalarAdapters,
        cacheHeaders = cacheHeaders,
        publish = publish,
    )
}

suspend fun <D : Fragment.Data> ApolloStore.getCachedFragment(
    fragment: Fragment<D>,
    cacheKey: CacheKey,
    customScalarAdapters: CustomScalarAdapters = CustomScalarAdapters.Empty,
    cacheHeaders: CacheHeaders = CacheHeaders.NONE,
): D? = runCatching {
    readFragment(
        fragment = fragment,
        cacheKey = cacheKey,
        customScalarAdapters = customScalarAdapters,
        cacheHeaders = cacheHeaders,
    )
}
.onFailure { e ->
    when (e) {
        is CacheMissException -> Timber.e(e, "Cache miss on fragment $fragment with cache key $cacheKey.")
        is ApolloException -> Timber.e(e, "Cache read error on fragment $fragment with cache key $cacheKey.")
        else -> Timber.e(e, "Unexpected error while reading fragment $fragment with cache key $cacheKey.")
    }
}
.getOrNull()

These extension functions significantly improved the readability of our cache manipulation code and provided better error handling for cache operations.

Testing and Handling Cache Exceptions

To ensure our Apollo response handling was correct, I first wrote unit tests for our methods that transform ApolloResponse into Kotlin Result types.

@Test
fun `safeWatch with CacheMissException and FetchPolicy#CacheFirst should return success flow`() = runTest {
    // Given
    val requestUuid = Uuid.randomUUID()
    val cacheResponse = ApolloResponse.Builder(
        operation = IsFollowingCatalogQuery("ID"),
        requestUuid = requestUuid,
    )
        .exception(CacheMissException(CacheKey(Catalog.type.name, "CATALOG_ID").toString()))
        .build()
    val data = IsFollowingCatalogQuery.Data(
        catalogById = IsFollowingCatalogQuery.CatalogById(
            __typename = Catalog.type.name,
            catalogFollowData = CatalogFollowData(
		            __typename = Catalog.type.name,
		            id = "CATALOG_ID",
		            viewerEdge = CatalogFollowData.ViewerEdge(
		                __typename = CatalogViewerEdge.type.name,
		                id = "VIEWER_EDGE_ID",
		                isFollowing = true,
		            )
		        ),
        )
    )
    val networkResponse = ApolloResponse.Builder(
        operation = IsFollowingCatalogQuery("CATALOG_ID"),
        requestUuid = requestUuid,
    )
        .data(data)
        .build()
    every { mockApolloCall.watch() } returns flowOf(cacheResponse, networkResponse)

    // When
    mockApolloCall.safeWatch(FetchPolicy.CacheFirst) { it }.test {
        // Then
        val result: Result<IsFollowingCatalogQuery.Data> = awaitItem()
        assertTrue(result.isSuccess)
        assertEquals(expected = data, actual = result.getOrNull())
        assertNull(result.exceptionOrNull())
        awaitComplete()
        ensureAllEventsConsumed()
    }
}

Then I fixed the exception propagation in our watchers with this approach:

• We added fetchPolicy and refetchPolicy parameters to our watchers with default values. These defaults match those used by the Apollo Kotlin SDK.
• We transform the ApolloResponse into a Result, enabling us to handle either Success or Failure cases.
• If the fetchPolicy is CacheFirst or CacheAndNetwork, we are skipping the CacheMissException, as Network will emit after either a Success or an ApolloNetworkException.
• If the fetchPolicy is NetworkFirst, we are skipping the ApolloNetworkException, as Cache will emit after either a Success or an CacheMissException.

inline fun <D : Query.Data, R> ApolloCall<D>.safeWatch(
    fetchPolicy: FetchPolicy = FetchPolicy.CacheFirst,
    refetchPolicy: FetchPolicy = FetchPolicy.CacheOnly,
    crossinline transform: (D) -> R,
): Flow<Result<R>> = this
    .fetchPolicy(fetchPolicy)
    .refetchPolicy(refetchPolicy)
    .watch()
    .mapNotNull { response ->
        val result = response.toResult(transform)
        val exception = result.exceptionOrNull()

        when {
            exception is CacheMissException && fetchPolicy == FetchPolicy.CacheFirst ->
                null
            exception is CacheMissException && fetchPolicy == FetchPolicy.CacheAndNetwork ->               
                null
            exception is ApolloNetworkException && fetchPolicy == FetchPolicy.NetworkFirst ->
                null
            else ->
                result
        }
    }

The key insight here is that different fetch policies have different fallback strategies, and our exception handling needs to respect these strategies. By filtering out expected exceptions that will be followed by either success or a different type of exception, we ensure that our UI only receives meaningful errors that require user attention.

Completing the Migration

After resolving these cache-related issues, I was finally able to complete the migration to Apollo 4:

• Replaced executeV3() with execute()
• Updated watch() calls to remove fetchThrows = true
• Fixed ApolloCompositeException handling

Additional Improvements: Custom Type Adapters

While diving deep into the Apollo documentation, I also discovered we could use type adapters for scalar values. I implemented this for our Currency scalar:

import java.util.Currency
object CurrencyAdapter : Adapter<Currency> {
    override fun fromJson(reader: JsonReader, customScalarAdapters: CustomScalarAdapters): Currency =
        Currency.getInstance(reader.nextString())

    override fun toJson(writer: JsonWriter, customScalarAdapters: CustomScalarAdapters, value: Currency) {
        writer.value(value.currencyCode)
    }
}

Future Improvements

HTTP Batching

HTTP batching allows multiple GraphQL operations to be sent in a single HTTP request, reducing network overhead. This is particularly useful for applications that execute multiple queries simultaneously, as it can significantly improve performance by reducing the number of network requests. We are currently using HTTP batching on our Web platform without encountering any issues.

Persisted Queries

Persisted Queries improve network performance by sending a query hash instead of the full query text. This reduces payload size and can improve security. The server maintains a mapping of hashes to query strings, allowing it to execute the appropriate query when it receives a hash. Note that implementing Persisted Queries requires backend support.

Conclusion

What began as a simple version upgrade became a comprehensive overhaul of our GraphQL implementation. By switching to Declarative cache IDs, adding __typename to all operations, and properly handling cache exceptions, we've significantly improved the cache hit of our Apollo GraphQL integration.

The key takeaway: when upgrading Apollo Kotlin, be prepared to revisit your caching strategy. The improvements in version 4 expose issues that might have been hidden in version 3, but fixing them leads to a more robust implementation.

Medium Android App — Migrating from Apollo Kotlin 3 to 4: Lessons Learned was originally published in Medium Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

How we made our filtering 10x cheaper by removing our Bloom Filters

Bloom Filters are great tools to make fast and cheap filtering. They also come with plenty of problems and can easily get expensive and cumbersome. We switched to user-based direct database queries, which made our filtering cheaper and easy to maintain. Here’s the full breakdown of that migration.

Intro: This is a 4-part series breaking down improvements to the algorithm behind the Medium’s Daily Digest over the past year. When we started this work, the Digest was suboptimal — and since it’s a huge distribution surface, reaching millions of readers every day, we started working on incremental improvements.

By the end of these projects, the digest was 10% more likely to convert users to paying members, less expensive to run, more flexible and easier to maintain and it’s now providing higher quality recommendations for all our users, including our “power readers”.

This is told through the lens of our engineering team tackling a series of challenges one by one. Medium has a small team but we operate on a big scale. We’re working our way through some technical debt and at the same time, striving to provide the best experience for our readers. This is the source of many interesting challenges.

I hope this series helps you understand how the recommendations algorithm work and can help others who are facing similar technical challenges.

This is probably the most technical story in the series, but I will keep it as simple as possible and hopefully this is interesting for non-technical readers too.

Some Concepts

Here’s a little cheat sheet with some concepts you may need to follow along with this story

Bloom Filters at Medium

A lot of the filters I mention in this series are backed by Bloom Filters (I’ve described some of those filtering rules in Part 1 if you haven’t read it already). We use Bloom filters to remove stories we think won’t interest readers from their feeds and other recommendations:

For example:

• our “muted” filter removes all stories from writers that you have muted
• our “read” filter removes all stories that you have already read
• our “presentation” filter removes all stories that have already been presented to you 3 times or more

These all rely on Bloom Filters

So what’s a Bloom Filter?

I asked Claude to summarize Bloom filters in a really simple way and it went with a funny analogy I’m going to try here.

A Bloom filter is like a super-efficient bouncer at a club who has a really good memory but isn’t perfect. It lets you do two things:

1. let someone into the club
   → in code that would be an add(string) function
2. lets you ask if someone is in the club. There are two possible answers to this:
   → “yes, probably”
   → “no, definitely not”
   → in code that would be a check(string) --> bool function

That doesn’t sound super useful like that but we’ll see next that it’s actually kinda well suited for recommendation systems.

At Medium we’re using it to store information such as “user a read story x” or “user a muted user b”. We add those to the “club” as strings, like read|user_x|story_y . Later on, when we want to know if user x has already read story y, we just ask our “bouncer”: is read|user_x|story_y in the club?

The scale of the filtering

What’s nice with Bloom filters is that they are able to store the information very efficiently and they are able to handle big amounts of requests per second. We don’t really need to know more about the inner workings of Bloom filters for this series but you can read more about it here (and it has some Excalidraw schemas too 🤌).

When we’re building a feed for a user (the digest for example), we’re sometimes sourcing up to 5000 stories as the initial “shortlist” of stories. This shortlist goes through many different steps and when we’re done, there’s only a handful of stories, ready to be sent in your digest 🙌

Ideally we should filter out stories as early as possible in the process. If we take the filters I’ve listed above, we’re asking ourselves 3 questions for every story in the short list:

• did the user read this story already?
• is the writer of the story muted by the user?
• was the user presented this story more than 3 times already?

So that can add up to 15k questions in total for a single user feed, and we process thousands of feeds per second. So that’s more than 15M questions per second. You can see how that can get out of hand and become expensive very quickly, we’re going to need some solid infrastructure to handle this.

Fortunately our Bloom filter “bouncer” is able to answer an insane amount of questions per second for pretty cheap, which is exactly what we’re looking for.

Bloom Filter downsides

The biggest downside for us is that even though the Bloom filters are super memory efficient, we overused them so much by adding literally billions of items that they started getting really really big and expensive. Like I mentioned before, there are only two operations you can do on a Bloom filter: add or check. There’s no way to delete any data from the filter (remove people from the club, if we stick with the bouncer analogy). And so the club can only grow in size, there’s no way to perform routines cleanups for information that we don’t need anymore.

The only way to reduce the size of the club is to start a new club with a new bouncer and then retire the old club and bouncer.

There are also a few more downsides:

• you can’t list the items that are stored (you can’t get a list of people who are in the club…). So we can’t go and see what’s stored for a given user, this makes it really hard to debug issues with the filters
• you can’t remove anyone from the club, which means that you can’t change your mind. For example if a user “unmutes” a writer, there’s no way to reflect that in the Bloom filter. That writer will be muted forever for that user. Yes, that’s very janky.
• this is derived data. For example the user “mutes” are stored in a proper database, and anytime there is a mute action we need to forward that information to the bloom filter. This is inconvenient and introduces data drift issue and more complexity overall.

All in all, there were too many downsides so we decided to explore a different approach.

Replacing the “Muted Filter”

When we build the feed for a given user, it happens in real time. It can be when you load the Medium homepage, or when we trigger the daily digest generation. We’re typically in the following situation:

• we have a shortlist of 1000 stories
• we want to filter out the stories from writers that are muted by the user

There are two ways to go about this:

A. perform a lookup in the database table that keeps track of muted writers for the 1000 (userID, writerID) pairs and check the mutedAt attribute

This is extremely expensive which is why we have to introduce a Bloom filter, where lookups are faster and cheaper

B. or fetch ALL of the muted writers for the current user and then cross-reference that with your shortlist of stories

Approach B has a massive advantage: most users do not mute anyone. So on average, we’re reading a very small amount of data from the database. If we were to do that with dynamoDB, B is a Query that retrieves on average less than 1 item from the DB for each feed. This is very cheap and fast. For “power muters” — users who mute massive amounts of writers, in the thousands — we can still handle this in real time although with higher latencies.

So this approach immediately obliterates the need for a Bloom filter. We have a fast, cheap, reliable way to filter out muted writers. We can also use our “ground truth” database directly, no need for derived data with all the headaches this involves.

So that’s one less Bloom filter! Let’s move on to the next one.

Replacing the “Presentation Filter”

The question becomes a bit trickier when we look at the “presentation” filter. We’re typically in a situation where:

• we have a shortlist of 1000 stories
• we need to remove all the stories that were presented to the user in a feed (in the past) 3 times or more

We’re currently doing that filtering using a Bloom filter. But you might wonder how we can even do that with Bloom filters. Remember there’s only two operations you can do with a bloom filter:

• add someone to the club
• ask if someone is in the club

Bloom filters were not built to maintain counters, only true / false information. So we have to hack our way around it by layering them. In terms of clubs and bouncers, it’s like we have a big festival with a bouncer. Inside the festival there’s a private club with another bouncer. And inside that private club there’s a VIP zone with another bouncer… Ultimately we only want to know if someone is inside the VIP zone, so we only need to ask the VIP bouncer. But we still need the two other bouncers to keep track of who’s eligible to get in the VIP zone…

Going back to Bloom filters: we use 3 layers of Bloom filters, on top of each other, with each one encoding the information “post was presented to user x times”

From the recommendations algorithm’s perspective it’s fairly simple, we just lookup the (user, post) pairs in the Bloom filter that encodes the “3” value of the counter (we just ask the VIP bouncer)

NB: that’s the implementation in place at Medium. I don’t know what went into consideration when building it this way. But FIY there are other (probably better) ways to build counters with Bloom Filters. It’s possible to use bit-arrays for example.

So that’s for the Bloom filter implementation. Now how can we handle that differently?

The user-based approach involves fetching all of the “presentation history” of a user (ie all of the posts that were presented to the user in a feed before) and then going over that list to compute presentation counts for each story.

That’s a little bit more complex than the “muted filter”, because:

• a lot of users have really massive presentation histories, in the tens of thousands or even bigger. Those are readers who come to Medium every day and are exposed to many recommendations
• on average, users have a lot of posts in their presentation histories. Typically less than 100 though. That’s because this includes many less engaged users who only came a handful of times on Medium, and they bring the average down.

So the average cost is going to be higher than for the muted filter because we’re going to retrieve more data from the database AND some users have such big presentation histories that it’s not possible to fetch it in real time. So how can we tackle that?

Here, we’re saved by the fact that this filtering rule is not a hard requirement. Nothing says that “3 presentations is the absolute max” for a given (user, post). We built this rule to make feeds more diverse and less repetitive. We chose 3 as a reasonable default, but it’s okay if in some cases we reach higher counts.

So we can go with a “best effort” strategy here. We fetch the most recent 5k presentations for the user and simply act as if everything before that never existed. DynamoDB queries let us fetch thousands of items quickly, in the ~100ms. So that simple solution is very acceptable in terms of functionality and latencies. There are scenarios where the filter will not be doing its job properly, but they’re limited to certain edge cases with minimal impact on the user.

We tested this approach and found that the costs were reasonable. This solution gives us more flexibility, for instance we can now easily control and play with the maximum number of presentations (is 3 the right threshold? we’ll see that in Part 3). Down the line that will lead to a better experience for users. Now that this filter has been replaced, let’s move on to the final filter, which was the most challenging to migrate.

Replacing the “Read Filter”

Here’s what we’re trying to do:

• we have a shortlist of 1000 stories
• we need to remove all the stories that the user has already read

Very similar to the “Presentation Filter”, the user-based approach for this filter involves fetching the entire reading history of the user. We’re in the same situation where most of the time the user’s reading history is relatively small, with less than 100 posts on average. But we do have some users with massive reading histories.

Fun fact, one of or top readers is our very own Harris Sockel who was in charge of the Medium Newsletter, and had 22k stories in his reading history just for 2024!

This time, filtering already read posts is a hard requirement of the recommendations algorithm, no way around it. We don’t want to send you an email or a push notification about a post you’ve already read.

So what do we do? For this filter specifically we decided to do the filtering in two stages. Very early in the feed building process, we fetch the most recent posts in the user’s reading history (the last 5k posts you’ve read). For the vast majority of users, this will capture their entire reading history. And we use that to filter out already read posts from the feed (at this point, we can have up to 5k posts in the feed). But that’s not enough, we need to guarantee that there are no read posts in the final results for all users.

So to explain this further, we’ll need to go into our recommendation algorithm in a little bit more detail:

You can see that there are two different filtering steps. What we did is that we split out our “Read Filter” into two different implementations and we added each one at a different step:

We’re doing user-based queries in the early filtering step. This is nice because costs do not depend on the number of posts in the shortlist. Once we’re down to only a few posts, we can perform pairwise (user, post) lookups.

This solution works really well and doesn’t add too much complexity to the recs logic. This did require some gymnastics to maintain the requirements while keeping costs, recs performance and latencies under control. But this is much easier to control and debug than the previous implementation with Bloom Filters. It also allows to support things like clearing the reading history for a given user (you can do that from this page to get a “recs fresh start” on Medium). This is a functionality that didn’t work well with the Bloom Filter implementation.

Conclusion

All in all we were able to get rid of our Bloom filters entirely, resulting in big cost savings. The new implementation is ten times cheaper than what our Bloom filters were costing us.

This isn’t necessarily a fair comparison since our Bloom filters had grown out of proportion. If we had migrated from scratch to a new Bloom instances, the Bloom filter implementation might have been cheaper. It was hard to evaluate it in our situation because of the way we built our Bloom filters. We used a single instance and string prefixes to manage all of the different filters.

If we go back to our club and bouncer analogy, you can see it that way: let’s say there’s different crowds that go to the club, techno lovers, jazz enthusiasts, disco heads…

A. you can build a single club that hosts everyone. A single bouncer has the responsibility to memorize all the people in the club

B. or you can build one club for each crowd, each one with a different bouncer

If one day you don’t care about jazz enthusiasts anymore, with B. you can just retire the jazz club and it’s bouncer. But with A. you still need the bouncer for the other crowds, so you can’t retire the bouncer and it’s still holding the information about all the jazz enthusiasts that entered the club. Down the line you might be paying for a club that’s bigger than you need with approach A. With approach B. you pay exactly for what you need!

With Bloom filters it looks like this:

We were working with a single gigantic Bloom Filter, this meant that we weren’t able to retire anything and our Bloom Instance was ever-growing.

The bottom line is that this project helped us get our costs down and migrate to newer, more flexible implementations that are much easier to maintain. With our newer implementations:

• costs are easy to understand and are under control
• filtering costs don’t depend on the initial size of the feed “shortlist”
• there’s no need to maintain derived data

This also laid the groundwork for experiments on our filtering rules that we’ll see in our Part 3: Hard vs Soft Filtering and how this applies to Medium’s Recommendation System.

Thank you for reading this serires, you can stay tuned for the next installments of this series by following the Medium Eng Blog.

Engineering stories behind the Medium Daily Digest Algorithm: Part 2 was originally published in Medium Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

How we made our email story recommendations better

In this Part 1, you’ll understand how we improved one of the main ways our users are exposed to our product and how that led to a massive 7% increase on the average reading time for the digest users.

Intro: This is a 4-part series breaking down improvements to the algorithm behind the Medium’s Daily Digest over the past year. When we started this work, the Digest was suboptimal — and since it’s a huge distribution surface, reaching millions of readers every day, we started working on incremental improvements.

By the end of these projects, the digest was 10% more likely to convert users to paying members, less expensive to run, more flexible and easier to maintain and it’s now providing higher quality recommendations for all our users, including our “power readers”.

This is told through the lens of our engineering team tackling a series of challenges one by one. Medium has a small team but we operate on a big scale. We’re working our way through some technical debt and at the same time, striving to provide the best experience for our readers. This is the source of many interesting challenges.

I hope this series helps you understand how the recommendations algorithm work and can help others who are facing similar technical challenges.

Some Concepts

Here’s a little cheat sheet with some concepts you may need to follow along with this story

The Discovery

A little while back, Leigh, our Machine Learning Engineer and model training guru, started noticing something weird. The recommended stories in his digest were consistently not great matches for his reading profile. At the same time, the recommended stories in his homepage feed (what we call the “For You” feed) were consistently very well targeted.

This was a bit puzzling and unexpected. Those two recommendation surfaces rely on exactly the same algorithm. We source stories the same way, and we rank them using the same model and features. The only big difference lies in the filtering step.

The Investigation

So at this point, we thought maybe this is a bug or maybe there’s something in the filtering step that’s amok specifically for the digest. We started digging into the different filters we use for the digest vs the ones we use for the homepage feed. We have many different filtering rules. For example the “Filter Read” simply filters out all the stories that the user has already read. This one is applied on most recommendation surfaces. But some other filters are specific to certain recommendation surfaces:

One of the reasons why those two surfaces use different filtering is because the reporting does not work the same:

• on the homepage we can track exactly what posts were presented to the user
• with the digest email we have access to less information:
  → we know if the email was opened (thanks to a tracking pixel, explained below)
  → we know what posts we sent in a given digest
  → we know which posts were clicked in a given digest

Looking at this, we started realizing that the filtering applied to the digest is probably too aggressive. Every day, the algorithm sources the best 15 stories on the entire platform for the reader. But as soon as we detect an email was opened then we will never send the posts it contains again to that reader. That can be a bummer in some situations, for example:

• the user opens the email but doesn’t scroll past the top 5 stories
• all the stories in that digest will never be sent again, although 10 of them didn’t even get a chance to be presented to the reader

Having worked on email open tracking before, we also knew that there were other elements at play here and that are potentially making things even worse.

Apple Mail Privacy Protection

In 2021, Apple started Mail Privacy Protection. The idea was to better protect user data and to prevent email open tracking.

When we send the digest, we also send a pixel tracker in the email. That’s a tiny image URL that we send as part of the mail content and when the email is loaded, the email client has to fetch this image. The URL is engineered in a way that when it’s loaded, we know exactly which email loaded it and so we can record the email as “opened”.

Apple’s Mail Privacy Protection has completely destroyed this concept. They are now preloading all your emails directly from their servers, and your email client communicates only with the Apple servers. From our point of view, it means that pretty much all the emails from Apple users are marked as “opened”.

So a lot of our “email open events” are actually “remote servers opening your email for you so that no one can tell if you actually opened it or not” events. We’re still not sure exactly about the exact figures but several estimates suggest that 50 to 80% of our “email open events” are fake.

That means a lot of our email open events are “fake news”. And those 15 stories that we carefully selected for you will never be eligible for your digest again — even if a lot of user didn’t actually open the email.

So it does look like we’re filtering way too aggressively. At this point we were confident that this filtering rule was the culprit and that it’s responsible for degrading the quality of the recommendations in the digest. Now we just need to come up with a plan to fix it.

The Plan

We went over the filtering rules in place and decided we’d modify a few things:

This is now a much less aggressive setup. It means that there will be more repetition in digests, the hope is that the best stories are able to make their way to the user and that they don’t get “disqualified forever” too early. Several things are in place to ensure that the digests are not too repetitive and evolve over time:

We were still a little bit wary about potential repetitiveness of digests. We wondered if users were going to notice and complain about that, so we gave a heads-up to the support team to be on the lookout for such complaints

We put that to the trial in an A/B test and got incredible results very quickly.

Users in the experiment:

• were 10% more likely to convert to paying members
• read 7% more on Medium

A massive win for Medium!

Readers didn’t seem to notice any repetitiveness in the digests, so we decided to ship without thinking too much about it.

We were so happy about the results that we immediately went on the hunt for more filtering rules that we could tweak.

We first had to dig into the implementation of our filters (CF Part 2: How we made our filtering 10x cheaper by removing our Bloom Filters), before we could start making more changes (CF Part 3: “Hard vs Soft Filtering” and how this applies to Medium’s Recommendation System and Part 4: Cross-Digest diversification)

Some Final Thoughts

So it looks like the Apple Mail Privacy Program has been affecting our recommendation systems since 2021. Not all Apple Mail users were in the program initially and the number of users in the program has been scaling up over the years as well as the blast radius on Digest users. My recommendation to you, if you have any features that rely on email open tracking in your product, is to immediately audit the potential effects of those “fake email opens”.

To be clear we are absolutely in favor of user privacy measures. The Apple privacy program has forced us to re-evaluate the way we build our recommendations in a less intrusive way, which is a good thing.

We recently made the daily digest available in the Medium App, this was a much requested feature by our users. On top of that this is an opportunity to get better tracking than what we do in the email clients. Down the line this could help us improve the recommendations we put in those digests.

Thank for reading this series, you can stay tuned for the next installments of this series by following the Medium Eng Blog.

Engineering stories behind the Medium Daily Digest Algorithm: Part 1 was originally published in Medium Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.