How I Eliminated 19,494 Duplicate MongoDB Records (And What I'd Do Differently Today)

At a previous organization, I was handed ownership of a job platform serving 1,000+ users worldwide. Jobs on the platform could be scoped to a single city, a state, a whole country, or “available worldwide.” Simple enough. At least, you’d think so.

When I joined, this was early in my career and I was a frontend dev. I was interested in backend work but hadn’t done any real backend engineering yet — I wasn’t battle-tested. Somewhere along the line, I got handed the whole platform. Frontend, backend, all of it. Bracing moment. I forced myself to slow down, actually understand the system end to end, and build a real plan before touching anything.

After talking with product management, I pulled up the backend task backlog. I learn best by doing, so I started at the bottom — smallest bugs first, working my way up. It went well for a while. Then I hit a ticket tied to core business logic.

The Wake-Up Call

Some endpoints were clocking API response times north of 7,000ms. I genuinely thought it was a typo. Then I remembered running into something similar on the frontend side before — so I patched the UX so users never felt the full weight of the delay. But the actual problem was still sitting there, untouched.

Before touching any code, I made a checklist. We were running MongoDB with NestJS:

• What schemas was the endpoint touching?
• What optimization had already been applied to those schemas?
• How was the endpoint fetching data?
• How many records did each related schema currently hold?

Checklist in hand, I went into the codebase.

My innocent dev brain was not prepared for the chaos I found.

What Was Actually Happening

Here’s the setup: assume an empty jobs collection. A recruiter creates a job listing scoped to New York City. The backend takes the payload and creates one record with location set to New York City. Straightforward.

Now assume another empty collection, and this time the job is scoped to the United States instead of a single city. What do you think happens?

If you guessed “the same thing — one record, location set to United States” — I have bad news for you.

What the backend actually did: it looked at the payload, recognized “United States” as a country rather than a city, looked up every known city in the U.S. — and created a separate job record for each one.

Read that again.

Instead of one record, the platform generated one identical record per city, differing only in _id and location. And it got worse: a job marked “available worldwide” would spawn a record for every city on the planet.

This was a live product with real traffic. Querying that collection — even with indexes in place — was a nightmare.

For context on scale: a single job scoped to just the USA generated 19,495 records — one canonical listing plus 19,494 exact duplicates.

Database records (1 USA-wide job)	19,495
Exact duplicates among them	19,494
Query time on affected endpoints	~2.3s (worst cases exceeded 7s for worldwide-scoped jobs)

My first reaction was horrified confusion. After talking it over with management, I learned there was a reason for the original design — I won’t get into it here, it’s not the point of this piece. What mattered was that I had free rein to fix it if I could pull it off. Hold my Pokémon card.

The Constraints

Fixing the architecture wasn’t the hard part conceptually — the hard part was fixing it under real constraints:

• We still needed to resolve which cities a job was available in, for every existing feature that depended on that.
• Whatever I built had to slot into the existing infrastructure without a rewrite of everything downstream.
• I needed to show measurable, provable performance gains — not just “trust me, it’s better.”
• None of this could disrupt daily active users. No maintenance windows, no downtime.

Designing the Fix

I started with the schema. The core question: how do I preserve full location fidelity while collapsing tens of thousands of records into one?

I thought back to basic compression concepts — take data, strip it to a minimal representation, and pair it with the instructions needed to reconstruct the original. That’s essentially what I needed here: a compact way to represent “this job is available in X” without one document per X.

Two deliverables:

1. A new, efficient schema.
2. A migration path that collapsed the old duplicated records into the new schema.

For the new schema, I kept every existing field and replaced the flat location string with a structured locationObj:

interface LocationObj {
  country: string | string[];
  state: string | string[];
  city: string | string[];
}

The logic:

• Single city → city is a string, state and country resolve to that city’s parent state/country.
• A specific group of cities → city becomes a string array.
• An entire state → city becomes the sentinel value "all", state holds the actual state name.
• An entire country → state and city both become "all".
• Worldwide → country, state, and city all become "all".

So a job available across Berlin, Munich, and Hamburg looks like:

{
  title: "Marketing Manager",
  company: "TechCorp",
  locationObj: {
    country: "Germany",
    state: "all",
    city: ["Berlin", "Munich", "Hamburg"]
  }
}

And a job available anywhere in the world:

{
  title: "Software Developer",
  company: "TechCorp",
  locationObj: {
    country: "all",
    state: "all",
    city: "all"
  }
}

One document. Full location semantics preserved. No fan-out.

Migrating Without Downtime

The deduplication piece ran as a batch script against the old collection: group records that shared everything except location (same title, company, description, salary — the actual “identity” of a job), reconstruct the correct locationObj from the set of location values in that group, and write a single new document into the new collection.

Once the new collection was validated against the old one, I built new endpoints on top of the new schema, mirroring the contracts of the old ones, and prepared the frontend to point at them.

The cutover itself was a backfill with two catch-up passes, not a one-shot migration:

1. Run the migration script once against the live old collection.
2. Right before deploying the new endpoints, re-run it — this catches any jobs created while the first pass was running, since the old collection kept accepting writes the entire time.
3. Deploy the new endpoints and switch the frontend over.
4. Re-run the script once more right after — this catches anything created under the old schema during the deploy window itself, in case the rollout wasn’t instantaneous.

I tested the full path — frontend, backend, and DB — on local and staging before any of this touched production, so by the time it shipped, the swap itself was just a normal deploy. From a user’s perspective, they kept seeing exactly the jobs they were supposed to see, the whole way through.

Querying the New Schema

The query logic has to walk the hierarchy — country, then state, then city — because “all” at any level short-circuits the rest:

const query = {
  $or: [
    { "locationObj.country": "all" },
    { "locationObj.country": userCountry, "locationObj.state": "all" },
    { "locationObj.country": userCountry, "locationObj.state": userState, "locationObj.city": "all" },
    { "locationObj.country": userCountry, "locationObj.city": userCity },
    { "locationObj.country": userCountry, "locationObj.city": { $in: [userCity] } }
  ]
};

Combined with a compound index on locationObj.country, locationObj.state, and locationObj.city, this scales linearly with the number of actual jobs — not the number of cities they happen to be available in.

The Results

Metric	Before	After	Change
Database records (1 USA-wide job)	19,495	1	99.99% fewer
Query time	~2.3s	~0.12s	~95% faster
Storage footprint	847 MB	0.043 MB	99.99% smaller
Index size	156 MB	0.008 MB	99.99% smaller
Memory usage	~4.2 GB	~0.21 MB	99.99% less RAM
Exact duplicate documents	19,494	0	Eliminated

Management was, unsurprisingly, impressed.

How I’d Do It Now

Looking back at this with a few more years under my belt, the schema still holds up. What I’d change is the migration and verification process — at the time, “backfill, then re-run twice around the cutover” felt rigorous. It worked. But it also relied on timing more than I’d want to admit if I were reviewing this today.

The more rigid version of this — the one I’d actually reach for now — is the expand-contract pattern:

1. Expand: Deploy dual-writes behind a feature flag — every job creation or update writes to both the old and new schema at the same time.
2. Sync: Use MongoDB Change Streams to mirror any old-schema writes into the new schema in real time during the transition, instead of relying on timed reruns of a migration script. This closes the race-condition window entirely — there’s no gap to miss, because nothing is waiting for a scheduled pass.
3. Contract: Once reads are fully on the new schema and stable, kill the old write path.

I’d also make the migration script itself idempotent and checkpointed — track the last processed _id so a crash mid-run means it resumes instead of reprocessing or silently skipping documents. Mine was a script I babysat through completion; a checkpointed version is one I could walk away from.

On verification — “compare the counts” was good enough to convince management, but it wasn’t rigorous. I’d now do a real reconciliation pass: diff the reconstructed locationObj against the actual set of cities in the old schema, per job, and flag any mismatch automatically instead of trusting a spot check. And before flipping reads over, I’d run a dark-launch period — querying both schemas in production and logging discrepancies without serving the shadow result to users — so any gap shows up before it’s user-facing, not after.

One more thing I’d change that isn’t about the migration at all: I’d want this caught before it ever shipped. The root failure was that nothing stopped a single job-creation event from silently producing thousands of documents. A write-layer guardrail — “creating one job should never result in more than one document, alert if it does” — plus a five-line regression test asserting exactly that, would have caught this in review instead of in production months later.

Lessons Learned

Embedding vs. referencing isn’t the real question — normalization is. The original design wasn’t wrong because it embedded or referenced incorrectly; it was wrong because it materialized every possible resolution of a hierarchical value instead of storing the hierarchy itself. The general MongoDB advice of “embed for read-heavy, reference for write-heavy” didn’t apply here — the fix was closer to a compression problem than a modeling problem.

Sentinel values are underrated. Using "all" as a stand-in for “every value at this level” let one field type (string | string[]) represent five distinct scopes without a single boolean flag or separate scope enum.

Get the actual scale of the problem before designing the fix. I didn’t just need “make it faster” — I needed a number to prove it. Knowing it was 19,495 records for one job is what made the case to management, and it’s what makes this story worth telling.

---

Fields, numbers, and code above are reconstructed from a project where I owned the full backend under an NDA — details are generalized where necessary, but the architecture, the failure mode, and the metrics are real.

---

Cover photo by Mika Baumeister on Unsplash