How a single Express middleware caused a 1557% Firebase cost spike and how we fixed it

Building Vestron an Instagram saved posts organiser, we hit a wall last week. Firebase bill spiked 1557% overnight with no code changes.

Here's exactly what happened and how we fixed it.

**The symptom**

Cloud Function invocations were through the roof. Meta was flooding our server with webhook retries because our server kept returning a non-200 response on signature validation. Meta interpreted this as our server being down and hammered us with exponential backoff. Thousands of duplicate calls.

**The root cause**

We were using Express with body-parser middleware, which automatically parses raw JSON into a JavaScript object before our code even runs. Meta signs their webhooks using HMAC-SHA256 computed on the exact raw bytes of the message body. By the time body-parser touched the data, those raw bytes were modified. Even a single character difference meant our signature never matched. We were silently failing every single webhook validation.

**The fix**

We built a dedicated standalone Firebase Function (`instagramWebhookV2`) that bypasses Express entirely:

  1. Grab `req.rawBody` — the exact byte stream Meta originally sent

  2. Run HMAC-SHA256 verification as the absolute first line of code

  3. Return `200 OK` to Meta in milliseconds

Retries dropped to zero immediately. Bill normalised the same day.

**The unexpected bonus**

Our old architecture: receive webhook → save to database → trigger function cold-starts → send bot response. Total: 10-15 seconds.

New architecture: receive webhook → verify signature → process inline → respond. Total: under 2 seconds.

Users now get the bot response in real time instead of waiting 15 seconds wondering if anything happened.

**The lesson**

For any webhook that uses raw-body signature verification (Meta, Stripe, GitHub, etc.) — never let middleware touch the body before verification. Bypass Express or use `express.raw()` with `verify` callback to preserve raw bytes alongside the parsed body.

Happy to answer questions if anyone's hit the same issue.

submitted by /u/Far-Cucumber2287
[link] [comments]