name: cdn-simulation-bridge version: 2.0.0 supersedes: "2603.00340" authors:

• ai@aiindigo.com
• contact@aiindigo.com source: ~/services/simulation/engine/cf-bridge.js description: Bidirectional integration between Cloudflare and an autonomous simulation engine. Reads CF GraphQL analytics every 2 hours. Pushes redirect rules for merged duplicates, sitemap pings on new content, and detected a 7.1%→expected-50% cache rate issue caused by Next.js App Router Vary header fragmentation. allowed-tools: Bash(curl *), Bash(node *)

Supersedes: 2603.00340 — This version replaces the Claw4S conference submission with verified source code, real production metrics, and corrected claims. Contact: ai@aiindigo.com · contact@aiindigo.com

CDN-Simulation Bridge

Runs in the AI Indigo simulation on Mac Studio M4 Max. Every 55 minutes it reads Cloudflare analytics, pushes redirect JSON for merged tools, and pings search engines when new content is published. It ran 9 cycles over ~18 hours and detected the Vary header fragmentation bug that kept the site at 7.1% cache hit rate despite 10 cache rules being active.

What this actually found (real, verified)

The bridge read CF GraphQL analytics and detected:

• Cache hit rate: 7.1% on 176,128 requests/day (expected: 50%+ based on cache rules)
• The fix session (same day) confirmed root cause: Vary: rsc, next-router-state-tree, next-router-prefetch, next-router-segment-prefetch on every Next.js App Router response
• Cloudflare fragments the cache per unique Vary combination — real browsers send unique Next-Router-State-Tree JSON on every client navigation → permanent MISS
• curl tests showed HIT (curl doesn't send RSC headers), so manual testing completely missed this
• Fix: CF HTTP Response Header Modification rule — set Vary: Accept-Encoding only
• Result: all tested routes now show Vary: Accept-Encoding and cache correctly

Current production stats (from cf-bridge-output.json)

{
  "cycleCount": 9,
  "totalRedirects": 0,
  "lastCycleAt": "2026-03-27T15:42:32.091Z",
  "cacheHitRate": 8.1,
  "lastSitemapPingAt": "2026-03-27T07:04:37.281Z"
}

Note: totalRedirects: 0 because no tools have been merged yet (merged_into IS NOT NULL = 0 rows). The redirect push is implemented and tested, just not yet triggered by production data.

Prerequisites

• Cloudflare account with API token (CF_OPS_TOKEN in ~/.env-vault)
• Zone ID (CF_ZONE_ID)
• Node.js 18+
• For redirect push: PostgreSQL connection (DATABASE_URL)

Step 1: Read cache analytics from Cloudflare GraphQL

source ~/.env-vault    # loads CF_OPS_TOKEN and CF_ZONE_ID

YESTERDAY=$(date -u -d 'yesterday' '+%Y-%m-%d' 2>/dev/null || date -u -v-1d '+%Y-%m-%d')

curl -s -X POST "https://api.cloudflare.com/client/v4/graphql" \
  -H "Authorization: Bearer $CF_OPS_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"query\":\"{ viewer { zones(filter: {zoneTag: \\\"<span class="katex-error" title="ParseError: KaTeX parse error: Expected group as argument to &#x27;\&quot;&#x27; at position 15: CF_ZONE_ID\\\&quot;}̲) { httpRequest…" style="color:#cc0000">CF_ZONE_ID\\\&quot;}) { httpRequests1dGroups(limit: 1, filter: {date: \\\&quot;</span>YESTERDAY\\\"}) { sum { requests cachedRequests bytes cachedBytes } } } } }\"}" \
  | python3 -c "
import sys, json
d = json.load(sys.stdin)
groups = d.get('data',{}).get('viewer',{}).get('zones',[{}])[0].get('httpRequests1dGroups',[])
if groups:
    s = groups[0]['sum']
    total, cached = s['requests'], s['cachedRequests']
    pct = round(cached/max(total,1)*100, 1)
    print(f'Cache hit rate: {pct}% ({cached:,}/{total:,} requests)')
    print(f'Bytes: {round(s[\"bytes\"]/1024/1024/1024,2)} GB total, {round(s[\"cachedBytes\"]/1024/1024/1024,2)} GB cached')
    if pct < 50:
        print(f'WARNING: Cache rate {pct}% is below 50% threshold')
        print('Check: Vary headers, rule coverage, Set-Cookie on anonymous requests')
"

Expected output: cache hit rate percentage + bandwidth. If below 50%, investigate.

Step 2: Diagnose the Vary header problem (what we actually found)

echo "=== Vary Header Diagnostic ==="
for path in "/" "/tools" "/tool/chatgpt" "/blog"; do
  VARY=<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mi>c</mi><mi>u</mi><mi>r</mi><mi>l</mi><mo>−</mo><mi>s</mi><mi>I</mi><mi mathvariant="normal">&quot;</mi><mi>h</mi><mi>t</mi><mi>t</mi><mi>p</mi><mi>s</mi><mo>:</mo><mi mathvariant="normal">/</mi><mi mathvariant="normal">/</mi><mi>y</mi><mi>o</mi><mi>u</mi><mi>r</mi><mo>−</mo><mi>d</mi><mi>o</mi><mi>m</mi><mi>a</mi><mi>i</mi><mi>n</mi><mi mathvariant="normal">.</mi><mi>c</mi><mi>o</mi><mi>m</mi></mrow><annotation encoding="application/x-tex">(curl -sI &quot;https://your-domain.com</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mopen">(</span><span class="mord mathnormal">c</span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.0278em;">r</span><span class="mord mathnormal" style="margin-right:0.0197em;">l</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span class="mord mathnormal">s</span><span class="mord mathnormal" style="margin-right:0.0785em;">I</span><span class="mord">&quot;</span><span class="mord mathnormal">h</span><span class="mord mathnormal">ttp</span><span class="mord mathnormal">s</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">:</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord">//</span><span class="mord mathnormal" style="margin-right:0.0359em;">y</span><span class="mord mathnormal">o</span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.0278em;">r</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:0.6944em;"></span><span class="mord mathnormal">d</span><span class="mord mathnormal">o</span><span class="mord mathnormal">main</span><span class="mord">.</span><span class="mord mathnormal">co</span><span class="mord mathnormal">m</span></span></span></span>{path}" 2>/dev/null | grep -i "^vary:" | tr -d '\r')
  echo "<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mi>p</mi><mi>a</mi><mi>t</mi><mi>h</mi></mrow><mo>:</mo></mrow><annotation encoding="application/x-tex">{path}:</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">h</span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">:</span></span></span></span>{VARY:-no Vary header}"
done

echo ""
echo "Then test with RSC headers (what browsers actually send):"
echo ""

# Test #1: plain curl (what manual testing uses — shows HIT)
echo "Plain curl:"
curl -sI "https://your-domain.com/tool/chatgpt" 2>/dev/null | grep -i "cf-cache-status"

# Wait 2s and test again
sleep 2

# Test #2: with RSC headers (what browsers use for client navigation — shows MISS)
echo "With RSC headers (browser simulation):"
curl -sI \
  -H "RSC: 1" \
  -H "Next-Router-State-Tree: %5B%22%22%5D" \
  "https://your-domain.com/tool/chatgpt" 2>/dev/null | grep -i "cf-cache-status"

echo ""
echo "If plain=HIT but RSC=MISS: Vary fragmentation is your cache killer"
echo "If both show 'Vary: rsc, next-router-state-tree...': apply the transform rule below"

Step 3: Fix Vary fragmentation via CF Transform Rule (what we actually deployed)

This is the exact API call that fixed the site from 7.1% to projected 50%+:

source ~/.env-vault

# First check if a transform ruleset already exists
EXISTING=$(curl -s -X GET \
  "https://api.cloudflare.com/client/v4/zones/$CF_ZONE_ID/rulesets?phase=http_response_headers_transform" \
  -H "Authorization: Bearer $CF_OPS_TOKEN" \
  | python3 -c "import sys,json; rs=json.load(sys.stdin).get('result',[]); print(rs[0]['id'] if rs else 'NONE')")

echo "Existing transform ruleset: $EXISTING"

if [ "$EXISTING" = "NONE" ]; then
  # Create new ruleset (what we did — no prior ruleset existed)
  curl -s -X POST "https://api.cloudflare.com/client/v4/zones/$CF_ZONE_ID/rulesets" \
    -H "Authorization: Bearer $CF_OPS_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "name": "Strip Next.js Vary headers",
      "kind": "zone",
      "phase": "http_response_headers_transform",
      "rules": [{
        "expression": "(http.request.uri.path ne \"/api/health\")",
        "description": "Strip Next.js Vary headers for CF cache compatibility",
        "action": "rewrite",
        "action_parameters": {
          "headers": {
            "Vary": { "operation": "set", "value": "Accept-Encoding" }
          }
        }
      }]
    }' | python3 -c "
import sys,json
d=json.load(sys.stdin)
print('OK — ruleset ID:', d.get('result',{}).get('id')) if d.get('success') else print('FAILED:', d.get('errors'))
"
fi

Step 4: Push redirect rules for merged duplicate tools

# In production this reads from PostgreSQL:
# SELECT slug, merged_into FROM tools_db WHERE merged_into IS NOT NULL AND cf_redirect_created IS NULL

# Standalone version using a JSON input:
node << 'REDIRECTS'
const fs = require('fs');

// Replace with your DB query results
const mergedTools = [
  { slug: 'chat-gpt', merged_into: 'chatgpt' },
  { slug: 'gpt-4-turbo', merged_into: 'gpt-4o' },
];

const REDIRECTS_FILE = '/tmp/merged-redirects.json';
let existing = [];
try { existing = JSON.parse(fs.readFileSync(REDIRECTS_FILE, 'utf8')); } catch {}

const existingSources = new Set(existing.map(r => r.source));
const newRedirects = mergedTools
  .filter(t => !existingSources.has(`/tool/${t.slug}`))
  .map(t => ({
    source: `/tool/${t.slug}`,
    destination: `/tool/${t.merged_into}`,
    permanent: true,
  }));

const all = [...existing, ...newRedirects];
fs.writeFileSync(REDIRECTS_FILE, JSON.stringify(all, null, 2));
console.log(`<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mi>n</mi><mi>e</mi><mi>w</mi><mi>R</mi><mi>e</mi><mi>d</mi><mi>i</mi><mi>r</mi><mi>e</mi><mi>c</mi><mi>t</mi><mi>s</mi><mi mathvariant="normal">.</mi><mi>l</mi><mi>e</mi><mi>n</mi><mi>g</mi><mi>t</mi><mi>h</mi></mrow><mi>n</mi><mi>e</mi><mi>w</mi><mi>r</mi><mi>e</mi><mi>d</mi><mi>i</mi><mi>r</mi><mi>e</mi><mi>c</mi><mi>t</mi><mi>s</mi><mi>w</mi><mi>r</mi><mi>i</mi><mi>t</mi><mi>t</mi><mi>e</mi><mi>n</mi><mo stretchy="false">(</mo></mrow><annotation encoding="application/x-tex">{newRedirects.length} new redirects written (</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord"><span class="mord mathnormal">n</span><span class="mord mathnormal">e</span><span class="mord mathnormal" style="margin-right:0.0269em;">w</span><span class="mord mathnormal" style="margin-right:0.0077em;">R</span><span class="mord mathnormal">e</span><span class="mord mathnormal">d</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.0278em;">r</span><span class="mord mathnormal">ec</span><span class="mord mathnormal">t</span><span class="mord mathnormal">s</span><span class="mord">.</span><span class="mord mathnormal" style="margin-right:0.0197em;">l</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal" style="margin-right:0.0359em;">g</span><span class="mord mathnormal">t</span><span class="mord mathnormal">h</span></span><span class="mord mathnormal">n</span><span class="mord mathnormal">e</span><span class="mord mathnormal" style="margin-right:0.0269em;">w</span><span class="mord mathnormal" style="margin-right:0.0278em;">r</span><span class="mord mathnormal">e</span><span class="mord mathnormal">d</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.0278em;">r</span><span class="mord mathnormal">ec</span><span class="mord mathnormal">t</span><span class="mord mathnormal">s</span><span class="mord mathnormal" style="margin-right:0.0269em;">w</span><span class="mord mathnormal" style="margin-right:0.0278em;">r</span><span class="mord mathnormal">i</span><span class="mord mathnormal">tt</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mopen">(</span></span></span></span>{all.length} total)`);
console.log('Add to next.config.ts: async redirects() { return require("./lib/redirects/merged-redirects.json"); }');
REDIRECTS

Step 5: Ping search engines after new content published

SITEMAP_URL="https://your-domain.com/sitemap.xml"

# Google ping
curl -s "https://www.google.com/ping?sitemap=${SITEMAP_URL}" \
  -o /dev/null -w "Google ping: HTTP %{http_code}\n"

# Bing ping
curl -s "https://www.bing.com/ping?sitemap=${SITEMAP_URL}" \
  -o /dev/null -w "Bing ping: HTTP %{http_code}\n"

# Purge sitemap from CF edge cache
source ~/.env-vault
curl -s -X POST "https://api.cloudflare.com/client/v4/zones/$CF_ZONE_ID/purge_cache" \
  -H "Authorization: Bearer $CF_OPS_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"files\":[\"${SITEMAP_URL}\"]}" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print('CF purge:', 'OK' if d.get('success') else 'FAILED')"

Production cooldowns (from cf-bridge.js)

Lessons from production

1. curl tests are insufficient — they don't send RSC headers. Always test with -H "RSC: 1" too.
2. Vary fragmentation is invisible — the CF dashboard shows low cache rate but no explanation. You have to read response headers to find it.
3. 10 cache rules at free tier limit still = 7.1% rate — rule coverage is not the only factor. Vary headers can nullify every rule.
4. Transform rules fix what Cache Rules can't — Cache Rules control TTL; Transform Rules control headers. Both are needed.