AI時代のObservability設計 - アプリケーション / インフラ / CI / LLMすべてを監視する（設計編）

comments (12)

1. Kenvia dev.to2026-07-07

   The sink-vs-pump framing in the comments is strong. I’d separate “is it hot right now?” telemetry from “what evidence changed the decision?” records. Prometheus-style counters are great for current pressure, but decision/evidence records need to survive as structured facts: which call site, which source, which retry path, what was repaired, what was trusted, and what changed the downstream result. That split matters even more once AI is consuming the telemetry. The trace is useful, but the facts that explain the decision need to outlive the prompt transcript.

   1. Ryosuke Tsujivia dev.to2026-07-07

      You're pointing at exactly the split the whole harness is built around, though I'd push the durable half one step further than the record itself. The fix landing as a PR is the first layer of that: a PR is a durable evidence record, which source, which repair, what a separate pass verified, all of it outlives the transcript by design. The transcript is what the model saw, the PR is what actually grounded the decision, and those two diverge the moment anything gets retried. We also index those PRs into a separate release graph, so the evidence isn't just archived, it's queryable over MCP later: which change touched what, why it merged, what it was linked to. The record stays a live fact instead of a dead log. But the record alone still only tells you what happened once. The second layer is that every self-healing fix is required to add a lint or type gate as part of the same PR, so the fix doesn't just leave a trace, it leaves a constraint. The same class of failure can't recur, because the evidence got compiled into a rule the next run has to satisfy. I wrote about that recurrence loop in more depth [here](https://dev.to/ryantsuji/fixed-before-anyone-notices-stronger-after-every-fix-self-healing-recurrence-prevention-series-1e86) if you're curious. That's the part I care about most: the decision doesn't just survive as a fact you can reconstruct later, it survives as something the system now enforces.

2. Laksmana Tri Moerdanivia dev.to2026-07-07

   The part about per-call-site cost attribution is where I want to push back, because I don't think the design as written can actually produce what it claims. You write about wanting to slice by call-site context: "the db-graph table description generation cost $X," "that one prompt cost $Z." But gemini.cost.usd is a Prometheus counter carrying model, service, and type labels. The instant service holds the call-site identity you actually want to slice by, the cardinality goes sideways. A 46-repo codebase pushing field type inferences and table descriptions, each a distinct label value, produces series counts Prometheus is not built to absorb. And if service stays coarse to stay safe, then "the db-graph table description cost $X" stops being answerable from the counter, and you're left with what Cloud Billing already hands you. I learned this the expensive way. Same shape of problem, different vendor. The meter looked healthy, dashboards rendered confident numbers, and somewhere around week three the scrape started timing out because the label space had quietly grown past what the TSDB could chew. The cost chart kept moving on the wall while the storage underneath was choking. So the three escape hatches I've seen work: keep the counter coarse and ship per-call-site detail to BigQuery as structured records (which is the same backend you chose for Claude Code, for the same reason), or attach exemplars with trace IDs so the time series stays clean and a human can drill down on a spike, or just admit that "which prompt cost the most this week" is a SQL question wearing a PromQL costume. The post frames Gemini and Claude Code as wanting different backends. My read is they want the same backend, for a reason the article doesn't quite state. Gemini goes to Prometheus because you can wrap the call. But "I can instrument this call" and "this should live as a time series" are two separate claims, and the cardinality ceiling is exactly where the second one breaks from the first. One thing I'd genuinely like to know: is service in your labels coarse, like a fixed handful of services, or is it carrying a call-site dimension I'm not seeing in the post? Because if it's coarse, the real-time dollar number is real and useful, but the "tune what you can attribute" payoff is softer than the framing makes it sound. And if it's not coarse, I'd love to see how you're keeping the cardinality from eating the TSDB, because that's the part I never solved cleanly.

   1. Ryosuke Tsujivia dev.to2026-07-07

      You're right, and you've caught the article overstating what the counter can do. Let me give you the concrete answer. service is coarse. It's a bounded set of app/pipeline names, meet-pipeline, gcs-transformer, db-dictionary, code-graph, and so on, defaulting to the Cloud Run service name. The gemini.cost.usd counter carries exactly two labels, model and service. There's no call-site dimension on it, precisely because I'd hit the ceiling you're describing if I tried. So your read is correct on both counts. The real-time dollar number is real and useful, but the "which single prompt cost $Z" line in the post promises finer attribution than the counter delivers. That's a framing error on my part, not a thing the counter does. Where I'd push back slightly is on what the Prometheus side is actually for in practice. It isn't the aggregation surface at all. It's a tripwire. The counter exists so sum(increase(...[1h])) crossing a threshold pages Slack, so a runaway prompt or batch gets caught in minutes instead of the next morning's billing. Coarse service is enough for "something in gcs-transformer is on fire right now." That's the job it's doing, and it does it well. The per-call-site question, "which prompt burned the most this week," is exactly the SQL-shaped question you describe, and you're right that it wants the same backend as Claude Code. The token records can land in BigQuery with as much call-site context as I want to attach, no cardinality ceiling, and that's where that question should be answered. The article draws the Gemini/Claude split at "can I wrap the call," but your point is sharper: "can I wrap it" and "should it live as a time series" are different questions, and the second one is where the two converge back onto BigQuery. That distinction is better than the one I published, so I've gone and fixed that section of the post to say what the counter actually does. Thanks for the push.

3. juan gonzalezvia dev.to2026-07-07

   Lo que más me gusta de este hilo es que todos los ejemplos convergen en la misma idea: las comprobaciones más fiables se hacen en el extremo del sistema, no en el origen. La conciliación del coste, el recuento esperado de ejecuciones de CI, la frescura de una tabla ETL... todos son casos de la misma regla: verificar la evidencia, no asumir que el proceso funcionó. Creo que ese patrón va incluso más allá de la observabilidad. Es una regla de diseño para sistemas con IA: cada capa debería producir evidencia verificable para la siguiente, y ninguna debería confiar únicamente en que la anterior "dice" haber tenido éxito. Al final, la confianza no emerge porque un componente sea fiable, sino porque cada afirmación deja un rastro independiente que otro componente puede comprobar.

   1. Ryosuke Tsujivia dev.to2026-07-07

      That last line is the whole thing: trust doesn't come from a component being reliable, it comes from every claim leaving an independent trace something else can check. That generalizes further than I framed it in the post. It's basically the same reason the AI review layer works in what's coming in Part 2. The model's output isn't trusted because the model is good, it's trusted because it lands as a diff that a separate pass, and a human if needed, can verify against the graph. Same rule, one layer up. Every claim leaves a trace.

4. Lior Karaevvia dev.to2026-07-07

   The "watch the sink, not the pump" framing from the comments thread is the sharpest distillation of the whole piece. The CI post-hoc pull design is the one I'd steal immediately. The structural guarantee that PR code never touches the Grafana API key isn't just a security win — it's an architectural clarity win. When the shipping workflow itself becomes observable, you've turned a silent failure mode into a monitorable service. That's the kind of design that pays forward for years. The BigQuery vs Loki decision for Claude Code usage is also the right call for a reason slightly different from what's stated. Claude Code on a subscription means the real question isn't cost per token — it's understanding usage patterns well enough to justify the subscription tier or negotiate it. SQL aggregation is exactly the right shape for that question. One thing I'd add to the LLM layer: tracking cache hit ratio per repository rather than just aggregate. The 96% you're seeing overall is impressive, but if two repositories are pulling the aggregate up while three others are cold-missing consistently, the fix (CLAUDE.md tuning, prompt restructuring) is very different per repo. Curious whether you're already slicing it that way.

   1. Ryosuke Tsujivia dev.to2026-07-07

      Yeah, we slice per repo. Interesting thing is the spread is tighter than your hypothesis assumes. Nothing's cold-missing here, pretty much everything clears 90%, and cortex itself sits at 97.7%, which for a repo this size I think is genuinely high. So the per-repo cut mostly confirms the fleet is healthy rather than surfacing outliers to fix. Where it would earn its keep is exactly the scenario you describe, a couple of repos dragging while others quietly starve, and I'd want that visible the moment it happens rather than discovering it in the aggregate. Your subscription reframe is spot on though. The question isn't cost per token, it's whether usage justifies the tier, and SQL aggregation is the right shape for that.

5. Vinicius Pereiravia dev.to2026-07-06

   The pricing-table choice is the one place I'd add a tripwire. Computing cost client-side from a GEMINI_PRICING constant buys you real-time visibility and per-call-site granularity, but it also creates a second copy of a fact the provider owns, and duplicated facts drift: the day the price sheet changes, every dashboard keeps rendering confident, specific, wrong dollar numbers, and nothing errors. The fix is cheap, a scheduled job that reconciles your computed spend against the billing API's actuals at day granularity and alerts when the delta exceeds rounding. You keep the inline numbers and the reconciliation catches the quiet divergence before finance does. Same logic as any denormalized cache: the copy is fine as long as something keeps checking it against the source of truth. The decoupled CI shipping has a similar quiet failure worth naming. If the post-hoc pull breaks, the outcome is not an error, it's an empty Loki, and an empty failure stream reads as good news, silence is the one signal every dashboard renders as health. Worth emitting an expected-volume check (N runs happened in GitHub, N landed in Loki, the diff should be zero) so missing data becomes a loud fact instead of a comfortable one. Really solid writeup, the match-the-backend-to-the-question framing is the right way to think about AI-consumable telemetry, and the identity-stamped-once-at-the-edge design is quietly the best part, one attestation point instead of every client asserting who it is. Looking forward to the PII and self-healing half.

   1. Ryosuke Tsujivia dev.to2026-07-07

      Both of these are correct, and you've named the thing they have in common better than the article did. A client-side computed number is a denormalized copy of a fact the provider owns, and a denormalized copy is only safe if something keeps checking it against the source of truth. That's the frame, and I didn't have it in the post. On the pricing table: no, there's no reconciliation job today. The price constant just sits there, and you're exactly right that the failure is silent and confident, which is the worst kind. It's a gap I've been aware of and haven't closed, and "reconcile computed spend against billing actuals at day granularity, alert when the delta exceeds rounding" is the right shape for the fix. Cheap, and it turns a quiet divergence into a loud one before finance finds it. Good catch. The CI one is the same shape on the transport side, and it's the better catch because the failure is even quieter. An empty failure stream doesn't just render as health, it renders as the outcome you were hoping for. Part 2 (next week) spends its closing section on this exact family of problem, but framed at the code entrance: swallowed exceptions, errors logged at info level, the faucet being broken so nothing flows in. What your comment adds is the transport-side version I didn't cover, where the faucet is fine but the pipe between it and Loki broke. The expected-volume check (N runs in GitHub, N landed in Loki, diff should be zero) is the missing piece there. We do run that final-layer freshness idea for our ETL pipelines, monitoring the last table rather than trusting the source succeeded, but the CI shipping path specifically doesn't have it, and it should. The identity-stamped-once observation is the part I'm happiest someone noticed. One attestation point instead of every client asserting who it is was the whole reason for that shape. Thanks for reading this closely, and the PII/self-healing half lands next week.

   2. Vinicius Pereiravia dev.to2026-07-07

      The final-layer freshness you already run on ETL is the general form of both fixes, honestly: watch the sink, not the pump. Whether the pump is a price constant, a CI shipping workflow, or a source job, the only signal that cannot lie by omission is the destination containing what it should by now. And your reframe is the sharper half of this thread: an empty failure stream does not just read as health, it reads as the outcome you were hoping for, and hoped-for readings are exactly the ones that should cost evidence. If the comfortable interpretation requires a matching count, and the uncomfortable one is the default, the dashboard fails toward truth instead of toward comfort. Looking forward to part 2.

   3. Ryosuke Tsujivia dev.to2026-07-07

      Nice way to put it, "watch the sink, not the pump." I knew the cost side was a weak spot but hadn't worked out how to close it, and tying it to the CI case makes the fix obvious.