AI時代のObservability設計 - PIIとAIの検索性を両立させ、自動修復する（実践編）

comments (16)

1. Mudassir Khanvia dev.to2026-07-15

   the entry point design gap is where this lands for me. the stack is solid but the faucet is still owned by humans and lint, and that's the actual ceiling for autonomous operation. pattern i've seen work: type the log call site directly. if logger.error requires an Error object and not a string, dropped stacktraces stop being a guideline violation and become a type error the compiler catches before CI. the lint rule enforces the type, not the convention. is that the direction you're thinking for the harness, or does the problem look different from where you're sitting?

   1. Ryosuke Tsujivia dev.to2026-07-15

      Yeah, that's the direction, and I already run a weaker version of it. Lint doesn't warn here, it blocks. A catch that swallows without logging fails the build, and so does a log call that drops the stacktrace. Typing the call site so logger.error takes an Error and not a string is the stronger form of the same move, and I agree it's better. It moves the same block from CI lint to the compiler, which is strictly earlier and harder to skip past. But that closes the half that was always mechanizable. "The stacktrace got dropped" is a correctness property, and correctness properties belong in the type system. The ceiling I meant sits one level up, and types don't reach it. It's deciding whether a condition is an error at all. Concrete case from this week. A scheduled importer walks a list of articles and one returns 404 because it isn't published yet. Is that an error? I decided no, it's an expected skip, log a warning and exit 0. A rate-limit exhaustion on the same loop is an error, exit 1. Nothing in the type system tells you which is which. logger.error taking an Error forces me to log a real Error once I've decided it's an error, but the decision that a 404 here means "not ready" rather than "broken" is a claim about what the absence means in the business, and that claim is the faucet. So the framing I'd land on is that you push every mechanizable property down to the compiler precisely so the only thing left at the top is the irreducible classification call. Your typed-call-site move is how you clear the mechanizable layer. What stays human isn't "did we handle the error right," it's "is this even an error." That one doesn't compile.

2. Yunetzivia dev.to2026-07-15

   Smart take on PII: hash at write and search time so AI can find what it needs without exposing data. Curious about collision handling and performance overhead in practice. Also loved the CI-to-PR self-healing angle—chef's kiss.

   1. Ryosuke Tsujivia dev.to2026-07-15

      Thanks, really glad the self-healing angle landed. On both counts the honest answer is "bounded, so neither really bites." Collisions are birthday-bound against a population capped by the log retention window, not an ever-growing set, so the space stays comfortably clear at our scale. If that ever stops being true, the collision constant just becomes a retention knob someone's already watching. Performance is a non-issue in the same spirit. It's one hash on write and one on search, which is nothing next to the query and the model cost it sits beside. The expensive part of an investigation was never the hash.

3. Mike Czerwinskivia dev.to2026-07-14

   The resolve-inside-the-MCP-server design is the right structural move, and the reasoning for rejecting the admin-screen alternative is the most honest part of the post, most writeups don't admit they considered the human-in-the-loop version and picked autonomy over it on purpose. The birthday-bound math checks out for the customer base you have today, but it's computed against today's size, not the trust boundary's actual lifetime. 20M records at 50% collision sounds comfortably far off, until the company grows for five years and nobody revisits the constant. A correlation key that degrades gracefully at current scale can still degrade silently at future scale, because nothing in the design re-checks the assumption, it was true when written and nobody's watching whether it's still true. The other thing I didn't see addressed: what happens on HMAC key rotation. If the key ever needs to rotate, either every historical log becomes uncorrelatable with new ones (a quiet loss of the "look up Customer A's history" capability you built the whole six-layer design to preserve), or the old key has to be retained somewhere to re-hash on read, which reopens exactly the key-outside-the-stack boundary the enumeration-resistance argument depends on. Worth stating which of those two you'd actually do, because right now the design reads as if the key never rotates, and "never rotates" is a strong assumption for a secret whose whole job is staying secret.

   1. Ryosuke Tsujivia dev.to2026-07-14

      Both land, and the second one (rotation) is a real gap in the post, the design as written does read as if the key is immortal, and you're right that "never rotates" is a bad assumption for a secret. The thing that resolves both of your points is one I didn't make explicit: logs aren't kept forever. They rotate out on a retention window, and once you lean on that, the clean design is to put the hash and the key on the same clock as the logs they serve. Concretely: tie a key's lifetime to the retention window of the logs hashed under it. When you rotate, you keep the old key only as long as logs hashed with it are still alive, and re-hash on read against whichever key covers that log's era. The moment those logs age out, the old key is deleted with them. That dodges both horns of your dilemma, correlation survives within a log's lifetime because the covering key is still there, and the key never accumulates or lives forever outside the stack because it dies when its logs do. It also softens your first point as a side effect. The birthday-bound math is against a bounded population, not an ever-growing one, because retention caps how many records coexist. The collision assumption stops being a constant nobody revisits and becomes a function of the retention window, which is a number someone is already looking at. Full honesty: this is the design, not the current state. It's early enough that the key doesn't rotate yet and retention-coupling isn't wired in, so today it genuinely is the immortal-key version you called out. You named the exact thing that has to get built before rotation is ever needed. Good catch, both of them.

   2. Mike Czerwinskivia dev.to2026-07-14

      Tying the key to the retention window is the clean fix for the immortal-key problem, and I think it exposes a tradeoff the post's original goal has to answer next. If each era gets its own key, the same email hashes to a different value in era N and era N+1. That's fine for a single log lookup, but it breaks the exact capability the six-layer design was built to preserve: look up Customer A's whole history. A search spanning two eras can't recognize the same person across the key boundary without re-hashing every candidate against every live era key and matching independently, which turns a hash lookup back into a fan-out. So the honest question is whether cross-era correlation is a deliberate loss (search only ever answers within-this-retention-window, which may be fine, most investigations don't need five years back) or whether there's a second, longer-lived index key that exists precisely to survive rotation, in which case that key inherits the immortal-key problem you just solved for the log-hash key, just moved one layer over. Worth stating which one this is, because right now the design reads as having quietly traded correlate-forever for correlate-within-a-window without saying so out loud.

   3. Ryosuke Tsujivia dev.to2026-07-14

      It's the window, and I'd argue the window is the right boundary rather than a quiet downgrade, because it falls out of the same retention clock instead of being a separate decision. Concretely, I wouldn't build a second long-lived index key, that just moves the immortal-key problem one layer over like you said. I'd set the key lifetime equal to the retention window, which caps the number of live keys at two: any log still alive was hashed under either the current key or the previous one, because anything older than one rotation has already aged out of Loki. So a lookup hashes the input under at most two keys and ORs the results. The fan-out you're describing is real but fixed at 2x, not an unbounded re-hash. Hide that behind the MCP tool for agents and the API layer for humans, and the caller still asks "Customer A's history" once. So it's not correlate-forever and it's not correlate-within-one-key. It's correlate-as-far-back-as-the-logs-still-exist, which is the honest ceiling anyway: you can't correlate history that's already been rotated out of Loki, key or no key. The retention window was always the real limit on how far back a lookup can see. Tying the key lifetime to it just makes the crypto boundary agree with the boundary that already existed, and keeps the live-key count at two by construction. You're right it should be stated out loud, though, "correlation is bounded by retention, by design" is the line the post is missing.

   4. Mike Czerwinskivia dev.to2026-07-14

      Fixed at 2x and derived from the same retention clock instead of a separate decision is the version that actually closes it, correlate-as-far-back-as-the-logs-still-exist is the honest ceiling and tying the crypto to it removes an assumption instead of adding one. Worth stating the one edge case before this ships: at the exact rotation boundary, is there a moment where a log written a second before rotation and read a second after needs a key that's already been retired, or does retention lag rotation by enough margin that this never actually happens in practice? If the rotation and the retention-based deletion aren't atomic with each other, you could briefly need a third key or have zero keys covering a thin sliver of logs right at the seam. Probably a non-issue if rotation cadence is much shorter than retention window, but worth the one-sentence guarantee in the design doc so nobody has to rediscover it during an incident. Good exchange, this is the kind of thread that's worth linking back to when the immortal-key question comes up again.

   5. Ryosuke Tsujivia dev.to2026-07-14

      Right on the seam, and here's the one-sentence guarantee: the only logs that can fall into a zero-key gap are the ones already at the retention edge, i.e. the ones being aged out anyway. A log written a second before rotation and read a second after is, by definition, a log that's crossing out of the retention window at that same moment, so "can't find a key for it" and "it's past retention" collapse into the same case. As long as deletion never runs ahead of rotation (retention lags rotation, never leads it), the sliver only ever contains logs that were already leaving. That makes the guarantee "no live log is ever without a covering key," which is the line for the design doc, exactly as you said, so nobody rediscovers it mid-incident. This whole thread was a genuine pleasure. You pushed on the two things I'd have most regretted leaving implicit, retention-coupling and the rotation seam, and both are sharper in the post now because of it. Linking back here when the immortal-key question resurfaces is exactly right. Thanks, Mike.

4. Vinicius Pereiravia dev.to2026-07-14

   The dual-sided HMAC is the right shape for keeping plaintext out, but I'd name what it moves rather than removes. A deterministic hash is a stable pseudonym: same email, same value, which is what makes search work and also what lets anyone with query access reconstruct one person's whole history under that token. Plaintext egress is closed; linkability is not. And the search side is an online oracle: it hashes arbitrary input and looks for a match, so anyone who can query can confirm whether an email they already suspect is in the logs, one guess at a time. HMAC stops offline brute force of the stored hash, but the search endpoint answers the membership question for free. Neither breaks your design, they just belong in the threat model beside it: the boundary is real for plaintext, and does not cover correlation or confirmation. For a known-domain field like email, that gap is where re-identification actually lands.

   1. Ryosuke Tsujivia dev.to2026-07-14

      Both correct, and both worth naming explicitly in the threat model. But they land on the other side of the boundary this design draws, so let me make the boundary itself explicit, because I left it too implicit in the post. The searcher here is assumed to already have PII access. Support and engineering can already see this person's data in the DB; that's their job. What this design removes is not their ability to correlate or confirm, they already have it, it's the need to route plaintext PII through the model to do a log investigation. The boundary is "does the AI ever see the plaintext," not "can an authorized human correlate." So the linkability you describe (one stable pseudonym reconstructs a history) and the online oracle (confirm-by-query) are both real, but they're capabilities the searcher already holds by virtue of DB access. The hash doesn't grant them; it just lets the same authorized person do the log side without handing the email to the model. Where your framing sharpens mine: for someone who has query access but not DB access, those two gaps would be a genuine escalation, and that's exactly the case to guard. We keep the search side auditable (every resolve + query is logged) precisely so that "authorized human doing their job" and "someone fishing the oracle" are distinguishable after the fact. That's a detection control, not a prevention one, and you're right that for a known-domain field like email, confirmation is the sharp edge. Naming it beside the plaintext boundary is the honest way to draw the diagram, and I'd rather have both lines on it than pretend the boundary covers more than it does.

   2. Vinicius Pereiravia dev.to2026-07-14

      Agreed on the boundary, and the audit trail is the right call for the human-without-DB case. The one place it thins is the direction your post is driving toward. Detection works because a person fishing the oracle looks anomalous, but an agent doing self-healing queries at volume by design, so a confirmation probe hides inside its own normal traffic. The baseline is the anomaly you would otherwise flag. For that persona I would pair the audit log with a prevention control on the search side, scoping a lookup to a case context or capping distinct identifiers per session, so an authorized query is bounded and not just logged. Good exchange.

   3. Ryosuke Tsujivia dev.to2026-07-14

      You're right that detection thins exactly where the series is heading, and that's the sharp version of it. Once the querying persona is an agent doing self-healing at volume, "anomalous" stops being a usable signal because volume is the baseline. Where I'd push on the framing: an agent firing a confirmation probe isn't really the agent's malice, it's almost always an agent that's been prompt-injected into doing it. And once injection is on the table, the oracle is a small downstream symptom, the same compromised path can exfiltrate through any tool it can reach, not just the hash lookup. So I'd put the prevention control at the injection layer (tool-permission scoping, input provenance, output review) rather than teaching the PII search side to count distinct identifiers. Capping the oracle while injection is unhandled feels like bolting one window shut. That said, your per-session bound is the right move as defense-in-depth, and whether it's worth it is a product call. For a domain where a single re-identification is catastrophic (health, finance), a second wall on the search side that holds even after injection succeeds is worth the cost. For an internal platform where the searcher and the agent are both inside the trust boundary, I'd spend that budget on hardening the injection surface first. Good exchange, genuinely, this is the kind of threat-model back-and-forth that's hard to get.

   4. Vinicius Pereiravia dev.to2026-07-14

      You're right that injection is the root and the oracle is one symptom, harden the injection surface first, I'm with you there. The one thing I'd keep beside that is why the search wall earns its place: tool-scoping, provenance, and output review are all probabilistic, their false-negative rate is unknown and the attacker gets to lower it. The per-session bound is the one control whose guarantee doesn't move when injection gets better. So I'd frame the product call less by domain and more by that, put the deterministic wall wherever a probabilistic guarantee failing is unacceptable, which is often wider than just health and finance. Good exchange, genuinely.

   5. Ryosuke Tsujivia dev.to2026-07-14

      That reframe is the version I'll keep: put the deterministic wall wherever a probabilistic guarantee failing is unacceptable. Cleaner than drawing it by domain, and it generalizes past security. Thanks for pushing on every layer of this, genuinely one of the best exchanges I've had here. Enjoy the sand in Niterói.