Local resource

Observability

_docs/OBSERVABILITY.md

Observability

Identity certainty

Log flow_identity_certainty_applied with only snapshot ref, attribution mode and fixed-cardinality subject counts, including deterministic no-activity runs. Invalid broker input logs flow_run_identity_certainty_rejected with the allowlisted error code and terminal action. Never log subjects, source identifiers, names, JIDs, phones, raw side-contract JSON, prompt fragments, or model output while abstention is active.

FlowRunResult.input_diagnostics.identity_certainty and broker metadata.identity_certainty use the same sanitized ACK: contract/policy, snapshot ref, mode and counts. Dashboards should compare accepted V2 requests, invalid-contract terminal failures and Core-side quarantines. Alert on any unexpected invalid-contract burst or Core quarantine; do not label metrics by tenant, monitoring, snapshot or subject.

Langfuse is the only observability path for AI model requests.

Los fixtures de regresion de degradaciones no son observabilidad ni contienen traces productivos. El importador no realiza requests de modelos y no escribe en Langfuse. Solo versiona fingerprints tecnicos aprobados y expected results completamente sinteticos; ver _docs/DEGRADATION_REGRESSION_FIXTURES.md.

Scope

Tracked as Langfuse generation observations:

  • Flow AI JSON steps: smoke, initial report, chat id summarization, daily summary and notifications, interweek report.
  • Vision model calls for images, static stickers/GIFs, legacy video frames, and embedded document images.
  • Whole-video Gemini calls for video and animated stickers/GIFs.
  • Audio transcription model calls.

Not tracked as separate model observations:

  • adapter normalization
  • Temporal scheduling
  • conversation builder
  • response builder
  • local file parsing
  • link metadata fetch

Runtime logs still use structlog for operational status and errors.

Link preview processing emits the Prometheus counter cis_link_processing_total{outcome,error_code,cache_status} once per URL. The labels are deliberately bounded: outcomes are fetched, unsupported, degraded or failed; unknown error and cache values collapse to other. URLs, domains, tenants, monitorings, requests and message IDs are never metric labels. cache_status=hit includes callers sharing an in-flight fetch, even when that fetch fails; only the caller that starts the fetch records miss. The same accounting applies to the per-activity cache counts. This includes HTTP 429: in-flight waiters record hit, while subsequent reads of the stored negative entry record negative_hit.

Expected degradations, including HTTP 429 with caching disabled, emit link_processing_degraded (negative-cache hits do not repeat the warning). Besides the existing bounded execution identifiers, the event includes error_code, outcome, the configured byte limit, observed and declared byte counts when available, content_type, redirect count, HTTP status and the bounded rejection stage (content_length, stream, http_status or unknown). It never includes the source/final URL, domain, query string, headers or fetched text. HTTP-status rejections include redirect count and declared size before reading the body; their content type is recorded only when it belongs to the downloader's explicit allowed types, never as arbitrary upstream header text. The first HTTP 429 retains these safe fields through negative-cache insertion. They remain internal cache metadata, are stripped from public link results, and do not cause repeated warnings on negative-cache hits or in-flight waiters.

Grafana should use Prometheus for rates and the runtime log backend for single-run investigation. Suggested panels:

sum by (outcome, error_code) (rate(cis_link_processing_total[15m]))
sum(increase(cis_link_processing_total{outcome="degraded"}[15m]))
/
clamp_min(sum(increase(cis_link_processing_total[15m])), 1)

An initial warning rule may require both a degradation ratio over 10% and at least 20 attempts in 15 minutes. Keep those thresholds in Grafana/alerting configuration and tune them from production baselines; a single expected degradation must not page.

Rollout coordination emits quiesce_startup_waiting, quiesce_startup_released, api_work_rejected_while_quiesced, broker_quiesce_requested, broker_shutdown_requested, broker_ingress_drained, worker_shutdown_requested, and worker_drain_failed_during_cancellation. Correlate them with the bounded operation ID, component/worker, signal and stable reason only. /health exposes the additive quiesce snapshot without failing liveness; /ready returns 503 while quiesced. Never log marker contents beyond those validated fields.

Report PDF isolation

GET /health includes the additive report_pdf component with the spawn-per-render strategy, capacity, active/available slots, oldest active render, hard deadline, and available|saturated|degraded status. Saturation is normal backpressure and does not change the top-level health status or HTTP 200. If the supervisor cannot reap a worker, degraded changes the top-level status and HTTP response to 503 so readiness fails closed until the process is replaced.

The PDF supervisor emits report_pdf_admission_rejected, report_pdf_complexity_rejected, report_pdf_worker_timeout, report_pdf_worker_cancelled, report_pdf_worker_crashed, report_pdf_worker_reap_failed, and report_pdf_worker_completed. These events contain only bounded operational identifiers, durations, byte counts, and outcomes; they never contain Markdown, generated PDF bytes, monitoring display names, or extracted report text.

The initial-preview upgrade page is deterministic. Category summaries, taxonomy labels, pricing copy and omitted-media detail are never added to logs or Langfuse. Control Plane observes only bounded category/media availability states and the fixed pricing contract version.

Trace Attributes

Trace names use the compact service prefix CIS, for example CIS - Initial Ingestion Report Flow.

Custom tags are intentionally small and groupable:

  • flow:<flow_id>
  • stage:<step_id>
  • source-media:<audio|image|sticker|gif|video|document> for media model

    • calls only

Do not duplicate Langfuse-native fields as tags. Latency, session id, user id, environment, version, and model are provided by Langfuse columns/fields.

Generation Metadata

Every generation includes:

  • serviceName
  • flowId
  • recipeId
  • stepId
  • promptDisplayName
  • promptPath
  • promptSha256
  • modelRequested
  • returned model/provider when available
  • modality/source/media type details for media calls
  • runId for media calls, allowing every image, frame, and transcription to be

    • correlated back to its Temporal execution

  • structured-output request details such as responseFormat, jsonSchemaName,

    • and providerRequireParameters when applicable

  • costSource when model cost was calculated or supplied by the gateway

Generation model_parameters should only contain runtime knobs useful for quick comparison, such as temperature, max tokens, or vision detail. Protocol details stay in metadata instead of becoming chips in the Langfuse header.

Operational logs use workflow_start_deduplicated, temporal_workflow_input_rejected, and temporal_activity_input_compacted to distinguish repeated delivery, deterministic payload rejection, and bounded payload compaction without logging conversation text.

Report auto-split adds structured operational events without adding Langfuse generations: report_preflight_completed, report_split_planned, report_partition_completed, report_split_reduce_prepare_started, report_split_reduce_prepared, report_split_reduce_started, report_reduce_completed, report_split_result_resolved, report_split_inline_fallback and report_split_failed. Correlate them with run_id, execution_mode, bounded byte/count fields and the truncated plan digest. They may include partition/conversation/message/media counts, configured limits, estimated prompt tokens and artifact bytes, but never conversation text, prompts, source URLs or model output. The five split activities also emit the required activity_completed/activity_failed events with activity=flow and a stage name. report_split_failed may expose only the stable report_split_media_url_ttl_insufficient or report_split_media_url_expiry_unverifiable code plus bounded ordinals/counts; it must never include the source URL, signature, token or query string.

Prometheus exposes the same canary boundary with fixed, non-sensitive labels: cis_report_autosplit_preflight_total, cis_report_autosplit_workflows_total, cis_report_split_partitions, cis_report_split_payload_bytes, cis_report_split_activity_total, and cis_report_split_activity_duration_seconds. API, worker and broker processes serve the registry on a separate CIS_METRICS_PORT listener when it is non-zero; that operational listener must be published only on the private interface. Each container needs a distinct private port and scrape target. Histogram buckets are configurable through the three REPORT_SPLIT_*_HISTOGRAM_BUCKETS variables so deployments can keep them aligned with payload budgets and activity timeouts. Tenant, monitoring, request, workflow, chat and message identifiers are deliberately excluded from labels.

Age-band normalization emits monitored_age_band_context_processed with only flow_id, recipe_id, the consumer flag and the bounded outcome (applied, absent, unknown, invalid or disabled). It never logs DOB, exact age, band value, monitored identity or raw side-contract data. The band is not added to Langfuse tags or generation metadata; when applied, it exists only inside the protected prompt section required for personalization.

Report-feedback normalization emits report_feedback_context_processed with only flow_id, recipe_id, the consumer flag, a bounded outcome (applied, applied_with_omissions, absent, empty, invalid or disabled) and input, applied and omitted item counts. It never logs tenant/feedback/report IDs, summaries, tags, source values, timestamps or internal statuses. The event is the operational counter source for report_feedback_context_processed_total{flow,outcome}; item-count dashboards aggregate its numeric fields. Feedback data is not added to Langfuse tags or generation metadata.

Audio preflight logs audio_activity_degraded plus a successful degraded activity_completed with error_code=no_audio_track when the media has no audio stream. No Langfuse generation is created because no model is called. Technical probe failures emit one audio_activity_failed error with failure_stage=audio_probe, the stable audio_probe_failed:* code and one traceback per attempt; they remain retryable by Temporal.

The reusable-media resolver emits activity_completed with activity=flow, stage=media_insight_resolve, run/batch and fixed outcome counts plus latency on every successful or fail-open return. It never logs cache keys, source URLs or media content.

The preceding CP-to-CIS boundary is observable without content: media_insight_workflow_start_returned records submitted media/envelope counts, whether a tenant scope was supplied, and whether Temporal used a strictly-new or new-or-existing idempotent start policy. It is an attempt signal, not proof that a new workflow consumed the payload. media_insight_wave is the durable proof that a workflow selected eligible envelopes. If CP reports emitted envelopes while no start-returned event appears, inspect broker/request validation. If start returned but no wave appears, first account for an idempotent existing execution; then inspect tenant propagation and recipe overrides. If waves are eligible, inspect activity_completed stage=media_insight_resolve outcomes before expecting CP traffic: off and observe modes intentionally bypass the network, and invalid processor signatures bypass every entry. Only a valid write/read batch with no CP resolve request points to text-processing worker registration or Temporal activity delivery.

activity_completed stage=media_insight_finalize records the terminal callback outcome (ready or failed) for a claimed provider result. This Activity is independently retryable, so repeated attempts with the same opaque claim ref are at-least-once callback delivery rather than repeated provider requests. CP owns the durable gauges and alerts for active pending, expired pending, recent failure, ready, and hit populations.

Broker workers emit structured Kafka lifecycle events: group join/rejoin and partition revocation, transport timeout, coordinator loss, heartbeat expiry, transport unavailability, and commit loss after a published result. kafka_consumer_restart_scheduled exposes bounded in-process recovery attempts. kafka_consumer_group_rejoined includes the measured rebalance_duration_ms; all events include the consumer group, client id, and topic names but never Kafka message values. broker_split_result_pending is an expected retryable checkpoint, not a failed report: it records the poll window, last Temporal status and action=leave_offset_uncommitted_for_idempotent_redelivery; no result or diagnostic envelope is published for that event. Cache pressure may emit run_repository_single_result_exceeds_cache_budget for the protected current result while older terminal entries are evicted.

When vision retries a provider capability mismatch without detail, successful results set model_trace.fallback_used=true and generation metadata visionDetailFallbackUsed=true. A persistent mismatch records the stable vision_provider_capability_mismatch code and visionDetailFallbackAttempted=true. Operational logs include only identifiers, model id and provider exception type; they never include the request, data URL, image bytes or provider response body.

When the provider omits only the required visual description but supplies non-empty OCR text, vision_structured_output_text_fallback records the stable vision_text_only_fallback code. The media result and model trace are marked as degraded/fallback without logging the OCR text or raw provider response.

Usage And Cost Tracking

Flow text and vision calls send Langfuse-native usage_details with input, output, and total token counts. They also send cost_details with input, output, and total USD costs. If OpenRouter returns usage.cost, that value wins for cost_details.total; otherwise the service uses the local pricing table in services/pricing.py, which should be refreshed from OpenRouter's models API when model ids change. If OpenRouter returns a provider-specific version alias that is not priced locally, input/output cost attribution falls back to the requested model while keeping usage.cost as the exact total when present.

Audio transcription calls send usage_details.input_audio_seconds and total using Groq billable seconds. Groq ASR has a 10 second minimum per request. cost_details.input and cost_details.total use the Groq per-hour ASR rates in services/pricing.py; cost_details.output is 0.0.

The private wardian-audio-transcription-tool.v1 endpoint always overrides model I/O capture to off, regardless of the global debug setting. Its Langfuse generation may retain technical attribution such as model, duration, status and cost, but never the tutor transcript, object key or media bytes. CP exposes only bounded technical counters for requested, completed, degraded, failed and cleanup-failed guidance audio operations.

Langfuse dashboard widgets may round small costs visually. There is no project setting for decimal precision in the current Langfuse UI. For sub-cent values, inspect the observation detail/table columns, export CSV/JSON, or query Langfuse metrics/observations APIs instead of relying on the Home dashboard big-number format.

Video generations additionally record mediaResolution, returned request id, token usage and the actual OpenRouter cost when available. Operational activity_completed logs include backend, source_media_type, media_resolution, local animated_media_fps, latency, cost and fallback status. No video bytes or data URLs are persisted.

File-backed media downloads emit safe_http_download_completed, safe_http_download_failed or safe_http_download_cancelled. Fields are limited to activity, outcome, downloaded/configured bytes, redirect count, normalized content type, latency, whether a trusted hash was verified, cancellation and a stable error code. They never include source/final URLs, query strings, signed tokens, headers, temporary paths, SHA-256 values or media content. Expected cancellation is logged without a traceback; technical errors are still re-raised for the Activity retry policy.

Model I/O Capture

LANGFUSE_CAPTURE_MODEL_IO=true can include request/response content when privacy policy allows it. Data URLs are redacted before they reach Langfuse. Audio transcriptions include Groq's returned transcript and normalized segments in the generation output when model I/O capture is allowed. Groq does not return a hosted playback URL for the uploaded audio; when the original message has a public url_archivo, the service includes it as sourceAudioUrl in Langfuse. Locally extracted video audio and other temporary files do not get a synthetic playback URL.

Exception: monitored-conversation audio with AUDIO_EVIDENCE_GATE_MODE=shadow|enforce forces model I/O capture and raw debug off. Its structured completion log contains only the typed decision/reason codes and fallback state; it must not add transcript, source URL or raw identifiers. The private tutor-guidance audio tool remains on its separate contract and already forces capture off.

Production protected deployments set LANGFUSE_CAPTURE_MODEL_IO=false; raw prompt/response capture is not a substitute for the short-lived evidence artifact. Any exceptional override must use retention no longer than 24 hours. The model gateway emits a sanitized execution record (model ids, prompt digest, tokens, cost, latency, finish reason, retries and status) through an optional sink. It never emits prompt text through that sink.

Before projection, flow_result_surface_measured records canonical/gzip bytes, string-leaf count, UTF-8 text bytes and hash-derived duplicate text bytes for the fixed result classes output, primary, shadow, documents, and media. Labels are limited to flow, negotiated profile, class, transport and status; text values and hashes are never exported. Build p50/p95/p99 histograms from these numeric fields and alert if a minimal wire result contains any debug class or exceeds output + 48 KiB before outer externalization.

flowchart LR
  A["AI step"] --> B["OpenRouter/Groq request"]
  B --> C["Langfuse generation"]
  C --> D{"capture model I/O?"}
  D -->|yes| E["system/user, JSON output, or ASR transcript with media bytes redacted"]
  D -->|no| F["summary metadata only"]

Content trust boundary

The optional evaluator is an ordinary Langfuse generation named content_trust_evaluator; protected mode follows the same model-I/O capture policy as every other request. Operational events expose only policy/mode, pass|review|block, allowlisted reason codes, evaluator status, flow/recipe and run/request identifiers. They never log conversation text, OCR, transcript, document/link extraction, prompts, model free text or the output body. CP exports counters for pass/review/block, invalid assessments and withheld memory/delivery side effects.

Shadow Behavior

Flow AI steps call:

  • Primary google/gemini-3.5-flash-lite.
  • Shadow deepseek/deepseek-v4.1-flash.

Both calls are traced independently. Shadow failure is marked as a failed generation and does not fail the run.

Prompt Management

Use:

python scripts/sync_langfuse_prompts.py --dry-run
python scripts/sync_langfuse_prompts.py --label production

The sync script only publishes active prompts listed in prompts/INDEX.md.

Temporal evidence

Run results may expose sanitized aggregate input diagnostics under input_diagnostics.temporal_evidence. Do not add content, IDs/JIDs, worker keys or exact event timestamps to logs/traces. Model prompts never receive the raw side-contract. Only enforce/strict guarantee that rendered conversation evidence is temporally eligible; shadow/disabled are observational and may render supplied messages whose timestamps remain untrusted.

Raw WAHA normalization emits bounded compatibility counts under input_diagnostics.waha_compatibility. Failed no-message batches preserve the same aggregate-only block in structured HTTP or broker-v3 failure diagnostics; free-form event names and payload values are never copied into it.

Flow cancellation

Structured events:

  • flow_cancellation_command_rejected: invalid JSON, schema, timestamp, key or signature; contains only a safe error code.
  • flow_cancellation_completed: command id, opaque target fingerprint, fence version, outcome and cleanup status.
  • flow_run_blocked_by_cancellation_fence: broker stopped before hydration/start/publish.
  • canceled_run_late_completion_discarded: a local completion tried to overwrite terminal cancellation.

Alert on cancellation consumer restart loops, CP fence endpoint failures, receipt lag, cleanup_status=partial, and CP cancellation dead letters. Target fingerprints are opaque correlators and must not be expanded back to tenant/request identifiers in logs or receipts.

Checkpointed report progress

Checkpointed reports publish PII-free content-intelligence.flow-progress.v1 events with a monotonic sequence across preparing, main_step, partitioning, assembling and result_ready. They carry only aggregate partition/chat counts, checkpoint reuse, ambiguous retries, calls, token usage, cost, elapsed time and ETA. Final trust evaluator usage is included in the last sample; chat/message/actor IDs, names, prompts and outputs are excluded.

A workflow active after the broker observation window is pending, not failed. Alert after 10 minutes without progress, P1 after 30 minutes or profile SLA breach, and P1 for any ambiguous retry or result-digest conflict. model_output_truncated records the bounded technical fields step_id, model, finish reason and configured output budget without logging model output. The Langfuse generation retains provider usage and cost, finishes at error level, and the broker result exposes error_code=output_truncated, retryable=false and failure_stage=model_output_generation. Prometheus also increments cis_model_output_truncated_total{flow,step,model}; any post-policy-v2 sample is an operator warning and two samples in fifteen minutes are incident-worthy.

Optional longitudinal consistency

Longitudinal finalization records bounded flow/mode/outcome counters, structural failures, omitted-event counts and latency. Logs exclude message text, refs and baseline content. The offline CLI emits machine claims/status/digest only. Validation failures retain only closed technical codes: private exception causes and implicit contexts are detached before standard tracebacks or Temporal failure serialization. This also covers direct claim/transition schema construction in the pure finalizer. See Report consistency v1 for metric names and commands.

Offline age-band invariance

The four-band synthetic gate preserves risk/taxonomy, alert structure, evidence and complete memory, including negotiated v2 ACK and deterministic concern sidecars. Its closed narrative comparator and sanitized diagnostics are described in AGE_BAND_INVARIANCE.md. Passing the offline gate does not establish real-model semantic invariance or authorize a paid canary.

model_prompt_unicode_repaired reports stage, surrogate_pairs_combined and isolated_surrogates_replaced, without prompt content. model_failure_record_failed reports a best-effort execution-record failure without masking the original error.

Raw Unicode in retained results

Completed Temporal results may retain isolated surrogates from original media analysis. The run cache measures lossless escaped JSON, and result-surface metrics escape unencodable JSON code points while counting raw string bytes with surrogatepass. These are size-only operations: the cached evidence, identifiers, contract fingerprints and canonical output hashing are unchanged. Broker JSON serialization already escapes Unicode and can republish the completed result without restarting the workflow. A failure in size accounting must not trap a completed workflow in an ingress restart loop.