Oceans LLM

Appearance

Request Lifecycle and Failure Modes ​

See also: Model Routing and API Behavior, Provider API Compatibility, Pricing Catalog and Accounting, Budgets, Observability and Request Logs, Request Logs, MCP Invocations, Configuration Reference, Identity and Access, Data Relationships, ADR: V1 Runtime Simplification for Routing and Streaming

This page is the cross-cutting view. Neighboring docs own their own policy slices. This page explains how those slices connect during one request.

Source of Truth ​

Request Path ​

The live request path is single-route in this slice.

  1. The HTTP middleware assigns one canonical request id from x-request-id or generates one when absent.
  2. The gateway authenticates the API key.
  3. The allowed gateway model set is reduced by API-key grants, principal-centric allowlists, and model-level allowlists.
    • For human user-owned keys, a model-level allowlist passes when the user email or effective team key is listed.
    • For service-account-owned keys, a model-level allowlist denies the model in v1.
  4. The requested model is resolved.
    • A concrete model key stays concrete.
    • A tag: selector picks one allowed gateway model; blocked allowlisted candidates are skipped.
    • An alias resolves to a canonical execution model after gateway-model authorization. Alias and target allowlists are independent.
  5. The route planner builds an ordered route list.
    • Lower priority wins first.
    • weight only matters inside the same priority bucket.
    • Disabled routes and non-positive weights drop out.
  6. Capability filtering removes routes that cannot satisfy the API family and feature requirements. For example, /v1/responses requires responses, while /v1/chat/completions requires chat_completions.
  7. The budget guard runs after access and model resolution, before provider execution.
    • hard-limit rejection returns 429 budget_exceeded
    • no provider call occurs on this path
  8. Route compatibility metadata is passed into the provider adapter.
  9. The provider adapter applies any declared compatibility transforms to the outbound provider request.
  10. The first eligible route executes.
  11. The provider execution attempt is recorded as request-attempt metadata when a request-log summary is written.
  12. MCP tool invocations, when enabled for the request path, are logged as separate request-correlated audit records.
  13. Request logs are written for the user-visible outcome.
  14. Usage is normalized when possible.
  15. Pricing is resolved exactly or the request is marked unpriced.
  16. When the request has usable usage data, the gateway prepares the ledger row, applies post-provider budget math, and commits the priced or unpriced ledger row in that order.

Compatibility transforms can affect the provider request body and stream options for the selected API family. They do not change the public request model identity, alias resolution, API-key grants, or request-log attribution. Current OpenAI-compatible profile transforms are Chat Completions-specific; Responses and Embeddings use their own typed provider paths.

Worked Example ​

One common request path looks like this:

  • Request:
    • POST /v1/responses
    • API key belongs to service account growth-indexer
    • model is tag:fast
  • Access:
    • the API key grant allows gpt-4o-mini and claude-3-5-haiku
    • the owning team allowlist is unrestricted
  • Resolution:
    • tag:fast resolves to gateway model gpt-4o-mini
    • gpt-4o-mini is an alias of openai-gpt-4o-mini
  • Planning:
    • openai-gpt-4o-mini has two routes in config
    • route A has priority 50
    • route B has priority 100
    • route A wins before weight is considered
  • Capability filter:
    • the request asks for the Responses API family, no tools, no vision
    • route A stays eligible
  • Execution:
    • the provider request goes to the route A provider and upstream model through the Responses adapter
  • Logging:
    • request_logs.model_key stores gpt-4o-mini
    • request_logs.resolved_model_key stores openai-gpt-4o-mini
    • request_logs.provider_key stores the route A provider id
  • Accounting:
    • usage is normalized
    • pricing resolves exactly
    • usage_cost_events.pricing_status becomes priced
    • the service-account budget window includes the charge

Vertex Embeddings Example ​

Native Vertex text embeddings follow the same cross-cutting lifecycle as other /v1/* requests:

  • Request:
    • POST /v1/embeddings
    • model is gemini-embedding
    • input is ["query text", "document text"]
  • Resolution:
    • gemini-embedding resolves to a route with provider: vertex-global
    • upstream_model is google/gemini-embedding-001
  • Capability filter:
    • the route has embeddings: true
    • unrelated families such as chat_completions and responses are disabled on that route
  • Provider execution:
    • the Vertex adapter validates text-only input and supported parameters
    • the adapter calls Vertex :predict for legacy text-embedding models or :embedContent for google/gemini-embedding-2, fanning out array input when needed while preserving output indexes
  • Logging:
    • request logs record operation: embeddings
    • request-log attempts record the Vertex provider execution attempt when a summary row is written
  • Accounting:
    • Vertex provider token counts are aggregated into prompt/input token usage (statistics.token_count for :predict, usageMetadata.promptTokenCount for :embedContent)
    • exact Google Vertex pricing produces a priced ledger row
    • user, service-account, and matching user-model budget windows include the charge

Important failure examples:

FailureLifecycle pointOutcome
Gemini chat route has embeddings: falsecapability filteringThe route is removed before Vertex execution.
Route uses unsupported google/gemini-2.0-flash for embeddingsprovider support validationThe request fails with a deterministic invalid-request error instead of treating all google/* models as embedding-capable.
input is a token array, nested array, non-string, empty array, or empty stringprovider-local request validationThe request fails locally before the upstream call.
encoding_format: "base64"provider-local request validationThe request fails locally because native Vertex embeddings return float vectors only in this slice.
Vertex omits token countsaccountingThe request can still succeed, but the ledger status is usage_missing and budgets are not consumed.
Pricing catalog lacks an exact embedding model/rateaccountingThe request can still succeed, but the ledger status is unpriced and budgets are not consumed.

Model Visibility Versus Execution ​

A model can be visible and still fail at runtime.

  • /v1/models shows grant-visible gateway identities.
  • /v1/models does not promise that a route is executable right now.
  • Route viability still depends on:
    • provider existence
    • route enabled
    • positive weight
    • capability match
    • pricing readiness, if spend accuracy matters for the request path

This is why a model can appear in /v1/models and still fail with invalid_request or no_routes_available.

Failure Classes ​

These failures look similar from far away, but they mean different things.

invalid_request ​

  • The model resolved.
  • Capability filtering removed every remaining route.
  • Common causes:
    • embeddings against a chat-only route
    • Responses requests against a route with responses: false
    • tools against a route with tools disabled
    • vision against a route that does not advertise vision

no_routes_available ​

  • The model exists.
  • No usable route survived provider and route-viability checks.
  • Common causes:
    • missing provider id in the live config
    • all routes disabled
    • all routes have non-positive weight

budget_exceeded ​

  • A pre-provider hard-limit check blocked the request.
  • The HTTP response is 429.
  • No provider call occurs on this path.
  • Observability records this as a request outcome rather than as provider execution.

unpriced ​

  • The provider request succeeded.
  • Usage exists.
  • Exact pricing could not be resolved.
  • Common causes:
    • unsupported pricing_provider_id
    • unsupported Vertex publisher or location
    • unsupported billing modifiers
    • missing exact rate coverage

unpriced requests stay visible in reports but do not count toward budget totals.

usage_missing ​

  • The provider request succeeded.
  • Usage could not be normalized into the gateway accounting model.
  • The request log still records the user-visible outcome.
  • The ledger row stays visible, but it does not count toward spend totals.

Logging and Ledger Boundaries ​

Request logs and spend rows are related, but they are not the same object.

  • request_logs owns the user-visible request outcome.
  • request_log_payloads owns sanitized request and response bodies.
  • request_log_attempts owns ordered upstream provider execution metadata.
  • MCP invocation records own per-tool status, policy result, latency, and sanitized argument/result metadata.
  • usage_cost_events owns spend enforcement and spend reporting.

That separation matters in two common cases:

  • a request can be logged even when it becomes unpriced
  • a request can be logged even when a later accounting step hits a rough edge

For streaming requests, the request-log payload path parses SSE incrementally across UTF-8 and frame boundaries and retains the latest coherent usage snapshot seen before stream completion or failure. Chat Completions streams usually expose usage at top level. Responses streams expose usage on completed response events as response.usage. Bedrock streaming is normalized to OpenAI-compatible SSE before this logging path; see AWS Bedrock for adapter-specific frame and error mapping details. Bedrock Claude reasoning deltas are preserved as explicit provider metadata, not as normal delta.content, so request-log payload policy controls whether that debug metadata is stored. Stored stream events can be capped by payload policy without weakening usage or provider-error parsing.

Known Rough Edges ​

  • Request-log payload policy details, redaction rules, and retention status are owned by observability-and-request-logs.md.
  • Configurable retry and fallback execution is not part of the current request path; see issue #118.

For the current observability cleanup notes, see observability-and-request-logs.md.

What This Page Does Not Own ​