litellm Security & Engineering Audit
You shipped it.
Is it safe to operate?
We audited litellm across 9 engineering areas - 80 findings, 25 of them critical or high. Here's the picture.
https://github.com/BerriAI/litellm
9 areas evaluated
Each area scored 0-10 by an AnchorStack engineer.
Code Quality
4.6/10
LiteLLM has a well-structured and expanding test suite with strong CI quality gates, but is dragged down by several god files, pervasive type-safety bypasses, and widespread mutable global state. The ruff strict-budget ratchet and 20+ custom CI quality checks show intentional investment in quality discipline; however, files like proxy/utils.py (5,758 lines), litellm_core_utils/litellm_logging.py (5,526 lines), proxy/_types.py (3,827 lines), and proxy_server.py (3,180 lines) represent major maintenance liabilities. The codebase is expensive to change safely: the absence of strict MyPy mode, 1,649+ type: ignore suppressions, and 1,885 Any usages across trust boundaries mean many type errors are hidden rather than fixed.
Architecture
5.4/10
LiteLLM is a dual-purpose open-source system: a Python SDK providing a unified interface to 100+ LLM providers, and an AI Gateway (proxy) that adds authentication, rate limiting, budget enforcement, caching, and routing on top of it. The architecture is a modular monolith deployed as a Docker container or via Helm to Kubernetes, with FastAPI + async Python as the web layer, APScheduler for in-process background jobs, DualCache (in-memory + Redis) for rate limiting and key caching, and PostgreSQL via Prisma for persistent state. The layering concept is sound and the provider translation pattern is genuinely elegant. However, the system cannot currently handle its workload without correctness risk: spend data lives in an in-memory queue with a 60-second flush cycle, so budget enforcement reads stale state and pod crashes silently lose spend records. proxy_server.py at 16,000+ lines is a severe god module that collapses startup, config, scheduling, and HTTP routing into a single file, making the system difficult to test, evolve, or operate at the boundaries where things break.
Data
4.3/10
The data layer is a single PostgreSQL database accessed via Prisma ORM, with a well-considered schema covering org/team/user/key spend hierarchy, audit logs, daily aggregated spend, MCP server configs, guardrails, and workflow tracking. Migration discipline is solid for the enterprise path (100+ versioned Prisma migrations) but the default community path ships with `prisma db push --accept-data-loss`, which can silently destroy data. The most critical gaps are: LLM provider API keys and MCP credentials stored as plaintext JSON with no application-level encryption; no backup or restore evidence for self-hosted deployments; and spend data silently lost on pod crash because the durable Redis path is opt-in. Multi-tenant isolation relies entirely on the application layer with no Row Level Security safety net, and Float is used throughout for monetary values, introducing precision risk in financial accounting.
Security
5.3/10
LiteLLM is an open-source AI gateway with a substantive authentication layer (JWT with RBAC and PKCE SSO, timing-safe master-key comparison, SSRF guards, and parameterized SQL throughout), but it carries a critical unresolved risk in its credential store: LLM provider API keys and MCP secrets are persisted as plaintext JSON in PostgreSQL with no application-level encryption. Secondary gaps compound the exposure: the production container runs as root, the session JWT cookie lacks HttpOnly/SameSite/Secure flags, JWT audience validation is off by default, there are no HTTP security response headers, no rate limiting on authentication endpoints, and vulnerability scanning is limited to OSV and GitGuardian in CI with no SAST, no container scanning, and no pre-commit hooks. All multi-tenant isolation is enforced exclusively in application code with no Postgres Row Level Security backstop; a single authorization bug is the only barrier to cross-tenant data access. The system can protect users against opportunistic threats but is not adequately hardened for production deployments handling third-party credentials, regulated data, or high-value tenants.
AI Engineering
5.3/10
LiteLLM has invested meaningfully in AI-assisted engineering: CLAUDE.md is specific, project-scoped, and covers code style, testing philosophy, supply chain safety, commit conventions, and proof-of-fix requirements in unusual depth. The validation gate stack (Ruff, MyPy, Black, Semgrep, secret scanning, conventional-commits CI) is strong and largely required on every PR. However, the AI workflow has three structural gaps that compound risk: a direct contradiction between CLAUDE.md branch-naming instructions and the pre-push hook will cause Claude Code to produce branches that fail automation; there is no spec-driven design step before AI code generation; and there is no knowledge lifecycle (no ADRs, no memory, no retrospectives), so lessons from past AI-assisted work are never captured. Overall the team is using AI in a way that improves engineering leverage, but the contradictory context and absent knowledge infrastructure mean silent quality drift is likely as the AI footprint grows.
Infrastructure
4.3/10
LiteLLM's infrastructure is a tale of two deployment paths. The Kubernetes/Helm path is production-grade: reproducible multi-stage builds on pinned Chainguard Wolfi SHA256 digests, cosign image signing, a pre-upgrade Prisma migration job with ArgoCD/Helm hook support, HPA/KEDA/PDB hooks, and graceful shutdown support. The simpler paths — Docker Compose, CloudFormation EC2+RDS, and Azure Container Instances ARM template — are developer-quickstart artifacts with hardcoded credentials, mutable image references, and no security hardening. The critical gap across every path is the complete absence of PostgreSQL backup configuration: no managed backup, no pg_dump job, no PITR guidance, and the CloudFormation RDS instance defaults to zero backup retention. The default Helm standalone PostgreSQL ships with a known weak password, the production container runs as root with empty securityContext defaults, no CPU or memory resource limits are set (preventing HPA from functioning), and Redis is disabled by default, which silently degrades distributed rate limiting and spend durability to per-pod only. The infrastructure cannot be rebuilt, recovered, or scaled safely under its current default configuration without substantial operator knowledge not captured in any shipped artifact.
Observability
4.7/10
LiteLLM ships a rich telemetry layer (Prometheus metrics, OpenTelemetry tracing, Sentry, Slack/PagerDuty alerting, Datadog logs and metrics, structured JSON logging, and full health endpoints), but virtually all of it is opt-in and requires explicit operator configuration. Out of the box, a freshly deployed proxy has no error tracking, no external uptime monitor, unstructured console logs, and no alerting, so a silent failure will go undetected until a user complains. The highest-risk gaps are: LLM prompt/response content flowing to every configured log sink by default (PII exposure risk), no heartbeat monitoring for the critical background jobs that drive spend tracking and budget enforcement, and a complete absence of documented SLOs or uptime targets.
Delivery
5.8/10
LiteLLM's delivery posture is mature for an open-source distribution project, with strong GitHub Actions and CircleCI automation, enforced conventional commits, a clear branching guard on main, and excellent dependency discipline including SHA-pinned CI actions and OSV lockfile scanning. The primary risks are: the production Docker image build and push pipeline is not present in the repository (opaque provenance and no auditable signing chain from source to published image), no feature flag system exists for controlled rollouts of auth, spend-tracking, or LLM-routing changes, CodeQL and GitHub Actions security SAST only gate PRs targeting main and bypass the primary litellm_internal_staging development path, and there is no documented rollback runbook or post-deploy smoke test for self-hosted deployments.
Documentation
5.1/10
A new contributor can understand LiteLLM's purpose and get a development environment running thanks to a solid README, a detailed ARCHITECTURE.md (Mermaid diagrams, request flow, provider translation, background job inventory, data-access conventions), and a clear CONTRIBUTING.md. For an operator running LiteLLM in production, the picture is materially worse: the primary user-facing docs live in a separate repository (BerriAI/litellm-docs) with no local mirror, no operational runbooks exist anywhere in the repo, no decision records capture why key tradeoffs were made, and the only AI context file for non-Claude tools (AGENTS.md) is a one-liner stub. The .env.example contains hardcoded development credentials (master key sk-1234, DB password dbpassword9090) with no production-readiness warning. litellm/proxy/README.md is significantly stale (wrong port, missing most of the management endpoint directories). CLAUDE.md is the single strongest documentation artifact in the repository, but it is written for AI coding agents rather than human operators or new engineers. A maintainer who was not present when the system was built can read the architecture and understand what the code does; they cannot safely operate or recover the system from the documentation alone.
Where to look first
01
— Security
LLM provider API keys and MCP secrets stored as plaintext in PostgreSQL
LiteLLM_CredentialsTable.credential_values (Json) and LiteLLM_MCPServerTable.credentials (Json) store provider API keys and MCP OAuth tokens in cleartext; per-user BYOK credentials use base64, not encryption.
A database breach, misconfigured Postgres permission, or SQL injection exposes every LLM provider credential (OpenAI, Anthropic, Azure, etc.) and every MCP OAuth token across all tenants in one SELECT. An attacker with DB read access can impersonate any configured provider or MCP server. The blast radius scales with the number of tenants and providers. This is the highest-value target in the database for an attacker.
Encrypt credential_values before write using an application-layer key stored outside the database (env-var-loaded AES-GCM key, or a KMS-managed key via an existing secret manager integration). Store only ciphertext; decrypt on read in the management layer before returning to callers. Rename credential_b64 to credential_enc and store AES-GCM ciphertext. Rotate all existing credentials immediately after encryption is deployed.
02
— Code Quality
God files in proxy layer make changes unsafe
Four files in the proxy layer each exceed 3,000 lines and bundle unrelated concerns, making every change a high-blast-radius operation.
Any edit to PrismaClient, ProxyLogging, or request routing must touch files that also contain email logic, caching, admin endpoints, and spend tracking. Merge conflicts, accidental regressions, and review fatigue are constant. Automated tools (MyPy, test isolation) cannot reason about these files effectively.
Extract PrismaClient and ProxyUpdateSpend from proxy/utils.py into dedicated modules under litellm/proxy/db/. Split ProxyLogging into separate callback-dispatch and alert-routing modules. Decompose proxy_server.py into sub-routers (one file per route group) using FastAPI APIRouter. The proxy/_types.py monolith should be split into domain-scoped type files.
03
— Code Quality
Star imports from _types.py obscure the public surface of 28 modules
28 proxy files use `from litellm.proxy._types import *`, making it impossible to know which names are in scope without reading a 3,827-line file.
Any rename or deletion in _types.py silently breaks callers. IDE autocompletion and static analysis are ineffective. Adding new names to _types.py pollutes all 28 namespaces, raising the risk of shadowing and accidental re-export. The auth module (user_api_key_auth.py) is one of the consumers, making auth logic particularly hard to trace.
Replace all `from litellm.proxy._types import *` with explicit named imports. Introduce domain-scoped type modules (e.g., litellm/proxy/types/auth.py, litellm/proxy/types/spend.py) so each file imports only what it needs. Run `ruff --select=F401,F811` to catch any resulting dead or duplicate imports.
04
— Code Quality
1,885 Any usages and 1,649 type: ignore suppressions across trust boundaries
The ruff strict budget documents 1,885 ANN401 violations; separately, over 1,649 type: ignore comments silence MyPy across 250+ files.
Trust boundaries (auth, spend tracking, LLM API responses, DB writes) rely on untyped parameters. A mistyped spend value, auth object, or routing parameter passes through silently. The high type: ignore count means MyPy CI gives a false green — it passes but does not actually enforce type correctness on the majority of the codebase.
Enable mypy --strict incrementally using per-module ignores. Add a CI gate that counts type: ignore and enforces a ratchet similar to ruff-strict-budget.json. Prioritize removing Any from functions in litellm/proxy/auth/, litellm/proxy/db/, and litellm/proxy/spend_tracking/ first, as these handle sensitive data flows.
05
— Code Quality
301 complex functions and 1,813 over-argument functions currently tracked but uncapped
The ruff-strict-budget.json ceiling for C901 (complexity) is 301 and for PLR0913 (too many arguments) is 1,813 — meaning new code can add violations up to these baselines without breaking CI.
Functions exceeding cyclomatic complexity thresholds are statistically correlated with defect density. With 1,813 functions taking more arguments than the configured threshold, calls are likely to have argument ordering mistakes. The budget ratchet is not yet enforcing descent on these two rules — the baselines have never been lowered.
Run `make lint-strict-budget-update` to take a fresh snapshot and commit lower baselines. Add PR-level diffs that fail if either rule count increases (already supported by the `--base` flag in ruff_strict_gate.py). Prioritize refactoring the top-10 highest-complexity functions (identifiable via `ruff check --select C901 --output-format json`).
06
— Architecture
In-memory spend queue loses data on pod crash
Spend accumulated between flush cycles is silently discarded when a pod exits.
A pod handling high-cost LLM requests that crashes between 60-second flush cycles loses all spend accumulated in its in-memory SpendUpdateQueue. The DB records undercount actual spend, keys appear under-budget when they are not, and billing/quota reconciliation is wrong. In cloud environments where pod restarts are routine (OOMKill, rolling deploys, node preemption), this is a persistent accuracy problem rather than a rare edge case.
Make spend durable before acknowledging the LLM response: write to Redis atomically before returning to the client (the RedisUpdateBuffer path already exists but is opt-in via use_redis_transaction_buffer). Make that path the default so spend survives pod crashes. Separately, add a pod-shutdown hook that flushes the in-memory queue before the process exits (GracefulShutdownManager already exists; ensure update_spend runs during it).
07
— Architecture
Budget enforcement reads spend state that lags by up to 60 seconds
Budget checks compare against a DB-backed spend value that is stale by up to one flush cycle.
auth_checks.py reads valid_token.spend from the cached key object, which reflects the PostgreSQL row last written by update_spend (up to 60 seconds ago). At 10 requests/second each costing $0.01, a key can overspend by $6 beyond its budget before enforcement fires. The race is additive across pods: two pods each accumulating unflushed spend will both see the same stale DB value and both allow traffic up to the same apparent budget headroom. This is not hypothetical - it is the documented design and affects every budget-limited key in production.
Track in-flight spend per key in Redis using atomic INCRBYFLOAT immediately on each response (before the batch flush), and check this Redis-resident counter rather than (or in addition to) the DB-resident spend field during budget enforcement. This converts the enforcement check from a stale-read to a real-time one without changing the batch-write architecture for PostgreSQL.
08
— Architecture
proxy_server.py is a 16,000-line god module
The proxy entry point owns startup, config, scheduling, lifecycle, and 173 functions in a single file.
proxy_server.py (16,362 lines, 173 functions/classes, 550+ import lines) conflates FastAPI app setup, startup event orchestration (ProxyStartupEvent), APScheduler initialization, YAML config loading, DB migration checks, health check logic, and dozens of HTTP endpoint handlers. Any change to startup order, scheduler configuration, or a single endpoint requires touching this file. Testing any component in isolation requires importing this entire module and all its side effects. The APScheduler memory leak (documented in comments as causing 35GB allocations) was introduced and patched inside this file, invisible to reviewers who only touched endpoint code.
Extract ProxyStartupEvent into its own module. Move the scheduler setup into a dedicated scheduler module. Move endpoint handlers into their appropriate router files (many already exist but proxy_server.py still contains inline chat_completion, embedding, etc. handlers). The goal is a proxy_server.py that only instantiates the FastAPI app and includes routers - all logic lives in dedicated modules that can be imported and tested independently.
09
— Data
LLM provider API keys stored as plaintext JSON in the database
LiteLLM_CredentialsTable.credential_values is a Json column containing provider API keys with no application-level encryption before the write.
A database breach, a misconfigured Postgres permission, or an admin SQL console dumps all LLM provider credentials (OpenAI, Anthropic, Azure, etc.) in cleartext. Any internal user or service account that can SELECT from the table gets all credentials across all organizations. The blast radius scales with the number of tenants and providers configured.
Encrypt credential_values before writing and decrypt after reading using an application-level key stored outside the database (e.g., a KMS-managed key or an env-var-loaded AES key). Prisma does not natively support column-level encryption, so the encrypt/decrypt must happen in the management endpoint layer before calling prisma.create/findFirst. Rotate all existing credentials after encryption is deployed.
10
— Data
MCP server credentials and environment variables stored without encryption
LiteLLM_MCPServerTable.credentials (Json) and .env (Json) store MCP authentication secrets and environment variables without application-level encryption; LiteLLM_MCPUserCredentials.credential_b64 uses base64 encoding, not encryption.
A SELECT on LiteLLM_MCPServerTable yields OAuth tokens, API keys, and environment secrets for every configured MCP server. LiteLLM_MCPUserCredentials stores per-user BYOK credentials as base64, which is trivially decodable. Any internal actor with DB read access can impersonate any user against any MCP server.
Apply the same application-level encryption approach as for LiteLLM_CredentialsTable. Rename credential_b64 to credential_enc and store AES-GCM ciphertext rather than base64. Document the key management scheme (env var, KMS) so operators know how to rotate the encryption key.
11
— Data
Default schema migration path uses `--accept-data-loss` and can silently drop data
PrismaManager.setup_database() defaults to `prisma db push --accept-data-loss --skip-generate`, which allows Prisma to drop columns or alter types to match the schema even if doing so destroys existing data.
An operator upgrading LiteLLM by restarting the container on the default community path will have their database schema silently mutated. If a new schema removes or renames a column that has data, that data is gone without warning. This is particularly dangerous for the spend, metadata, and configuration tables which users rely on for billing and audit.
Remove `--accept-data-loss` from the default setup_database path. Gate schema changes behind `prisma migrate deploy` (already supported via ProxyExtrasDBManager) and surface a clear error if migrations are pending rather than auto-applying with data loss. Document the upgrade path in the operator guide.
12
— Data
No backup or restore evidence for self-hosted Postgres deployments
The repository contains no backup configuration, restore procedure, PITR documentation, or any restore test evidence for the self-hosted Docker or Helm deployment paths.
Operators running the default self-hosted deployment have a bare Postgres instance with no automated backup. A hardware failure, accidental DROP, or ransomware event has no recovery path. Spend data, API keys, user configurations, and audit logs are permanently lost. The SpendLogsPartitionManager can speed up retention cleanup but provides no disaster recovery.
Add a Helm values section and Docker Compose example that configures pgBackRest or pg_dump + S3. Provide an operator runbook covering: backup schedule, retention policy, restore procedure, and a test-restore checklist. Consider documenting Postgres-managed-service options (RDS, Cloud SQL) that include automated backups as the recommended path for production deployments.
13
— Data
In-memory spend queue is the default; pod crash silently loses spend data
The SpendUpdateQueue holds unflushed spend increments in process memory; the durable Redis buffer path requires opt-in via `use_redis_transaction_buffer`, so the default deployment loses all spend accumulated between flush cycles on pod restart.
Under the default configuration, a pod restart (OOMKill, rolling deploy, node preemption) silently discards up to 60 seconds of spend data accumulated in SpendUpdateQueue. Budget counters in Postgres undercount actual usage, allowing keys to exceed their configured limits. Billing and quota data is permanently incorrect with no reconciliation path. In cloud environments where pod restarts are routine, this is a persistent accuracy problem rather than an edge case.
Make the Redis buffer path the default when Redis is configured, rather than opt-in. Add a pod shutdown hook that flushes the in-memory queue to Redis (or directly to Postgres) before the process exits. Document GracefulShutdownManager's interaction with the spend flush so operators understand the shutdown guarantee.
14
— Security
Production container runs as root with no non-root user
The Dockerfile sets USER root in the runtime stage and never drops privileges before ENTRYPOINT, so the process serving live traffic runs as UID 0.
If any code execution vulnerability (RCE in a dependency, prompt injection through AI Skills, path traversal) is exploited, the attacker gets root inside the container. Combined with a hostPath mount, privileged securityContext, or weak Kubernetes RBAC, this escalates to node or cluster compromise. Running as root also means any file the process can create has root ownership, complicating forensic attribution.
In the runtime stage of the Dockerfile, add `RUN adduser -D -u 1000 litellm` (Wolfi/Alpine syntax), `RUN chown -R litellm:litellm /app /root/.cache/prisma /root/.cache/prisma-python`, then `USER litellm` before `ENTRYPOINT`. Update the Helm chart securityContext to set `runAsNonRoot: true`, `runAsUser: 1000`, and `allowPrivilegeEscalation: false`.
15
— Security
Session JWT cookie set without HttpOnly, SameSite, or Secure flags
The SSO callback sets the `token` cookie via `redirect_response.set_cookie(key='token', value=jwt_token)` with no security attributes; the cookie is readable by JavaScript, sent over HTTP, and has no CSRF protection.
Any XSS vulnerability in the admin UI allows an attacker to read the session JWT from document.cookie and take over admin accounts. Without Secure, the cookie is transmitted over plain HTTP in mixed-content or HTTP-only deployments. Without SameSite, cross-origin forms can trigger credentialed requests. The session JWT contains the user role and is used to authorize all management operations.
Change the set_cookie call to include `httponly=True, samesite='lax', secure=True` (or detect HTTPS from request.url.scheme as done correctly for the litellm_oauth_state cookie). The UI reads this cookie via document.cookie, so if JavaScript access is required, replace the cookie with a short-lived opaque code and exchange it for the JWT via a POST endpoint.
16
— Security
JWT audience and issuer validation disabled by default
When JWT_AUDIENCE and JWT_ISSUER env vars are unset, PyJWT skips aud/iss verification; a warning is logged once but tokens from any application sharing the same IdP signing keys are accepted.
In enterprise environments where multiple applications share an OIDC provider, a JWT minted for an unrelated application (e.g., an internal HR tool using the same Azure AD tenant) is accepted as valid by LiteLLM's proxy. An employee with a token for App A can make authenticated calls to the proxy without being an intended user. This is particularly dangerous for admin-scoped tokens if the admin scope name is guessable.
Make JWT_AUDIENCE required when enable_jwt_auth is true. At proxy startup, if enable_jwt_auth is configured and neither JWT_AUDIENCE nor any per-issuer audience is set, fail with a clear startup error rather than logging a warning. Alternatively, expose audience as a required field in the jwt_auth config block and validate it at config load time.
17
— AI Engineering
CLAUDE.md branch naming contradicts pre-push hook, causing agents to produce unshippable branches
CLAUDE.md tells Claude Code to use a litellm_ prefix with no slashes, but the pre-push hook rejects any branch that does not match the feature/|bugfix/|hotfix/|release/|chore/ pattern.
An agent following CLAUDE.md will create branches like litellm_fix_auth that the hook blocks on push, forcing either a manual rename or a --no-verify bypass. Both outcomes are bad: the first requires human intervention on every AI-assisted PR; the second permanently weakens the branch discipline gate. The contradiction also means agents cannot confidently execute the documented workflow end-to-end, so they will experiment or fall back to bypassing the hook.
Reconcile the two rules. Option 1: update CLAUDE.md to instruct Claude Code to use the hook-enforced format (e.g., feature/litellm-<description>) and remove the litellm_ prefix instruction for AI-generated branches. Option 2: add litellm_* to the protected-names list in .githooks/pre-push so the prefix is treated as a bypass (matching the actual workflow for internal contributors). Add the resolved rule to both CLAUDE.md and AGENTS.md so the instruction is unambiguous.
18
— Infrastructure
Default Helm standalone PostgreSQL ships with known weak credentials
values.yaml sets both postgresql.auth.password and postgresql.auth.postgres-password to 'NoTaGrEaTpAsSwOrD' with no mechanism to require override before deployment.
Operators who deploy the Helm chart with default values — including those following the quickstart documentation — get a PostgreSQL instance with a publicly known password. Any network-reachable Postgres port (the chart exposes port 5432 via a ClusterIP service that operators may expose) can be authenticated with this credential. This grants full access to all LiteLLM spend data, API keys, user records, and LLM provider credentials stored in the database.
Remove the default password values from values.yaml. Either require operators to supply a password at install time (`--set postgresql.auth.password=...`) or generate a random secret in a Helm pre-install hook and document the retrieval command. Add a Helm NOTES.txt warning when the default password string is detected. Consider migrating to an existingSecret reference as the documented default so operators are forced to create the secret themselves.
19
— Infrastructure
No PostgreSQL backup configuration exists in any deployment path
The Helm standalone PostgreSQL, Docker Compose database service, and CloudFormation RDS instance all ship with backup disabled or unconfigured; no backup, restore procedure, or PITR documentation exists in the repository.
Operators on any deployment path are running a bare PostgreSQL instance with no automated backup. The CloudFormation RDS instance has no BackupRetentionPeriod set, which defaults to 0 and disables automated backups entirely. A disk failure, accidental DROP, ransomware event, or botched migration has no recovery path. All spend data, API keys, user configurations, audit logs, MCP credentials, and organization structures are permanently lost. This risk is especially acute because the default schema migration path (prisma db push --accept-data-loss) can silently drop columns on upgrade, and there is no rollback.
For the Helm standalone path: add an optional backup sidecar or CronJob using pg_dump + object storage (S3/GCS/Azure Blob) with an example values.yaml override. For the CloudFormation template: set BackupRetentionPeriod to at least 7 and enable DeletionProtection. For Docker Compose: add a commented backup example or link to a docker-volume-backup recipe. Provide a restore runbook documenting the step-by-step recovery process. Prominently recommend managed PostgreSQL services (RDS, Cloud SQL, Azure Database) in the production deployment guide, as these include automated backup by default.
20
— Infrastructure
CloudFormation template hardcodes RDS credentials and omits production-critical security controls
The enterprise CloudFormation stack sets MasterUserPassword to 'litellmPassword', deploys into a single AZ, creates no VPC or security groups, and configures no encryption, backup retention, or deletion protection.
Any operator who deploys this CloudFormation stack into a production AWS account gets an RDS instance with a well-known password, no network isolation (no VPC security group restricting access to the RDS port), no encryption at rest, no automated backups (BackupRetentionPeriod defaults to 0), and a single-AZ topology with no failover. The AutoScalingGroup uses a t2.micro LaunchConfiguration with no UserData, meaning the EC2 instance has no runtime software and cannot actually serve LiteLLM. This template is non-functional as written and dangerous if deployed.
This template needs a full rewrite. Minimum requirements: randomly generated RDS password stored in AWS Secrets Manager, VPC with private subnets and security groups restricting port 5432 to the application tier only, storage encryption enabled, BackupRetentionPeriod >= 7, DeletionProtection: true, multi-AZ deployment, and actual EC2 UserData or ECS task definitions to bootstrap the application. Consider migrating to ECS Fargate which better fits the container runtime shape. Until rewritten, add a prominent deprecation notice and remove it from the enterprise distribution.
21
— Observability
Prompt and response content logged to all sinks by default
StandardLoggingPayload includes 'messages' (the raw LLM prompt) and 'response' fields that are forwarded to every configured logging destination unless turn_off_message_logging is explicitly set.
User prompts and model completions, which frequently contain PII or PHI, are sent to third-party systems (Datadog, S3, GCS, Langfuse, etc.) without operator awareness. In a multi-tenant deployment this creates cross-tenant data leakage risk in shared log destinations and potential compliance violations (GDPR, HIPAA, SOC 2).
Flip the default: set turn_off_message_logging=True at the CustomLogger base level and require operators to explicitly opt in to logging prompt/response content. Add a startup warning if content logging is enabled without a corresponding data-processing justification in config. Document the privacy implication prominently in the observability setup guide.
22
— Observability
Critical background jobs have no heartbeat or missed-run monitoring
The budget reset, spend log write, daily tag spend, spend log cleanup, and key rotation jobs emit ServiceLogger events but have no external dead-man's-switch or missed-run alert.
If any of these jobs silently stall (e.g., Redis lock held after a crash, asyncio task cancelled, uncaught exception in the scheduler), spend is not tracked, budgets are not enforced, and keys are not rotated. This can result in uncontrolled cost overruns or stale security posture with no alert firing. A stuck spend-write job for a multi-tenant deployment means billing data is lost.
Integrate a heartbeat monitor (Cronitor, Better Stack, or equivalent) for each critical scheduler job. Each job should emit a heartbeat ping on successful completion; the monitor fires an alert if the ping is missed for more than 2x the job interval. At minimum, add a Prometheus gauge (last_successful_run_timestamp per job) and alert on staleness.
23
— Delivery
Production Docker image build and push pipeline is absent from the repository
create-release.yml references cosign-signed ghcr.io images in release notes, but no workflow in .github/workflows/ or .circleci/config.yml builds or pushes images to any registry.
The provenance chain from source commit to published image is auditorially blind. If images are built outside this repository, there is no way to verify that the signed image corresponds to the released commit SHA, that the signing key is protected, or that the build environment is reproducible. Any external build pipeline is also outside the security controls visible here.
Add a GitHub Actions workflow (ideally triggered by create-release.yml on release publication) that: builds the Docker image from the tagged commit SHA, pushes to GHCR with both the version tag and a SHA-based immutable tag, signs with cosign using a Sigstore keyless flow or the existing cosign.pub key, and generates an SBOM. The workflow should be in this repository so the full chain from source to artifact is auditable.
24
— Documentation
Primary documentation lives in a separate repository with no in-repo mirror
The README defers almost all operator and reference documentation to docs.litellm.ai, which is served from the separate BerriAI/litellm-docs repository.
Operators and contributors cannot search documentation within the codebase repo. Docs can drift from code without any CI gate catching the discrepancy. An operator setting up a production deployment offline or auditing the system against its documentation finds only README, ARCHITECTURE, and CONTRIBUTING — no configuration reference, no env var catalog, no troubleshooting guide, and no runbooks. When code changes break a documented behavior, no automated check alerts the author.
Add a docs/ directory with at minimum: a production env var reference (all required and optional vars, their defaults, and their security implications), a configuration reference for the proxy YAML config, and a self-hosting guide that covers database setup, secret management, and Redis configuration. These do not need to duplicate the full docs site — they should be the content that operators need to stand up and maintain the system without internet access. Add a CI check that verifies any env var introduced in code appears in the env var reference.
25
— Documentation
No operational runbooks for any production workflow
The repository contains no documented procedures for deployment, rollback, backup, restore, incident response, or common operational tasks.
An operator whose deployment enters a degraded state has no documented procedure for diagnosing the failure, identifying whether the root cause is the DB, Redis, a provider, or the proxy itself, or verifying recovery. No rollback procedure tells an operator how to identify the last known-good image tag and redeploy. No backup or restore procedure covers the scenario where the database is corrupted or accidentally dropped. This gap is especially acute given the architecture audit findings (no backup config in any deployment path, silent spend data loss on pod crash) and the delivery audit finding that no post-deploy smoke test exists.
Add RUNBOOKS.md (or a runbooks/ directory) covering: (1) deploy — standard rolling update, Helm upgrade command, how to verify the migration job succeeded; (2) rollback — identifying the last good image tag, redeploying via Docker Compose or Helm, what to do if a migration ran partially; (3) backup and restore — how to take a pg_dump, how to restore, how to verify a restore; (4) incident diagnosis — how to use /health/liveliness vs /health/readiness, which Prometheus queries signal DB or Redis saturation, how to correlate Slack alerts to root causes; (5) spend flush recovery — what to do after a pod crash to verify spend data integrity.
26
— Code Quality
Broad exception swallowing masks failures in 1,600+ locations
`except Exception as e:` appears ~966 times and `except Exception:` appears ~646 times across the codebase, many in integration and logging callbacks.
Silent swallowing of exceptions in callback chains and provider integrations means provider errors, DB write failures, and spend-tracking failures can be silently discarded. This makes debugging production incidents very difficult and can allow spend to go unrecorded without any observable signal.
Audit all bare `except Exception:` blocks. Replace catch-all handlers in spend tracking, auth, and DB write paths with typed exception handling or explicit re-raise. For integration callbacks (observability integrations like Datadog, Prometheus), ensure failures are logged at WARNING or ERROR level with stack traces. Reserve bare `except Exception:` for top-level request handlers where failing open is an intentional choice.
27
— Code Quality
Two parallel proxy test directories cause navigation confusion and potential coverage gaps
tests/proxy_unit_tests/ (65 files) and tests/test_litellm/proxy/ (100+ files) coexist without clear ownership boundaries.
Contributors adding tests must choose between the legacy directory and the new mirrored structure. The same functionality (e.g., user_api_key_auth, proxy server endpoints) is tested in both, but the split is arbitrary. CI splits test runs across both directories, making it harder to assess coverage at the module level. The legacy directory still uses patterns (e.g., conftest.py duplicates) that the new structure has fixed.
Migrate all tests from tests/proxy_unit_tests/ into the corresponding tests/test_litellm/proxy/ subdirectory following the 1:1 mirror convention documented in tests/test_litellm/readme.md. Remove the legacy directory once migration is complete. Add a CI check that rejects new test files in tests/proxy_unit_tests/.
28
— Code Quality
Mutation testing is manual-only and covers a single folder
The mutation testing workflow requires manual dispatch (workflow_dispatch) and only targets litellm/proxy/management_endpoints/, leaving auth, spend tracking, and routing logic unmutated.
Surviving mutants in auth checks, budget enforcement, or LLM routing cannot be detected without running tests manually. The stated CLAUDE.md goal of >90% mutation kill rate cannot be verified for the majority of production-critical code paths.
Extend the mutmut scope in pyproject.toml [tool.mutmut] to include at least litellm/proxy/auth/ and litellm/proxy/spend_tracking/ in addition to management_endpoints/. Consider adding mutation runs on a weekly schedule (cron trigger) or on PRs touching sensitive paths. Track kill rate per folder and add a minimum threshold gate.
29
— Code Quality
Tests rely heavily on monkeypatching, violating the project's own dependency-injection rule
Monkeypatch.setattr appears ~2,339 times across 227 test files, despite CLAUDE.md explicitly banning it in favor of dependency injection.
Tests that monkeypatch module-level attributes test the wrong thing: they exercise the patcher's assumptions rather than the real system. If a class member is renamed or moved, the monkeypatch silently targets nothing and the test passes regardless. This is the same class of failure mode that CLAUDE.md warns against.
For new code, enforce dependency injection by passing collaborators (DB clients, cache clients, HTTP handlers) as constructor arguments or function parameters. This allows test authors to pass fakes without monkeypatching. Convert high-value existing tests (auth, spend tracking) as they are touched. Add a custom Ruff or flake8 rule that warns on `monkeypatch.setattr(litellm.`, the most common form of module-global patching.
30
— Code Quality
Mutable global state across 157 files couples modules and makes testing unreliable
The `global` keyword appears 527 times across 157 files, and PLW0603 (global variable assignment) has a budget ceiling of 183 in ruff-strict-budget.json.
Global state creates invisible coupling between modules. Test isolation breaks when one test mutates a global (e.g., litellm settings, cached clients) that another test reads. The 183 PLW0603 budget means new code can still add global mutation up to the current ceiling.
Replace module-level mutable singletons with dependency-injected objects (constructors, context managers, or factory functions). For configuration-style globals (litellm.*), consider a frozen dataclass or a Settings object that is constructed once and passed down. Lower the PLW0603 budget in ruff-strict-budget.json each sprint until it reaches zero.
31
— Architecture
APScheduler runs background jobs in the request-serving event loop
Slow DB writes in background jobs block request handlers sharing the same asyncio event loop.
APScheduler is initialized with AsyncIOExecutor, which runs all scheduled jobs as coroutines on the same asyncio event loop that handles HTTP requests. A slow update_spend write (e.g., PostgreSQL under load, network hiccup) that takes 2-3 seconds blocks the event loop and introduces latency spikes on all in-flight LLM requests during that window. The codebase documents one manifestation of this fragility: a jitter calculation bug in APScheduler caused 35GB of allocations and required removing jitter from all jobs as a workaround. This is a symptom of the design constraint, not the root cause.
Run background jobs in a separate asyncio event loop (a second thread with its own loop) or a separate worker process. Alternatively, use a lightweight task queue (the rq dependency is already declared in pyproject.toml) to push spend flush work off the main event loop. At minimum, ensure update_spend uses connection pooling and async DB writes so it yields control frequently rather than holding the event loop.
32
— Architecture
Wildcard imports from _types.py create pervasive hidden coupling
28 proxy modules use `from litellm.proxy._types import *`, making dependencies and namespace pollution invisible.
Any name added to _types.py is silently injected into 28 modules. Refactors that rename or remove a type in _types.py break 28 callers without import errors until runtime. Static analysis tools (mypy, pyright) cannot resolve what is in scope from a wildcard import, so type-checking coverage for these modules is degraded. This is the primary reason proxy_server.py's import block is 550 lines - it imports everything via wildcard and then re-imports specific names for clarity.
Replace all `from litellm.proxy._types import *` with explicit named imports. This is a mechanical change that a single grep-and-fix pass can resolve. IDEs can auto-generate the explicit import list from the wildcard. Do it one file at a time starting with the most-imported types.
33
— Architecture
Redis unavailability silently degrades rate limiting and budget coordination
When Redis is down, rate limiting becomes per-pod and cross-pod budget staleness worsens, with no operator alert.
Redis is the coordination bus for: TPM/RPM rate limit counters, API key caching, deployment cooldowns, spend queue buffering (optional), and the PodLockManager for distributed cron coordination. The DualCache falls back to in-memory only when Redis is unavailable, meaning each pod enforces its own rate limits independently. In a 4-pod deployment under Redis failure, a key's effective TPM limit becomes 4x its configured value. There is no alerting or documented runbook for this degraded mode, and operators may not notice until spend or rate limit anomalies surface in billing.
Add an explicit Redis health check that emits a high-severity alert (Slack, PagerDuty, Prometheus metric) when Redis becomes unavailable. Document the degraded-mode behavior in the operator runbook. Consider rejecting new requests when Redis is unavailable if the deployment is configured to require cross-pod rate limiting (a configurable strict-mode flag).
34
— Architecture
Budget inheritance logic scattered across auth_checks.py without a domain boundary
The org/team/user/key budget hierarchy is implemented as ~200 lines of nested conditionals in the auth path rather than as a named domain service.
Budget precedence rules (key budget > team member budget > team budget > org budget > global budget) are implemented inline in common_checks() and its callees. Adding a new entity type (e.g., agents, which now exist in the schema) requires modifying the auth path rather than extending a budget resolution service. Bug fixes to one level of the hierarchy risk inadvertently breaking another. There is no single place to read to understand the complete budget resolution algorithm.
Extract a BudgetResolver class that takes the resolved entity objects (key, team, org, user) and returns the effective budget constraints. The auth path calls BudgetResolver.resolve(entities) and gets back a BudgetConstraint. This is a pure function with no side effects, which makes it trivially testable. The current logic already has the data - it just needs to be encapsulated.
35
— Architecture
No idempotency guard on spend log writes during retry
update_spend retries DB writes without a deduplication key, so a crash mid-flush can double-count spend logs.
The batch flush job reads all queued spend updates and writes LiteLLM_SpendLogs rows to PostgreSQL. If the job succeeds in writing some rows but the process crashes before clearing the in-memory queue (or before Redis acknowledgment), the next flush attempt will write the same rows again. SpendLogs are append-only with no unique constraint on (request_id, timestamp), so duplicate rows silently inflate reported spend and downstream analytics. This also affects the daily spend aggregation tables (LiteLLM_DailyUserSpend, etc.).
Add a unique constraint on LiteLLM_SpendLogs(id) where id is the litellm_call_id (already a UUID on each request). On flush, use INSERT ... ON CONFLICT DO NOTHING so retries are idempotent. Apply the same pattern to the daily spend aggregate tables using (entity_id, date) as the conflict key with ON CONFLICT DO UPDATE SET spend = spend + EXCLUDED.spend.
36
— Data
LLM messages and responses stored in SpendLogs with no PII redaction controls
LiteLLM_SpendLogs.messages (Json) and .response (Json) store full LLM request and response content, which may include user PII, health information, or proprietary prompts, with no schema-level or application-level redaction controls.
Any user or service account with read access to LiteLLM_SpendLogs sees all LLM inputs and outputs across all tenants. For deployments handling healthcare, legal, or personally identifiable content, this constitutes a data minimization failure and can trigger HIPAA, GDPR, or SOC 2 findings. Spend logs are also the largest table and grow unboundedly, so the PII surface area expands with usage.
Add a `disable_message_logging` configuration option (may already exist; enforce it at the schema write level). When enabled, write null or a truncated token-count summary instead of full message content. For users who need message logging, document the data classification and access controls required. Apply retention-based purge to messages/response columns independently of request-level spend data.
37
— Data
Budget fields duplicated between LiteLLM_TeamTable and LiteLLM_BudgetTable
LiteLLM_TeamTable carries max_budget, soft_budget, tpm_limit, rpm_limit, budget_duration, and budget_reset_at directly alongside a budget_id FK to LiteLLM_BudgetTable, which has the same fields.
Two sources of truth for budget configuration on the same entity means application code must pick one and ignore the other, or must reconcile them. The auth_checks.py budget resolution logic already spans ~200 lines of nested conditionals; the duplicate fields make the effective budget for a team ambiguous without reading the code carefully. A write to the team table does not update the budget table and vice versa, so the two can silently diverge.
Migrate all budget configuration to LiteLLM_BudgetTable for all entity types. Remove the inline budget columns from LiteLLM_TeamTable (and similarly from LiteLLM_UserTable, LiteLLM_OrganizationTable where duplicated). All budget reads should go through the budget_id FK. This simplifies the auth layer and makes BudgetResolver (recommended in ARCH-007) straightforward to implement.
38
— Data
Members and admins stored as String arrays without FK enforcement
LiteLLM_TeamTable.members (String[]) and .admins (String[]) store user IDs as plain arrays rather than via FK-normalized membership rows, enabling dangling references when users are deleted.
When a user is deleted from LiteLLM_UserTable, their user_id remains in the members/admins arrays of all teams they belonged to. There is no referential integrity enforcement to clean up these stale references. Application code that trusts the members array without re-validating against the user table will act as if deleted users still exist. Similarly, a new user accidentally assigned the same user_id (possible if user_id is not UUID-based) would silently inherit team memberships.
Migrate team membership entirely to LiteLLM_TeamMembership (which already exists as a proper join table with a composite PK). Remove the members and admins String[] columns from LiteLLM_TeamTable. Enforce the delete cascade at the DB level so removing a user also removes their TeamMembership rows.
39
— Data
No Row Level Security; all multi-tenant isolation is application-layer only
The PostgreSQL schema has no RLS policies. The only barrier preventing cross-tenant data access is application-layer authorization logic.
For a system where multiple independent organizations share a single Postgres database, any bug in the application's authorization layer (auth_checks.py, management endpoints) can expose or corrupt another tenant's data. A SQL injection vulnerability, a misconfigured admin endpoint, or a privilege escalation bug has no DB-layer safety net. RLS would ensure that even if application code is bypassed (direct DB connection, SQL error, compromised service account), a user's query returns only their own rows.
Implement Postgres RLS on the highest-sensitivity tables: LiteLLM_VerificationToken, LiteLLM_SpendLogs, LiteLLM_UserTable, LiteLLM_TeamTable, LiteLLM_OrganizationTable. Set current_setting('app.organization_id') at connection time in the Prisma client setup and write policies like `USING (organization_id = current_setting('app.organization_id'))`. This is a defense-in-depth measure; it does not replace application-layer auth.
40
— Data
Float used for monetary spend values across all tables
Every spend column in LiteLLM_SpendLogs, LiteLLM_VerificationToken, LiteLLM_TeamTable, daily spend tables, and all entity tables uses Float, which is a binary floating-point type.
Floating-point arithmetic is non-associative and introduces rounding errors that accumulate across additions. At scale (millions of requests each costing fractions of a cent), cumulative floating-point error can cause spend totals to diverge from true cost by a measurable amount. Budget enforcement checks (spend >= max_budget) can fire early or late due to rounding. Postgres FLOAT8 does not guarantee exact decimal arithmetic, which is required for billing-grade financial data.
Migrate spend columns to NUMERIC(20, 10) (Prisma type Decimal). This is a data type migration that requires updating the Prisma schema, generating a migration, and updating application code to use Decimal arithmetic. The prisma-client-py Decimal type maps directly to Python's decimal.Decimal. Prioritize the primary accounting columns: LiteLLM_SpendLogs.spend, LiteLLM_VerificationToken.spend, LiteLLM_TeamTable.spend, LiteLLM_UserTable.spend.
41
— Data
LiteLLM_MemoryTable key is globally unique rather than scoped to user or team
LiteLLM_MemoryTable uses `key String @unique` as a global unique constraint; user_id and team_id are ownership stamps only and do not participate in uniqueness, enabling cross-tenant key collisions and potential data access confusion.
If two teams use the same key convention (e.g., 'system-prompt'), whichever team writes second silently overwrites the first. Conversely, an application that does not namespace keys will retrieve another team's memory entry when querying by key. The schema comment acknowledges this and says 'callers should namespace their keys', but there is no enforcement. For an AI gateway managing memory across many tenants, silent overwrites or cross-tenant reads are a confidentiality and correctness risk.
Change the unique constraint from `@unique` on key alone to `@@unique([key, team_id])` (or `@@unique([key, user_id])` for per-user scope). Update the upsert logic in management endpoints accordingly. Existing rows can be migrated with a one-time backfill that namespaces keys that have a team_id or user_id.
42
— Security
No HTTP security response headers on any endpoint
The proxy serves an admin UI and API responses with no X-Frame-Options, Content-Security-Policy, X-Content-Type-Options, HSTS, or Referrer-Policy headers set anywhere in the middleware stack.
The admin UI is vulnerable to clickjacking (no X-Frame-Options or CSP frame-ancestors). Browsers will MIME-sniff response content (no X-Content-Type-Options: nosniff). HTTPS deployments do not enforce HSTS. If the proxy is deployed over HTTPS, the absence of Strict-Transport-Security means browsers won't remember to require HTTPS on subsequent visits. Combined with SEC-003 (no HttpOnly on session cookie), a successful XSS on the UI page has no security header backstop.
Add a Starlette middleware (e.g., a custom BaseHTTPMiddleware) that injects `X-Frame-Options: DENY`, `X-Content-Type-Options: nosniff`, `Referrer-Policy: strict-origin-when-cross-origin`, and `Strict-Transport-Security: max-age=31536000; includeSubDomains` on all responses. For the UI path (/ui/*), also set a Content-Security-Policy header. This is a single middleware registration in proxy_server.py.
43
— Security
Default CORS policy allows requests from any origin
When LITELLM_CORS_ORIGINS is unset, allow_origins defaults to ['*'], permitting cross-origin requests from any domain; credentials are disabled with wildcard but this still opens unauthenticated API surface to any origin.
Any website can make cross-origin preflight-free requests to LiteLLM's LLM API endpoints (for simple request types). Bearer-token-authenticated endpoints are not directly at risk from CORS alone, but the wide-open default creates surface area for confused-deputy attacks if other authentication mechanisms (cookie-based) are in use. For a self-hosted enterprise gateway, the operator likely does not intend to serve arbitrary external origins.
Remove the '*' default. Either require operators to set LITELLM_CORS_ORIGINS explicitly (fail fast at startup with a clear message if it is unset and the server is not in development mode), or default to `['http://localhost:4000', 'http://127.0.0.1:4000']` for local dev and require explicit configuration for production. Document the security implication of '*' prominently in the configuration reference.
44
— Security
No rate limiting or lockout on authentication and login endpoints
The API key validation path, the UI login form, and the SSO initiation route apply no per-IP or per-identity rate limiting; an attacker can attempt unlimited authentication requests without throttling.
The master key (granting PROXY_ADMIN access to all management operations) can be brute-forced without rate limiting. A dictionary attack on the UI password form (UI_PASSWORD defaults to master_key) is similarly unconstrained. Token enumeration against the key validation path is possible because the error paths return distinct responses for expired vs. invalid keys.
Add slowapi or an equivalent rate-limiter with per-IP and per-endpoint limits on /user/auth, /login, /v2/login, /sso/key/generate, and the user_api_key_auth dependency. Apply exponential backoff after repeated failures and log rate-limit events as security-level audit entries. Consider adding a CAPTCHA or time-delay on the UI login form.
45
— Security
No pre-commit secret scanning; developer-local commits are unguarded
GitGuardian and OSV scanner run only in CI; there is no .pre-commit-config.yaml or detect-secrets configuration, so secrets committed locally are only caught after they have been pushed to the remote and are already in git history.
A developer who accidentally stages a real API key, database password, or JWT private key will not be warned until the GitGuardian CI scan runs. By that time the secret is in git history and must be rotated even if the commit is later amended or reverted. Given the large contributor base and many integrations (40+ LLM providers, 20+ observability tools), the risk of accidental secret commit is elevated.
Add a .pre-commit-config.yaml that runs `detect-secrets scan` (or trufflesecurity/trufflehog) as a pre-commit hook. Add installation instructions to CONTRIBUTING.md. Optionally enable the pre-commit.ci service so hooks run on PRs without requiring local setup. This catches secrets before they leave the developer's machine.
46
— Security
No vulnerability scanning beyond dependency CVEs and secret detection
The CI pipeline runs OSV scanner for dependency CVEs and GitGuardian for secrets, but has no SAST tool, no container vulnerability scanner, no IaC scanner, and no pre-commit hooks.
Static analysis findings (SQL injection patterns, deserialization risks, command injection in subprocess calls, unsafe template rendering) go undetected. Container image CVEs (base image Wolfi packages, Prisma binaries) are not scanned. Kubernetes YAML and Helm charts are not scanned for misconfigured security contexts or privileged containers. The single Semgrep rule in .semgrep/ only guards against committing the .claude/ directory.
Add Semgrep or Bandit to the CI pipeline to scan Python code for security patterns. Add Trivy or Grype to scan the built Docker image on each merge to the staging branch. Add kube-score or checkov to scan Helm charts for security misconfigurations (runAsNonRoot, readOnlyRootFilesystem, privileged, etc.). Route Critical/High findings to block the release gate.
47
— Security
AI Skills code execution passes model-generated code to sandbox without pre-execution inspection
SkillsSandboxExecutor.execute() accepts arbitrary Python code generated by an LLM and runs it inside a Docker/Podman/Kubernetes sandbox with no content inspection before execution.
Adversarial prompt injection could cause the LLM to generate code that, when executed in the sandbox, exfiltrates environment variables, reads mounted secrets, or makes outbound network calls to attacker-controlled infrastructure. If the sandbox has network access (not documented as disabled by default) or shares filesystem mounts with the host, the blast radius extends beyond the container. The sandbox is the only barrier between model output and code execution.
Before passing model-generated code to the sandbox, apply a static analysis pass (e.g., ast.parse + visitor to detect os.environ reads, subprocess calls, network socket creation, open() on paths outside /tmp). Run the sandbox with network disabled (`network_disabled=True` in Docker) and an explicit seccomp profile. Document the sandbox security model prominently so operators know what trust boundary they are accepting.
48
— AI Engineering
AGENTS.md is a one-liner stub that provides no usable context for non-Claude tools
AGENTS.md contains a single line: 'Read @CLAUDE.md for coding guidelines', which relies on Claude Code's @-file reference syntax and provides no structured guidance for Codex, Cursor, Copilot, or any other AI tool that reads AGENTS.md directly.
Any AI tool other than Claude Code that reads AGENTS.md gets no architecture context, no test conventions, no branch rules, and no safety constraints. If the team adds a second AI tool or if contributors use Cursor or Copilot, those sessions run without guardrails. Even for Claude Code, the AGENTS.md file adds no value on top of CLAUDE.md and creates a false sense that agent context is configured for multiple tools.
Either expand AGENTS.md to contain the full agent-facing context (not just a pointer), or clearly document that only Claude Code is supported and have AGENTS.md explain where the canonical context lives in plain prose. If CLAUDE.md is the canonical file, AGENTS.md should at minimum contain the most critical safety constraints verbatim (never commit secrets, run tests before committing, branch naming rules) so any tool gets the basics even without @-file resolution.
49
— AI Engineering
No spec-driven workflow; AI changes flow from issues to code without a reviewed design step
No formal spec, ADR, or acceptance criteria files exist in the repository. The PR template references GitHub issues and Linear tickets, but contains no field for design docs or acceptance criteria, so AI agents generate code directly against informal issue descriptions.
Agents making architectural changes (new routing strategies, auth flows, spend-tracking approaches) do so without a reviewed design document. Mistakes in interpretation are caught only at PR review, after the code is written. This is the highest-risk path for AI-generated changes: the agent may solve a technically correct interpretation of a vague issue that diverges from the intended design. For a codebase already flagged (COD-001) for god files and unclear separation of concerns, AI agents without spec guidance are likely to amplify existing structural problems rather than reverse them.
Add an optional but recommended spec template to .github/ for non-trivial changes (new features, architecture modifications). Reference it in CLAUDE.md: 'For changes that affect more than one layer of the stack, write a short spec (problem statement, proposed approach, key tradeoffs) and get a comment approval before implementing.' Add a Spec/Design field to the PR template. This does not require every PR to have a spec, only that one exists before AI generation starts on non-trivial work.
50
— AI Engineering
Mutation testing goal in CLAUDE.md is unenforceable; coverage is manual and limited to one module
CLAUDE.md instructs agents to target >90% mutation kill rate, but the mutation-test.yml workflow is manual-only (workflow_dispatch), covers only litellm/proxy/management_endpoints/, and uploads results as an artifact with no threshold gate.
The >90% kill rate goal exists only as text in an agent context file. There is no CI enforcement, no minimum threshold, and no feedback loop that tells an agent whether its tests actually meet the stated goal. Auth, spend tracking, and LLM routing logic -- the highest-risk production paths -- are entirely excluded from mutation testing. Agents writing tests for these areas have no signal that their tests can actually catch mutations. The code-quality audit (COD-007) flags this separately; the AI-engineering risk is that CLAUDE.md creates a false promise that guides agents toward a measurable goal that cannot currently be measured.
Either remove the >90% kill rate claim from CLAUDE.md until enforcement exists, or add a scheduled mutation job (weekly cron) with a minimum kill-rate threshold that fails CI below the target. Extend pyproject.toml [tool.mutmut] paths_to_mutate to include litellm/proxy/auth/ and litellm/proxy/spend_tracking/ at minimum. Consider adding a PR-level mutation run for files touched by a diff, using the --changed flag or a custom script that selects mutmut scope from git diff.
51
— AI Engineering
No knowledge lifecycle; lessons from AI-assisted work are never captured or compounded
The repository has no ADRs, decision records, retrospective files, or agent memory system. When CLAUDE.md is updated, there is no changelog, no ownership record, and no process for removing stale guidance.
When an agent makes an error that leads to a rule being added to CLAUDE.md, there is no record of why the rule exists, what failure it was added to prevent, or when it was last reviewed. Over time, CLAUDE.md will accumulate rules without context for their importance, making it harder for agents and new contributors to distinguish critical constraints from historical preferences. The mutation report (scripts/mutation_report.py) is the closest thing to a knowledge artifact, but it is uploaded to a transient CI artifact with 14-day retention and is never connected back to test improvements, CLAUDE.md updates, or agent feedback.
Add a lightweight decision log: a docs/decisions/ directory with short markdown files for non-obvious rules added to CLAUDE.md (what was the triggering incident, what alternatives were rejected, what the rule now says). This does not need to be an ADR for every change; only for rules that would surprise a new contributor or agent. Consider adding a 'Why this rule exists' annotation to the most critical CLAUDE.md constraints. Connect the mutation report output to a GitHub issue or PR comment so surviving mutants flow into the backlog rather than expiring in 14 days.
52
— Infrastructure
Docker Compose ships with hardcoded database credentials in plaintext
docker-compose.yml sets DATABASE_URL to a connection string with the hardcoded password 'dbpassword9090' directly in the file, committed to the public repository.
Operators who deploy using Docker Compose without customization run Postgres with a publicly known password. Users following quickstart tutorials often copy the compose file as-is into production. The password is visible in git history permanently. The PostgreSQL service also exposes port 5432 to the host network by default (ports: '5432:5432'), making it reachable from outside the container network if the host has any external interface.
Replace the hardcoded DATABASE_URL with a reference to an env_file variable (e.g., DATABASE_PASSWORD) and ship a .env.example with a placeholder. Remove the PostgreSQL host port exposure from the default compose file (keep it commented for debugging) since the litellm service already reaches Postgres via the internal Docker network. Add a setup step in the README that generates a random password and writes it to .env.
53
— Infrastructure
Azure Marketplace ARM template references a mutable Docker image tag
The ARM template sets imageName to 'ghcr.io/berriai/litellm:main-latest' by default, a floating tag that resolves to a different image on every pull.
Operators deploying via the Azure Marketplace get whatever image was last tagged as main-latest at deploy time, with no integrity guarantee. A supply chain compromise that pushes malicious code to main-latest before a deployment would be silently pulled. The Azure Container Instance has no restart policy tied to image updates, but new deployments or redeploys always pull the latest digest of the mutable tag. This is inconsistent with the release workflow, which creates versioned tags, and the Helm chart, which defaults to the chart appVersion.
Change the default imageName to a versioned release tag (e.g., 'ghcr.io/berriai/litellm:v1.x.y') tied to the ARM template version. Better, pin to a SHA256 digest. Add a parameter validation that rejects tags equal to 'main-latest' or 'latest'. Update the marketplace deployment guide to document how to pin to a specific release.
54
— Infrastructure
No CPU or memory resource limits set in Helm chart defaults; HPA cannot function
values.yaml leaves resources as an empty map, so pods have no CPU/memory requests or limits; the HPA targetCPUUtilizationPercentage default of 80 has nothing to measure against.
Without resource requests, Kubernetes cannot compute CPU utilization as a percentage for HPA, so autoscaling never triggers regardless of load. Without limits, a pod under high LLM traffic can consume all node CPU and memory, triggering OOMKill on co-located pods or degrading the node for other workloads. The documented APScheduler memory leak (35 GB allocations in production) indicates this has happened and the absence of a memory limit means it could recur without bound.
Define default resource requests and limits in values.yaml (e.g., requests: {cpu: '500m', memory: '512Mi'}, limits: {cpu: '2', memory: '2Gi'}) with a clear comment to override based on measured workload. Add a NOTES.txt warning when HPA is enabled but resources are not set. Document the resource sizing guidance in the operator runbook alongside the APScheduler memory leak mitigation.
55
— Infrastructure
Redis disabled by default in Helm chart, silently disabling distributed rate limiting and spend durability
redis.enabled defaults to false in values.yaml; without Redis, rate limiting is per-pod only, spend queue durability is lost on pod crash, and PodLockManager cannot serialize cross-pod cron jobs.
In a multi-replica Helm deployment, each pod enforces its own independent rate limit counters. A key configured with a 1000 TPM limit is effectively limited to 1000 * replica_count TPM. Spend data accumulated between flush cycles (up to 60 seconds) is silently discarded on pod restart with no durability guarantee. The PodLockManager cannot serialize the spend flush across pods, so concurrent flush jobs can interleave writes and produce double-counted spend records. These behaviors are not documented in the Helm values or the operator guide.
Change redis.enabled to true in values.yaml, or add a prominent NOTES.txt warning when replicaCount > 1 and Redis is disabled explaining the rate limiting and spend durability implications. Document the degraded-mode behavior explicitly in the operator guide under a 'Multi-Pod Deployment Requirements' section. Consider making the Redis sub-chart required (not optional) when replicaCount > 1.
56
— Observability
Structured JSON logging is disabled by default
Without JSON_LOGS=true, the proxy emits ANSI-colored text logs through all three verbose loggers, making log aggregation, field-level search, and alerting unreliable.
Operators using Datadog Logs, CloudWatch, or any log aggregation platform without JSON_LOGS enabled get unstructured text that cannot be reliably parsed. Incident response requires grep rather than structured queries. Correlation IDs and error codes embedded in the rich StandardLoggingPayload are not surfaced.
Enable JSON logging by default in the proxy startup path (litellm/proxy/proxy_cli.py). Provide a config flag to revert to text for local development. Update default Helm chart and Docker examples to include JSON_LOGS=true.
57
— Observability
No SLO, SLA, or uptime target defined anywhere in the repository
The codebase has no documented reliability targets, error budgets, or uptime commitments for the proxy service or its downstream dependencies.
Operators running production LLM workloads have no measurable reliability baseline. Without an SLO, alerting thresholds are arbitrary, error budgets cannot be tracked, and there is no principled basis for prioritizing reliability improvements. An implied SLA to end users exists with no measurement source to verify compliance.
Define a minimum SLO document (e.g., p99 latency < Xs, error rate < Y%, availability > Z%) alongside the Grafana dashboard. Wire these targets to Prometheus alerting rules so budget burn is visible. Add a CHANGELOG-style status page commitment, even if hosted externally.
58
— Observability
Error tracking is entirely opt-in with no fallback
Sentry integration is well-implemented (PII denylist, EventScrubber, environment tagging) but requires operators to add 'sentry' to callbacks and set SENTRY_DSN; a default deployment has zero error aggregation.
Exceptions in async callback chains and background task error paths are caught and logged to verbose_logger but not routed anywhere a human will see. Silent failures in logging integrations (Datadog batch flush errors, OTel export failures) are particularly likely to go unnoticed. Sentry's capture_exception is called in many places but is a no-op when unconfigured.
Log a startup warning when no error-tracking backend is configured. Consider integrating a lightweight fallback (structured error log to stderr with a distinct ERROR level and a sampling strategy) that operators can route to their existing aggregation without requiring Sentry. Document the Sentry setup as a required production step, not an optional integration.
59
— Observability
No operational runbooks or incident response documentation
The only runbook in the repository is an internal database migration guide for LiteLLM engineers; there are no runbooks for operators responding to production incidents.
When the proxy goes down or degrades, an operator has no documented procedure for diagnosing the failure, identifying whether it is a DB, Redis, provider, or proxy issue, or verifying recovery. This increases mean time to recovery and makes post-incident review difficult.
Add a concise operational runbook (in docs/ or alongside the Grafana dashboard) covering: how to interpret the health endpoints, how to correlate Slack alerts to root causes, what the key Prometheus queries are for diagnosing saturation or DB lag, and how to confirm recovery after a fix is deployed.
60
— Delivery
CodeQL and zizmor SAST do not gate the primary litellm_internal_staging development path
codeql.yml and zizmor.yml only trigger on pull_request targeting main, but per CLAUDE.md and CONTRIBUTING.md, nearly all development targets litellm_internal_staging rather than main.
Security SAST findings introduced during normal feature development are not surfaced until a release-cut PR from litellm_internal_staging to main. By that point the change may already be in the internal staging branch for days or weeks, increasing the cost of remediation.
Add litellm_internal_staging and litellm_oss_branch to the pull_request branches list in codeql.yml and zizmor.yml, matching the branch coverage of test-linting.yml and the unit-test workflows.
61
— Delivery
No rollback runbook or post-deploy smoke test for self-hosted deployments
The repository provides no documented procedure for rolling back a bad release, and no automated smoke test runs after deployment to verify the container is healthy.
For operators running litellm in production with auth, spend-tracking, and LLM routing, a bad release (broken migration, regression in auth, proxy crash) has no guided recovery path. Rollback requires manually identifying the prior image tag and redeploying, which adds mean-time-to-recovery under pressure.
Add a RUNBOOKS.md (or equivalent) documenting: (1) how to identify the last known-good image tag, (2) how to roll back via Docker compose or Helm, (3) what to check if migrations ran partially, and (4) the litellm_hotfix_* process for emergency patches. Consider adding a post-deploy smoke test (e.g., a healthcheck endpoint curl) to the Helm chart liveness/readiness probes and Dockerfile.
62
— Delivery
No feature flag system — all changes ship as full releases with no kill-switch
The codebase has no rollout-control mechanism: no LaunchDarkly, Unleash, Statsig, GrowthBook, Postgres flags table, or environment-driven flag infrastructure was found.
Any bug in auth, spend-tracking, LLM routing, or guardrail logic that ships in a release requires a full image rollback to remediate. There is no way to disable a specific feature without redeploying. For a proxy that sits in front of production LLM spend, this makes the blast radius of a bad release wide.
Introduce a lightweight feature flag mechanism. A Postgres-backed flags table (name, enabled, rollout_percentage, updated_at) read at startup and cached in-process is sufficient to start. Gate high-risk changes (new auth paths, new spend-tracking logic, new routing strategies) behind a flag for at least one release cycle before making them the default. Document the flag lifecycle and removal policy.
63
— Delivery
ggshield secret scan silently skips when GITGUARDIAN_API_KEY is absent
The ggshield step in test-linting.yml runs a conditional check and exits 0 without scanning when the API key environment variable is not set.
In fork PRs or environments where the secret is not configured, the secret scan gate is silently bypassed. A contributor could unknowingly commit a hardcoded credential that passes CI without any secret-scanning coverage.
Either (a) require GITGUARDIAN_API_KEY as a mandatory CI secret and fail the job when it is absent, or (b) replace the optional ggshield step with a tool that does not require an external key, such as detect-secrets (already a project dependency in proxy-runtime), trufflehog, or gitleaks, configured to always run. The existing test_no_hardcoded_secrets.py unit test is a partial safeguard but does not scan all files.
64
— Documentation
AGENTS.md is a one-liner stub providing no standalone value
AGENTS.md contains a single line ('Read @CLAUDE.md for coding guidelines') that relies on Claude Code's @-file reference syntax and is meaningless to any other AI tool or reader.
Any AI coding tool other than Claude Code — Cursor, Copilot, Codex, Gemini Code Assist — reads AGENTS.md directly and receives no architecture context, no test conventions, no branch naming rules, and no safety constraints. Even for Claude Code, if the @-file resolution fails or the session context window is full, AGENTS.md contributes nothing. As AI tooling in the team or open-source contributors evolves, having a stub AGENTS.md creates a false impression that agent context is configured for multiple tools. This also means the documented AGENTS.md standard (meant to be tool-agnostic context) is not being used correctly.
Expand AGENTS.md to contain the minimum critical safety constraints verbatim: never commit secrets, run make lint and make test-unit before committing, branch naming conventions, and the PR base branch (litellm_internal_staging not main). These should not be pointers but actual content, so that any tool reading the file gets the essentials. The full CLAUDE.md content can remain Claude-specific; AGENTS.md should be the tool-agnostic minimum. Add a note clarifying which AI tools are officially supported.
65
— Documentation
.env.example ships hardcoded development credentials without a production warning
The .env.example file contains LITELLM_MASTER_KEY=sk-1234 and a DATABASE_URL with the password dbpassword9090 — both publicly known, committed to the open-source repo, and usable as-is.
Operators who copy .env.example as a starting point for production deployment get a master key (sk-1234) that grants PROXY_ADMIN access and a database password (dbpassword9090) that is checked into the public repo and also appears in docker-compose.yml. Both values are widely known. The file also covers only 8 LLM providers out of 100+ and omits proxy-critical vars like STORE_MODEL_IN_DB semantics, Redis configuration, and security-sensitive vars (UI_PASSWORD, LITELLM_SALT_KEY, JWT_SECRET). There is no warning in the file or README that these values must be replaced before production use.
Add a prominent comment block at the top of .env.example: '# WARNING: All values below are development-only placeholders. Replace ALL values before deploying to any shared or production environment.' Replace LITELLM_MASTER_KEY with a placeholder like sk-change-me-in-production. Add a startup check that refuses to start if LITELLM_MASTER_KEY equals sk-1234 in non-development mode. Extend .env.example to cover all security-relevant vars with descriptive comments. Consider adding a checklist to CONTRIBUTING.md and the operator guide enumerating the vars that must be set before production deployment.
66
— Documentation
No decision records documenting rationale behind major technical choices
No ADRs, design docs, or decision records exist in the repository; the reasoning behind key architectural tradeoffs is entirely absent.
Engineers touching spend tracking, budget enforcement, caching, or monetary precision cannot determine whether the current approach was a deliberate design choice or a convenience that should be improved. Examples where the absence of a decision record creates risk: float columns for monetary spend values (accepted precision risk or oversight?); in-memory SpendUpdateQueue as the default (deliberate for performance or not yet durabilized?); APScheduler running in the request event loop (known limitation or unknown?); prisma db push --accept-data-loss as the community migration default (intentional for operator convenience?). Without decision context, AI agents and new contributors are likely to preserve bad patterns on the assumption they are intentional.
Add a docs/decisions/ directory with short markdown records for the most consequential non-obvious choices. Each record needs only: the decision, the alternatives considered, the tradeoff that drove the choice, and current status (accepted/superseded). Priority records: float vs Decimal for spend (DAT-010); in-memory queue vs Redis-default (ARCH-001/DAT-005); APScheduler vs separate event loop (ARCH-004); prisma db push default (DAT-003). Also add 'Why this rule exists' annotations to the most critical CLAUDE.md constraints so their triggering context is not lost.
67
— Documentation
litellm/proxy/README.md is significantly stale and misleading
litellm/proxy/README.md references port 8000 (correct is 4000) and lists only the early-era endpoint directories, omitting management_endpoints/, health_endpoints/, pass_through_endpoints/, and a dozen others that now constitute most of the proxy.
A contributor looking at litellm/proxy/ for the first time and reading README.md gets an inaccurate picture of the directory structure and an incorrect port. They may attempt to test against port 8000, miss entire subsystems (management endpoints, health endpoints, pass-through), and form a mental model of the proxy that is years out of date. The file also describes the proxy as 'A local, fast, and lightweight OpenAI-compatible server' — accurate for v0.1 but not reflective of its current role as an enterprise AI gateway.
Either update litellm/proxy/README.md to reflect the current port (4000), directory structure, and scope, or replace its content with a pointer to ARCHITECTURE.md which has accurate, maintained component documentation. The update is straightforward: fix the port, list the current top-level directories under proxy/ with a one-line purpose for each, and remove the outdated quick-start that duplicates the main README.
68
— Architecture
Model deployment sync has a 30-second visibility gap across pods
Newly added models are visible on some pods up to 30 seconds before others, causing inconsistent routing.
The add_deployment job polls PostgreSQL every 30 seconds to load model configurations. In a multi-pod deployment, a model added via the management API will be visible on one pod immediately but on others only at their next poll cycle. During that window, requests routed to pods that do not yet know about the model will fail with a model-not-found error. For deployments that dynamically add/remove models frequently, this creates a persistent routing inconsistency window.
Publish a Redis pub/sub notification on model add/remove operations. Each pod subscribes and immediately calls add_deployment on notification rather than waiting for the poll cycle. The 30-second poll remains as a fallback for pods that miss the notification. This reduces the visibility gap from 30 seconds to sub-second.
69
— Architecture
Management API has no versioning strategy for breaking changes
The /key, /team, /user management endpoints have no version prefix and no documented breaking-change policy.
The LiteLLM management API (/key/generate, /team/new, /user/new, etc.) is consumed by the UI, the Python client (litellm/proxy/client/), and third-party integrations. These endpoints lack a /v1/ or /v2/ prefix. When fields are added, renamed, or removed, consumers break silently. There is no changelog or stability guarantee, which complicates enterprise deployments that pin to a specific LiteLLM version and cannot take breaking management API changes.
Apply a /v1/ prefix to management API routes and document a breaking-change policy (semantic versioning on the API surface, deprecation notices before removal). The OpenAI-compatible LLM endpoints already set the right example with their /v1/ prefix.
70
— Data
Status, role, and policy fields use unconstrained String columns throughout
Fields like LiteLLM_UserTable.user_role, LiteLLM_GuardrailsTable.status, LiteLLM_PolicyTable.version_status, LiteLLM_UserNotifications.status, and LiteLLM_MCPServerTable.approval_status are String with no DB-level CHECK constraint or enum type.
Any value can be written to these columns including typos, arbitrary strings from API callers, or values from renamed constants. Application code must trust that the DB contains only valid values; a bad write causes silent authorization or workflow logic failures rather than a validation error at write time. Postgres native enums or CHECK constraints would catch invalid values at the DB boundary.
Convert status/role fields to Postgres enum types via Prisma (use `enum` in schema.prisma) or add CHECK constraints via raw migration SQL. At minimum, add CHECK constraints on the most security-sensitive fields: user_role, approval_status, and guardrail status. Prisma natively supports `enum` declarations that generate Postgres enum types.
71
— Data
SpendLogs table grows unboundedly without partitioning being enabled by default
LiteLLM_SpendLogs receives one row per LLM request and has no automatic retention or partitioning enabled out of the box; SpendLogsPartitionManager and partition-based DROP require an operator-run SQL runbook to activate.
At moderate traffic (100 req/s), LiteLLM_SpendLogs accumulates ~8.6M rows/day and ~3.1B rows/year. Without partitioning, DELETEs leave dead tuples that autovacuum struggles to reclaim at this volume, causing table bloat. Query performance degrades over time for the dashboard and analytics endpoints that scan by startTime. The only mitigation currently in the codebase requires reading a runbook and executing `partition_spend_logs.sql` manually before the problem manifests.
Enable the partition manager by default when `use_spend_logs_partitioning` is set in general_settings, and include the partition_spend_logs.sql runbook prominently in the operator documentation. Add the automated partition pre-creation and retention-based DROP to the default scheduler configuration so operators who set a retention window get automatic cleanup without manual steps.
72
— Security
No multi-tenant database isolation backstop; all authorization is application-layer only
PostgreSQL has no Row Level Security policies. A single application-layer authorization bug is the only barrier between a tenant's data and any other tenant's query path.
A SQL injection, a misconfigured admin endpoint, a cached-object confusion bug, or a privilege escalation in auth_checks.py exposes all tenants' data simultaneously. An attacker who achieves direct DB access (compromised Prisma client, leaked connection string) bypasses all authorization logic entirely. For a multi-tenant AI gateway processing customer data, credentials, and spend records, this is a material defense-in-depth gap.
Implement Postgres RLS on the highest-sensitivity tables: LiteLLM_VerificationToken, LiteLLM_SpendLogs, LiteLLM_UserTable, LiteLLM_TeamTable, LiteLLM_OrganizationTable. Set `current_setting('app.organization_id')` via `SET LOCAL` in the Prisma client's connection setup hook and write policies like `USING (organization_id = current_setting('app.organization_id', true))`. This is a defense-in-depth measure, not a replacement for application-layer auth.
73
— AI Engineering
Greptile AI review requirement is an honor-system checklist item, not a CI gate
The PR template requires a Greptile confidence score of 4/5 before requesting maintainer review, but this is a manual checkbox with no CI enforcement; contributors and AI agents can submit PRs without completing the Greptile step.
If Greptile is providing meaningful code review signal (the template implies it is), skipping it silently reduces review quality. AI-generated PRs in particular may skip the step because the agent submitting the PR does not know what Greptile is or how to trigger it. The result is that the stated minimum quality bar (4/5 Greptile confidence) applies only to contributors who voluntarily enforce it.
If Greptile review is a hard requirement, integrate it via a GitHub App or status check so it blocks merge until the score is recorded. If it is a soft recommendation, remove the 4/5 threshold from the checklist to avoid creating a false gate. In either case, update CLAUDE.md to explicitly tell agents how to trigger a Greptile review as part of the PR workflow.
74
— Infrastructure
Pod Disruption Budget disabled by default; all pods can be evicted simultaneously
pdb.enabled defaults to false in values.yaml, so Kubernetes node drains, cluster upgrades, or voluntary disruptions can evict all litellm pods simultaneously.
During a Kubernetes node drain (routine in managed clusters for upgrades), all litellm pods on the draining node are terminated simultaneously with no guarantee that at least one replica remains available. For deployments with replicaCount > 1, this defeats the HA intent of multiple replicas. A complete eviction window means all in-flight LLM requests fail, all in-memory spend queue data is lost without flushing, and the gateway is unavailable until pods reschedule on other nodes.
Set pdb.enabled to true by default with minAvailable: 1 (or '50%' for larger deployments). Document the PDB in the Helm chart README as required for production deployments. Add a NOTES.txt recommendation to enable PDB when replicaCount > 1.
75
— Infrastructure
Hardened Docker Compose references an unpinned Squid image from 2019
docker-compose.hardened.yml uses sameersbn/squid:3.5.27-2, a community image last updated circa 2019 with no SHA256 digest pin, as the egress proxy for the hardened deployment.
The hardened Compose variant exists as a good security reference showing how to run the proxy with a read-only filesystem, dropped capabilities, and an egress proxy. Its value is undermined by pinning the egress proxy to a seven-year-old community image with no digest. Squid 3.5 reached end-of-life and has unpatched CVEs. Using a mutable tag also means the image pulled today may differ from what was tested. Operators who follow this as a security reference could end up running a vulnerable proxy component.
Replace sameersbn/squid:3.5.27-2 with a maintained Squid image pinned to a SHA256 digest (e.g., ubuntu/squid or a distroless-equivalent). Consider switching to a simpler egress solution like a minimal tinyproxy image or documenting iptables-based egress restriction as an alternative that avoids the dependency on a third-party proxy image entirely.
76
— Observability
log_raw_request_response flag can expose upstream API keys in logs
When log_raw_request_response=True is set, the raw HTTP request (including Authorization headers carrying upstream provider API keys) is captured in the logging payload.
If this flag is enabled in a production deployment for debugging and logs are shipped to a third-party system (Datadog, S3, etc.), upstream provider API keys appear in the log destination. The flag is False by default but called with True in litellm/main.py for specific debug paths.
Add a warning log when log_raw_request_response is True at startup. In the raw request capture path, strip Authorization and api-key headers before storing the payload in StandardLoggingPayload, or refuse to log raw requests when a logging callback is active.
77
— Delivery
Helm chart appVersion is stale relative to the Python package version
deploy/charts/litellm-helm/Chart.yaml has appVersion v1.85.1 but pyproject.toml declares version 1.89.0, a four-minor-version gap.
Operators using the Helm chart get incorrect version metadata. Monitoring systems that read appVersion for dashboards or alerts will show stale data. The Helm chart may also reference a Docker image tag that no longer reflects the current feature set.
Keep Helm chart appVersion in sync with pyproject.toml as part of the release workflow. Add a CI check (similar to check-schema-sync.yml) that fails if the chart appVersion does not match the Python package version, or automate the update in create-release.yml.
78
— Delivery
Database migration release mechanics lack a rollback or forward-fix plan
Prisma migrations run automatically at container startup via litellm/proxy/prisma_migration.py with no documented ordering gate, approval step, or rollback procedure for destructive schema changes.
If a migration drops a column, renames a table, or introduces a constraint that conflicts with data in production, operators have no documented recovery path. Rolling back the application container does not automatically roll back the schema, leaving the database in a state the previous image cannot handle.
Document the migration release strategy in a runbook: (1) migrations always run before app code via entrypoint ordering (already the case), (2) destructive changes must use expand/contract across two releases, (3) the rollback plan for each migration must be stated in the migration file or PR description. Consider adding a --dry-run or --check mode to prisma_migration.py that CI can invoke to preview migration effects before deployment.
79
— Documentation
cookbook/ is an unindexed collection with stale content and no freshness signals
The cookbook/ directory contains 11 READMEs and notebooks with no index, inconsistent naming, no freshness dates, and at least one guide showing a PyPI version badge of v0.1.345.
Engineers looking for usage examples find cookbook entries that may be current or years old with no way to tell. cookbook/litellm_proxy_server/readme.md shows a v0.1.345 PyPI badge and describes a feature set incompatible with the current codebase. A user following the cookbook for LiteLLM proxy server setup will receive incorrect configuration guidance. The cookbook is not linked from the main README flow, so it may be discovered accidentally rather than intentionally.
Add a COOKBOOK.md index at the repository root (or at cookbook/README.md) listing each entry with a one-sentence description and the approximate last-verified version. Archive or clearly mark stale entries (cookbook/litellm_proxy_server/readme.md at minimum). Add a policy in CONTRIBUTING.md that new cookbook entries must include a 'Last verified with LiteLLM vX.Y' line. Link the cookbook index from the main README contributing section.
80
— Documentation
No legal docs discoverable from the repository despite hosted commercial product linked in README
README.md links to the enterprise hosted LiteLLM product and a demo booking page, but no privacy policy, terms of service, or legal links are present anywhere in the repository.
Users who discover LiteLLM through the README and want to understand their legal relationship with the hosted service have no link to follow. Operators self-hosting LiteLLM and handling user data have no template or guidance for their own legal obligations. The README's enterprise pitch ('For companies that need better security, user management and professional support') without any privacy or terms link is a credibility gap for enterprise evaluation.
Add links to litellm.ai's privacy policy and terms of service in the README (likely in the Enterprise section). A one-line addition: 'Privacy policy and terms of service: litellm.ai/legal' (or whichever URL applies) is sufficient. For the self-hosted distribution, add a note in the operator guide that deployers are responsible for their own privacy obligations and link to the relevant regulatory frameworks (GDPR, HIPAA, CCPA) that may apply.