Large Language Model Deployment: Hosting Options and Service Models

Large language model (LLM) deployment encompasses the full spectrum of infrastructure decisions, service contracts, and operational frameworks that determine how a trained model is made available for inference at scale. Hosting architecture choices — from fully managed cloud APIs to on-premises bare-metal clusters — directly shape latency profiles, data residency compliance posture, total cost of ownership, and organizational control over model behavior. This page covers the classification of LLM deployment models, the structural mechanics of each, and the tradeoffs that define procurement and engineering decisions across enterprise, government, and research contexts. For a broader map of the stack in which these services operate, see the AI Stack Components Overview.

• Definition and Scope
• Core Mechanics or Structure
• Causal Relationships or Drivers
• Classification Boundaries
• Tradeoffs and Tensions
• Common Misconceptions
• Deployment Evaluation Checklist
• Reference Matrix: Hosting Model Comparison

Definition and Scope

LLM deployment refers to the operational process of exposing a trained language model — whether proprietary, open-weight, or fine-tuned — to an inference workload. This is distinct from model training (covered under AI Model Training Services) and from the data preparation work described in AI Data Pipeline Services. Deployment scope includes the hardware layer (GPU clusters, accelerator pods), the serving framework (Triton Inference Server, vLLM, TGI), the API surface exposed to downstream applications, and the monitoring and observability instrumentation required to maintain service-level agreements.

The National Institute of Standards and Technology (NIST) Artificial Intelligence Risk Management Framework (NIST AI RMF 1.0) identifies deployment as a distinct lifecycle phase with its own governance requirements, separating it from design and training phases. Within NIST AI RMF, the "Deploy" function requires documented risk controls, traceability of model versions, and ongoing measurement — obligations that vary in burden depending on whether the deploying organization hosts the model itself or consumes it via a managed API.

Scope boundaries matter commercially and legally. A firm consuming an LLM through a public API (e.g., a foundation model provider's inference endpoint) holds a different set of liabilities and compliance responsibilities than one operating a self-hosted cluster. Regulated industries — financial services under OCC guidance, healthcare under HIPAA, federal agencies under OMB Memorandum M-24-10 (2024) — face deployment-specific documentation requirements independent of which model was selected.

Core Mechanics or Structure

LLM inference is computationally asymmetric relative to training. A 70-billion-parameter model requires approximately 140 GB of GPU VRAM at FP16 precision for weights alone, before accounting for KV-cache allocation during long-context generation. Serving frameworks address this through techniques including tensor parallelism (splitting weight matrices across multiple GPUs), pipeline parallelism (assigning transformer layers to sequential GPU stages), and quantization (reducing precision to INT8 or INT4 to shrink memory footprint by 2–4×).

The core serving stack for production LLM deployment typically involves four layers:

1. Compute substrate — GPU instances or accelerator pods (see GPU Cloud Services and On-Premises AI Deployment for infrastructure options).
2. Inference runtime — frameworks such as NVIDIA Triton Inference Server, Hugging Face Text Generation Inference (TGI), or vLLM, which handle batching, memory management, and token streaming.
3. API gateway — the HTTP/REST or gRPC layer that routes requests, enforces rate limits, handles authentication, and logs request metadata.
4. Observability layer — metrics pipelines feeding latency, throughput (tokens per second), error rates, and model drift indicators into monitoring dashboards (see AI Observability and Monitoring).

Autoscaling logic in cloud-hosted deployments typically responds to GPU utilization or queue depth metrics, scaling replica counts to meet demand. Cold-start latency — the time to load model weights into GPU memory from scratch — can exceed 90 seconds for large models, making warm-pool strategies a standard production requirement.

Causal Relationships or Drivers

Three causal clusters drive deployment architecture decisions:

Regulatory data-handling requirements are the primary driver of self-hosted versus managed-API choices in regulated sectors. HIPAA's minimum necessary standard and the FedRAMP authorization framework (FedRAMP Program Management Office) constrain which cloud tenancy models are permissible. A FedRAMP High authorization, required for federal agency SaaS processing CUI (Controlled Unclassified Information), imposes audit controls that most commercial LLM API providers do not satisfy, pushing agencies toward private deployment or specialized GovCloud offerings.

Inference volume economics determine whether per-token API pricing or reserved GPU compute is more cost-efficient. At low call volumes (under ~1 million tokens per day), managed APIs typically undercut the amortized cost of a dedicated GPU instance. Above approximately 10–50 million tokens per day (depending on model size and provider pricing), self-hosted inference on reserved or spot GPU instances often produces lower per-token costs, though this threshold shifts with hardware generation and provider discounting.

Latency and throughput SLAs drive architectural differentiation between streaming token delivery (Server-Sent Events or WebSocket) and batch inference pipelines. Real-time conversational applications target time-to-first-token (TTFT) under 500 milliseconds; batch document-processing pipelines tolerate latency measured in seconds or minutes in exchange for higher throughput. These requirements propagate upward into hardware selection and serving framework configuration.

Model customization requirements — whether through Fine-Tuning Services or Retrieval-Augmented Generation Services — also shape hosting decisions, as fine-tuned adapter weights require either a hosting environment that supports LoRA hot-swapping or a dedicated deployment separate from the base model.

Classification Boundaries

LLM deployment models fall into four structurally distinct categories:

Public managed API — The operator consumes inference capacity from a foundation model provider's shared infrastructure via API key. The provider manages all hardware, scaling, and model versioning. No model weights reside on operator infrastructure. Data flows to third-party servers; contractual data processing agreements govern retention and usage. Relevant context: Foundation Model Providers and AI API Services.

Dedicated hosted instance — The operator contracts for exclusive compute capacity within a cloud provider's environment, often a single-tenant GPU cluster. Model weights are loaded into operator-controlled (though cloud-provisioned) infrastructure. This boundary satisfies some data-residency requirements without requiring on-premises hardware. Relates to Managed AI Services.

Private cloud or VPC deployment — The operator runs inference within a logically isolated Virtual Private Cloud on a hyperscale cloud provider. Full control over networking, IAM, and logging. Model weights may be operator-supplied (open-weight models) or licensed for private deployment. This category encompasses most enterprise deployments on hyperscale infrastructure, as described under AI Infrastructure as a Service.

On-premises or air-gapped deployment — Model weights and inference compute reside on hardware physically controlled by the operator. Required for classified environments (DoD IL5/IL6), certain financial trading applications, and organizations with strict data sovereignty mandates. Operational overhead is highest in this category.

Tradeoffs and Tensions

The central tension in LLM deployment is between operational simplicity and control granularity. Managed API consumption eliminates infrastructure management but surrenders control over model versioning, token rate limits, data handling, and cost predictability. Self-hosted deployment restores those controls at the cost of MLOps staffing, hardware procurement cycles, and ongoing maintenance burden — areas addressed by MLOps Platforms and Tooling.

A second tension exists between cost optimization and performance headroom. Aggressive quantization (INT4) reduces GPU memory requirements by up to 75% relative to FP32 but degrades output quality on reasoning-intensive tasks, a degradation that varies by model architecture and is not uniformly disclosed by serving frameworks. Organizations must benchmark quality-accuracy tradeoffs on their specific task distributions before committing to quantized serving.

Vendor lock-in versus ecosystem breadth is a third contested area. Proprietary managed APIs offer rapid integration but create switching costs if pricing changes or the provider deprecates a model version. Open-weight model deployment (LLaMA family, Mistral, Falcon) trades away support guarantees for portability. The Open-Source vs. Proprietary AI Services framework provides a structured basis for that evaluation.

Security surface area expands with deployment complexity. An AI Security and Compliance Services assessment is a standard precursor to production deployment in regulated environments, covering prompt injection attack vectors, output logging requirements, and model inversion risks.

Common Misconceptions

Misconception: Managed API deployment eliminates compliance obligations.
Correction: Deploying via API shifts infrastructure responsibilities to the provider but does not transfer data processing obligations. Under GDPR Article 28 and HIPAA's Business Associate Agreement requirements, the operator remains a data controller and must ensure the provider's data processing practices are contractually documented. OMB M-24-10 explicitly requires federal agencies to maintain documentation of AI systems regardless of deployment model.

Misconception: Larger models always require dedicated hardware.
Correction: Quantized versions of 70B-parameter models can serve inference on a single 80 GB A100 GPU via INT4 quantization. A 7B-parameter model in INT4 requires approximately 4 GB VRAM, fitting on consumer-grade hardware. Hardware requirements are functions of precision, batch size, and context length — not model parameter count alone.

Misconception: Serverless inference eliminates cold-start latency for LLMs.
Correction: Serverless GPU infrastructure (as distinct from serverless CPU compute) still requires weight loading on cold starts. At 70B parameters, FP16 weight files exceed 130 GB; loading from object storage into GPU VRAM at practical network speeds produces cold starts measured in minutes, not milliseconds.

Misconception: Open-weight models are free to deploy at scale.
Correction: Open-weight licenses (e.g., Meta's LLaMA 3 Community License) impose commercial use restrictions above 700 million monthly active users and prohibit use of model outputs to train competing foundation models. GPU compute costs for self-hosted inference are real and scale linearly with token volume.

Checklist or Steps (Non-Advisory)

The following phases represent the standard operational sequence for LLM deployment evaluation and execution, as reflected in NIST AI RMF deployment-phase practices and enterprise MLOps reference architectures (MLOps Foundation, ml-ops.org):

1. Requirements definition — Document latency SLAs (TTFT targets), throughput requirements (peak tokens per second), data residency constraints, and model quality benchmarks for the target task.
2. Model selection and licensing review — Identify candidate models; audit applicable open-weight or commercial licenses for deployment scope; confirm compatibility with AI Service Level Agreements obligations.
3. Deployment model classification — Apply the public API / dedicated hosted / private cloud / on-premises classification to each candidate based on regulatory requirements and cost modeling.
4. Infrastructure provisioning — Specify GPU instance types, memory configurations, and network topology; engage GPU Cloud Services or on-premises procurement as appropriate.
5. Serving framework configuration — Select inference runtime (Triton, vLLM, TGI); configure tensor/pipeline parallelism, batching strategy, and KV-cache allocation.
6. Security hardening — Apply input validation, output logging, IAM scoping, and network egress controls; document threat model against NIST SP 800-218A (Secure Software Development Framework).
7. Observability instrumentation — Integrate token throughput, TTFT, error rate, and model drift metrics into the AI Observability and Monitoring pipeline.
8. Load testing and SLA validation — Run synthetic traffic at 2× expected peak; validate latency percentiles (p50, p95, p99) against documented SLAs.
9. Staged rollout — Deploy to a canary slice (typically 5–10% of traffic) before full cutover; monitor quality metrics for regression.
10. Cost governance baseline — Establish per-token or per-hour cost baselines; configure budget alerts; link to AI Stack Cost Optimization review cadence.

Reference Table or Matrix

LLM Hosting Model Comparison

References

• NIST AI Risk Management Framework 1.0 (NIST AI 100-1) — National Institute of Standards and Technology
• FedRAMP Program Management Office — General Services Administration
• NIST SP 800-218A: Secure Software Development Framework for AI and ML — NIST Computer Security Resource Center
• OMB Memorandum M-24-10: Advancing Governance, Innovation, and Risk Management for Agency Use of Artificial Intelligence — Office of Management and Budget (2024)
• MLOps Foundation Reference Architecture — ml-ops.org (open community reference)
• HHS HIPAA Security Rule Guidance — U.S. Department of Health and Human Services
• NIST SP 800-53 Rev 5: Security and Privacy Controls for Information Systems — National Institute of Standards and Technology

For an orientation to the full service landscape in which LLM deployment sits, the /index provides the top-level classification of AI stack services covered across this reference network.