Quick Start
Capstone outcome
You will package an API-based Generative AI service without embedding its provider key, understand the production request path and use a deployment checklist covering health, security, cost and observability.
Start from Basics
Consider a small FastAPI service with one endpoint. It receives a user question, validates the input, calls an LLM provider and returns a bounded answer. The container should package application code and dependencies. The deployment environment should supply the API key and environment-specific values.
01import os02from fastapi import FastAPI, HTTPException03from pydantic import BaseModel0405app = FastAPI()0607class AskRequest(BaseModel):08question: str0910@app.get("/health")11def health():12return {"status": "healthy"}1314@app.post("/ask")15def ask(request: AskRequest):16api_key = os.getenv("LLM_API_KEY")17if not api_key:18raise HTTPException(500, "LLM provider is not configured")19# Call the approved provider client here.20return {"answer": f"Demo response for: {request.question}"}
The demo deliberately omits a real provider call so the container lesson does not become tied to one SDK version. For this Byte, remember the boundary: code and libraries belong in the image; provider keys and environment-specific values do not.
Core Explanation
Decide what belongs inside the image
Open each boundary before releasing an AI application.
Package repeatable application files in the versioned image.
$ docker run --read-only support-app:1.0Click execute — no real system changes are made
The created object will appear here
Architecture / Flow Diagram
Types / Components
Build a production-minded image
01FROM python:3.12-slim0203ENV PYTHONDONTWRITEBYTECODE=1 \04PYTHONUNBUFFERED=10506WORKDIR /app0708RUN addgroup --system app && adduser --system --ingroup app app0910COPY requirements.txt .11RUN pip install --no-cache-dir -r requirements.txt1213COPY --chown=app:app app.py .1415USER app16EXPOSE 80001718CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]
Example requirements.txt:
01fastapi==0.116.102uvicorn==0.35.0
The image uses a small official Python base, installs dependencies before copying frequently changing source, runs as a non-root user and starts one foreground process.
Keep the API key outside the image
Avoid these patterns:
01# Do not do this02ENV LLM_API_KEY=real-secret-value03COPY .env .
Image layers may be cached, inspected, pushed to a registry or shared with other environments. Deleting the line in a later layer does not reliably erase the earlier secret history.
Simple Tanglish — image boundary
Image-க்குள் app code + dependencies மட்டும். API key, database password மாதிரி secrets image-க்கு outside-ல் இருக்கணும். Container start ஆகும்போது hosting platform secret setting மூலம் value கொடுக்கணும்.
For local development, inject a value at runtime:
01docker build -t genai-api:1.0 .02docker run --rm -p 8000:8000 \03-e LLM_API_KEY="$LLM_API_KEY" \04genai-api:1.0
For production, use the cloud platform’s secret manager and workload identity or access policy. The platform should expose the value only to the running workload that needs it.
Local Container vs Deployed AI Service
| Question | Local container | Deployed AI service |
|---|---|---|
| Who reaches it? | Developer on localhost | Authenticated users through managed ingress |
| Where is the key? | Local runtime environment | Platform secret system |
| What proves success? | One request works | Health, quality, latency, safety and cost evidence |
| What happens on failure? | Developer restarts | Platform policy, alerts and rollback |
Real-World Examples
- Before Docker
- A developer’s local API key must not travel inside the image.
- Team action
- Inject the key from the platform at runtime.
- After Docker
- The same secret-free image can move safely.
- Before Docker
- The container process starts, but the app may not be ready for traffic.
- Team action
- Separate liveness and readiness checks.
- After Docker
- The platform sends traffic only when appropriate.
- Before Docker
- A new AI version may change latency, quality or cost.
- Team action
- Release the version to limited traffic and keep rollback ready.
- After Docker
- The team observes evidence before wider rollout.
Imagine Pannunga
Imagine an airport journey. Packing the suitcase correctly is important, but the suitcase alone does not operate the airport. Check-in, identity verification, security screening, gates and tracking systems make the complete journey safe and observable.
The Docker image is the prepared suitcase. The deployment platform supplies the controlled entrance, runtime secret, health checks, traffic limits, monitoring and rollback. A well-packed image is essential—but it is only one part of a production AI service.
A useful assistant must survive more than a demo
A college deploys an assistant that explains course concepts. The local demo works with one developer’s API key. In production, hundreds of learners may ask questions together, submit very long prompts or trigger provider rate limits.
The team deploys one versioned image, injects the key from a secret manager, caps input size, sets provider timeouts, records latency and error categories, and limits initial traffic. A health endpoint checks whether the web process can serve requests; a separate readiness strategy verifies dependencies without generating expensive model calls every few seconds.
The production question is no longer “Does the container start?” It is “Can the complete system serve learners safely, predictably and within cost?”
Use Cases
“Does the image contain only app code?”
Keep provider secrets outside“Does it run with safe permissions?”
Scan and run as non-root“Can this version roll forward and back?”
Deploy an immutable image tag“Is it healthy, useful and within cost?”
Monitor health, quality, latency and usageKey Takeaways
- Package application code and dependencies in the image.
- Inject provider keys at runtime through the platform.
- Run the application with only the permissions it needs.
- Use an immutable image version for rollout and rollback.
- Liveness and readiness answer different questions.
- A healthy container does not prove useful AI answers.
- Observe latency, errors, quality, safety and cost together.
- Release gradually and keep a tested rollback path.
Final Thought + Next Path
A container makes the AI application repeatable; the surrounding platform makes it operable. Keep secrets outside, deploy a clear image version, check health without expensive model calls and observe answer quality as well as uptime. This completes the Docker handbook path from first image to controlled AI deployment.
FAQ + Knowledge Check
Choose an answer, inspect the explanation and explain the idea in your own words.
Should the LLM API key be copied into the image? No. Inject it at runtime through the deployment platform.
Does a healthy process prove good AI answers? No. Quality evaluation is separate from infrastructure health.
Knowledge check: Which image property makes rollback clear? An immutable version tag or digest. What should limited traffic provide? Evidence before wider rollout.
Health, readiness and AI-specific checks
| Check | Question answered | Avoid |
|---|---|---|
| Liveness | Is the application process alive? | Calling the paid LLM on every probe |
| Readiness | Should this instance receive traffic? | Returning ready before required setup completes |
| Functional evaluation | Are answers meeting expected quality? | Using only one happy-path prompt |
| Cost metric | What does usage cost per request or learner? | Monitoring tokens without business context |
| Safety signal | Are blocked or risky requests handled correctly? | Logging sensitive prompt content indiscriminately |
Before cloud deployment
- Tag the image with an immutable version or commit identifier.
- Scan dependencies and the built image for known vulnerabilities.
- Confirm the process runs without root privileges.
- Supply secrets through the platform, never the image.
- Configure CPU, memory, concurrency, timeout and scaling boundaries.
- Define health and readiness endpoints.
- Capture structured errors, latency, provider usage and model cost.
- Run an evaluation set covering normal, edge and unsafe inputs.
- Release to a limited audience and keep a rollback path.
Troubleshooting map
| Symptom | Evidence to inspect | Possible cause |
|---|---|---|
| Container repeatedly restarts | platform events and application logs | startup command, missing variable or memory limit |
| Requests time out | latency trace by dependency | provider delay, network, concurrency or timeout mismatch |
401 from provider | provider error category and secret version | invalid, expired or unauthorized key |
| Local works, cloud fails | bind address, port and platform contract | wrong listening port or localhost binding |
| Cost suddenly rises | request count, token usage and retry metrics | unbounded input, retry storm or abuse |
| Answers degrade after release | evaluation results by version | prompt, model, retrieval or data change |
Common mistakes
Common mistakes
- Copying a provider key into the Dockerfile or build context.
- Using a paid model request as a frequent liveness probe.
- Deploying an unversioned image tag with no rollback evidence.
- Logging full prompts and outputs without privacy controls.
- Retrying every provider error and creating a cost-amplifying retry storm.
- Measuring infrastructure uptime while ignoring answer quality and user outcome.
Interview answer
How would you containerize and deploy an LLM application safely?
Strong answer: I build one minimal, versioned image that runs as a non-root user and contains no secrets. The deployment platform injects the provider key from its secret manager. I separate liveness from expensive functional evaluation, configure resource and concurrency limits, capture latency, errors and cost, test a representative evaluation set, and release gradually with rollback available.