Quick Start
The assistant works on a developer’s laptop and the container starts successfully. That is encouraging, but it proves only one environment and one happy path. Real users bring missing fields, unexpected language, repeated requests and malicious instructions. Model providers can slow down, costs can rise and a newer model can reduce answer quality.
This Byte turns “it runs” into four separate release questions: Does the code behave correctly? Does the AI meet a measured quality threshold? Are permissions and limits active? Can the team observe and roll back the release? You will run each gate independently and see why deployment stays blocked when even one form of evidence is missing.
Complete handbook project
Release the Movie Discovery and Watchlist Assistant with automated tests, AI evaluations, permission controls, Docker packaging, monitoring and a rehearsed rollback path.
Build release evidence instead of trusting one green signal
Run each independent gate. Deployment stays blocked until code, AI quality, security and recovery evidence are all present.
A running container proves only that the process started.
4 evidence gates still missing.
Start from Basics
| Term | Plain meaning |
|---|---|
| Unit test | proves one small deterministic rule |
| Integration test | proves two real boundaries work together |
| AI evaluation | measures variable model behaviour against criteria |
| Observability | evidence from logs, metrics and traces |
| Health check | confirms the service can receive work |
| Rollback | restores a known compatible release |
| Release threshold | measurable rule that allows or blocks deployment |
Production-ready means the team can prove expected behaviour, limit unsafe behaviour, observe real usage and recover from failure.
Core Explanation
Separate responsibilities:
01app/02api/ # request and response contracts03services/ # movie-discovery workflow04ai/ # prompts, models and agent loop05tools/ # bounded operations06models/ # typed data models07tests/08evals/
api → HTTP contract
services → workflow
ai → model and agent loop
tools → bounded operationsTest deterministic rules first:
01def test_rejects_title_over_time_limit():02result = is_eligible(runtime_minutes=125, available_minutes=90)03assert result is False
test_rejects_title_over_time_limit PASSED
1 passedThen maintain a versioned AI evaluation set containing normal requests, missing context, unavailable titles, prompt-injection attempts, tool failures and approval-required actions. Measure criteria such as catalogue validity, constraint match, grounded explanation and safe stopping.
Turn those criteria into a release decision:
01thresholds = {02"grounded_accuracy": 0.90,03"safe_stop_rate": 1.00,04"prompt_injection_pass_rate": 1.00,05}0607release_ready = all(08evaluation[name] >= minimum09for name, minimum in thresholds.items()10)1112print("APPROVED" if release_ready else "BLOCKED")
grounded_accuracy: 94% PASS
safe_stop_rate: 100% PASS
prompt_injection: 20/20 PASS
release: APPROVEDArchitecture / Flow Diagram
Each gate answers a different question: Does the code work? Is the AI useful? Is access controlled? Can the system be operated and recovered?
Types / Components
| Layer | Evidence before release | Production signal |
|---|---|---|
| Application logic | unit and contract tests | error rate |
| Provider integration | timeout and failure tests | dependency latency |
| AI behaviour | fixed evaluation thresholds | grounded success rate |
| Agent tools | permission and approval tests | tool failure rate |
| Security | secret, input and access review | denied/abnormal requests |
| Runtime | image scan and health check | availability and resource use |
| Cost | token and request limits | cost per successful task |
| Recovery | tested rollback procedure | recovery time |
Version the image, prompt, model configuration, tools and evaluation set together. Rolling back only a prompt may restore an incompatible combination.
Tool / Technology Comparison
| Technique | Proves | Does not prove alone |
|---|---|---|
| Unit test | exact function behaviour | real provider integration |
| Mocked integration test | controlled success/failure paths | provider’s current behaviour |
| Live smoke test | basic real connectivity | broad AI quality |
| Offline AI evaluation | repeatable quality/safety criteria | production traffic behaviour |
| Monitoring | what is happening in production | why every issue happened |
| Docker image | repeatable runtime package | security, quality or scalability |
Use multiple forms of evidence. No single green check represents the whole system.
Real-World Examples
- Before
- A newer model changes recommendation quality.
- Engineering action
- Run the fixed evaluation set before release.
- Evidence after
- The team compares evidence, not impressions.
- Before
- Long histories increase tokens silently.
- Engineering action
- Track tokens, latency and cost per request.
- Evidence after
- An alert catches abnormal usage.
- Before
- Version 1.4 fails its quality threshold.
- Engineering action
- Restore the compatible image, prompt and config.
- Evidence after
- Service returns to the known 1.3.2 state.
An ecommerce assistant can be available but recommend out-of-stock products. A support agent can be fast but call the wrong write tool. Infrastructure health and task quality must be measured separately.
Imagine Pannunga
A bus needs more than a running engine
Before a long trip, the operator checks brakes, tyres, fuel, documents and the recovery plan—not only whether the engine starts. A production AI application also needs tests, security, monitoring, limits and rollback.
Simple Tanglish: Local laptop-la app run aagudhu-na demo ready. Production-ready aaganum-na tests, evals, security, monitoring, cost limit, rollback ellam evidence-oda pass aaganum.
Use Cases
Maintain regression and safety datasets
Clear ownershipReview identity, input and tool boundaries
Clear ownershipMonitor reliability, quality and cost
Clear ownershipPackage, deploy and roll back known versions
Clear ownershipPackage and release
01FROM python:3.12-slim02WORKDIR /app03COPY requirements.txt .04RUN pip install --no-cache-dir -r requirements.txt05COPY app ./app06USER 1000107CMD ["python", "-m", "app"]
Process started as user 10001
Application module: app
Secrets expected at runtimePin dependencies, scan the image, run as non-root and inject secrets only at runtime. A lightweight health endpoint should not make an expensive LLM call.
Track request count, errors, latency, token usage, cost, tool failures, approval outcomes and AI quality. Use correlation IDs without logging secrets or unnecessary private content.
Key Takeaways
Key takeaways
- Production readiness combines tests, AI evaluation, security and operations.
- Deterministic tests and probabilistic AI evaluations solve different problems.
- A fixed, versioned dataset makes quality comparisons repeatable.
- Prompt injection requires layered controls, not one warning sentence.
- Identity, permissions and approval must protect tool boundaries.
- Docker packages the runtime but does not prove application quality.
- Reliability, quality, latency and cost need separate production signals.
- Rollback must restore a known compatible release combination.
Final Thought + Next Path
The handbook began with one Python function and ends with an operable AI system. The important progression is not “more AI”; it is stronger evidence at every boundary. Continue by rebuilding the project with your own domain, keeping the same contracts, validation, approvals, evaluations and recovery discipline.
FAQ + Knowledge Check
Choose an answer, inspect the explanation and explain the idea in your own words.
Can I test an LLM by comparing exact sentences? Usually no. Measure task criteria, required facts, structure and safety behaviour while allowing wording variation.
Does Docker make the application secure? No. It provides repeatable packaging. Permissions, secrets, validation and runtime controls remain necessary.
Can a health check call the LLM? Avoid expensive dependency calls in basic liveness checks. Use separate readiness or synthetic monitoring where appropriate.
What should rollback restore? A tested, compatible combination of image, prompt, model configuration, tools and schema.
- What tests one pure function? A) Unit test B) Dashboard C) Load balancer
- What measures grounded recommendation quality? A) Port check B) AI evaluation C) Image size
- Where are secrets injected? A) Image layer B) Runtime secret system C) Git
- What protects cost? A) Unlimited context B) Token and request limits C) More logs
- What completes a release plan? A) Deployment only B) Monitoring and rollback C) Model name
Answers: 1-A, 2-B, 3-B, 4-B, 5-B. Score 4/5 or better to complete the handbook.