Shipping your GenAI app to the real world

1 From notebook to product

So far, your GenAI code has probably lived in a Jupyter notebook or a script you run by hand. Deployment means packaging that code and putting it somewhere that runs continuously and is reachable over the internet, so other people (or other programs) can use it without you being involved each time.

Serving your app behind an API

The standard way to expose a GenAI app is behind an API (Application Programming Interface) — a fixed "front door" that accepts requests and returns responses. Instead of running a cell, a user's browser or app sends an HTTP request like POST /chat with their question, and your server returns the model's answer as JSON. Your server sits in the middle: it receives the request, builds the prompt, calls the LLM (recall the request → response flow from Session 1), and sends the result back.

Demo vs production — they are not the same

A demo just needs to work once, in front of a friendly audience, on your laptop. Production means real users depend on it, so it must be reliable, secure, observable, and affordable. This is the single biggest mindset shift of this session.

POST /chat
{ "message": "What is our refund policy?" }

→ 200 OK
{ "answer": "You can request a refund within 30 days...",
  "sources": ["policy.pdf p.4"] }

2 Deploying on AWS

AWS (Amazon Web Services) is the most popular cloud provider. You don't need to memorise its hundreds of services — for a GenAI app, a handful of patterns cover almost everything. Let's stay conceptual but concrete.

The big decision first: hosted model API vs self-hosting

Common patterns for deploying your app code

3 Scaling for many users

Scaling is handling more users without your app slowing down or crashing. There are two directions.

Vertical vs horizontal scaling

Load balancing

When you run many copies, a load balancer sits in front and distributes incoming requests across them, so no single copy gets overwhelmed. If one copy becomes unhealthy, the balancer stops sending traffic to it. This is what makes horizontal scaling work.

Stateless services + external state

For horizontal scaling to work, each copy of your server must be stateless — it shouldn't store anything important in its own memory, because the next request from the same user might land on a different copy. This connects directly to Session 1: the LLM itself is stateless, and to memory sessions from earlier in the course where we stored conversation history externally.

Async & queues for long agent runs

A simple chat reply comes back in seconds. But an agent (Sessions on agents & tool use) might call many tools, do multi-step reasoning, and take minutes. Making a user's request "hang" for minutes is fragile — connections drop, and one slow job ties up a server. The fix is asynchronous (async) processing with a queue.

4 Caching to cut cost & latency

Every LLM call costs tokens (money) and time (latency) — recall the token economics from Session 1. Caching stores previous results so repeated or similar work can skip the model entirely. A cache hit means the answer was found in the cache (fast, free); a cache miss means it wasn't, so you do the real work and store it for next time.

Three kinds of caching for GenAI

5 Rate limiting

Rate limiting controls how many requests are allowed in a given time window. It protects you in two directions: limiting your users so no one abuses or overloads your app, and respecting the model provider's limits so your app doesn't get blocked.

Provider limits: RPM and TPM

Model providers cap your usage, usually as:

• RPM (requests per minute) — how many API calls you can make per minute.
• TPM (tokens per minute) — how many tokens (input + output) you can push through per minute.

Go over either and the provider returns a 429 Too Many Requests error. Your app must handle this gracefully instead of crashing.

Strategies

6 CI/CD & reliability practices

CI/CD stands for Continuous Integration / Continuous Delivery (or Deployment). It's an automated pipeline that, whenever you change your code, runs your tests and (if they pass) deploys the new version — without error-prone manual steps.

Testing a GenAI app — it's not just normal tests

Regular software tests check exact outputs. But LLMs are non-deterministic, so you also need the AI-specific testing from earlier sessions:

• Normal unit/integration tests — your non-AI code (API routes, parsing, database calls).
• Prompt / eval tests — run each prompt across a set of example inputs and check the outputs are good. This is the prompt versioning & testing idea from Session 2 (prompt engineering), ideally at low temperature so results are repeatable.
• LLM-as-judge tests — use another model to grade your app's outputs against criteria (helpfulness, correctness, tone). This is the LLM-judge approach from Session 5, automated into your pipeline so quality is checked on every change.

Staged rollouts

Don't push a new version to everyone at once. Roll out gradually so a bad change hurts few people:

Monitoring & observability

Once live, you must see what's happening. This is exactly the tracing & observability from Session 8, now in production. Track:

• Logs — what happened, including errors.
• Metrics — request rate, latency, error rate, token usage & cost, cache hit rate.
• Traces — follow a single request through every step (prompt build → tool calls → model call → response), invaluable for debugging multi-step agents.

7 Capstone project & viva prep

Your capstone project is where you tie together everything from the course into one working GenAI application. The viva (oral examination / demo) is where you present and defend it. Here's how to do both well.

How to present a GenAI demo

• Lead with the problem. Start with who has the problem and why it matters — not with your tech stack.
• Show it working, live, early. A real run beats slides. Have a known-good example ready.
• Tell the architecture story. Walk the request → response flow (Session 1): prompt → retrieval/RAG → model → tools/agent → response. Show where caching, rate limiting, and observability fit.
• Explain your choices. Why this model? Why RAG vs fine-tuning? Why this temperature? Evaluators love reasoned trade-offs.
• Show the numbers. Token cost per request, latency, eval scores, cache hit rate — this proves production thinking.
• Be honest about limits. Name what doesn't work yet and what you'd do next. It reads as maturity, not weakness.

What evaluators look for

★ Course wrap-up — you did it! 🎉

That's the whole GenAI course. Take a second to appreciate how far you've come — you started not knowing what a token was, and you can now design, build, evaluate, deploy, and defend a real GenAI product.

Final self-check

📚 References & Further Reading

Class material

• 📄 Original course notes / handout (source sheet) — open the shared GenAI class material for this session.
• Class handout: "Deployment, Scaling & Final Demos".

Papers, docs & deep dives

• AWS Well-Architected Framework — the reliability, scaling, and cost pillars behind Topics 2–3.
• OpenAI production best practices — moving from notebook to production reliably.
• Anthropic prompt caching — prompt/KV caching to cut cost and latency (Topic 4).
• Anthropic rate limits (RPM/TPM) — provider limits and 429 handling from Topic 5.
• Amazon SQS developer guide — async queues + background workers for long agent runs.