Syllabus
Syllabus for LLMOps and Evaluation module
| Field | Value |
|---|
| Technical Group | ? |
| Topic Name | LLMOps and Evaluation |
| Topic Code | ? |
| Version | 1.0 |
| Training Audience | Freshers |
[!NOTE]
This topic introduces LLMOps and Evaluation knowledge; adapting trainees with skills, lessons, and practices specific to evaluating RAG systems, implementing observability tools like LangFuse and LangSmith, and designing deep architectural experiments.
| Name | Code | Description |
|---|
| Ragas Evaluation Metrics | ? | Evaluate RAG quality using Faithfulness, Answer Relevancy, Context Precision, and Context Recall. |
| Observability Tools | ? | Master open source and native observability tools (LangFuse & LangSmith) for tracing and debugging. |
| Experiment Comparison | ? | Design experimental frameworks to rigorously compare Naive, Advanced, GraphRAG, and Hybrid systems. |
? LO1 - Able to comprehend and apply Ragas framework metrics for automated LLM evaluation.? LO2 - Able to calculate Faithfulness and Answer Relevancy for generation quality.? LO3 - Able to assess Context Precision and Context Recall for retrieval performance.? LO4 - Able to set up and integrate LangFuse for Open Source tracing and prompt management.? LO5 - Able to configure LangSmith for native LangChain execution tracing and playground debugging.? LO6 - Able to implement best practices for production LLMOps (Sampling, PII, Alerts).? LO7 - Able to design an experimental infrastructure for comparing different RAG architectures.? LO8 - Able to analyze trade-offs (quality, cost, latency) across Naive, Advanced, Graph, and Hybrid setups.
- Unit 01: Ragas Evaluation Metrics
- Unit 02: Observability: LangFuse & LangSmith
- Unit 03: Experiment Comparison: Naive, Graph, Hybrid
| Activity Type | Percentage | Description |
|---|
| Concept/Lecture | 40% | Concepts, theory (3 Sessions) |
| Assignment/Lab | 40% | Assignment, Lab |
| Guides/Review | 10% | Self review/Cross review |
| Exam | 10% | Pre-Test, Final Topic Test |
- Advanced AI Module: LLMOps and Evaluation Guidelines
- Python 3.10+
- Ragas, LangChain, LangSmith, LangFuse
- Neo4j, Qdrant/ChromaDB
- IDE (VSCode / PyCharm)
| Component | Quantity | Weight (%) | Note |
|---|
| Quiz | 5 | 20% | In-class labs |
| Assignments | 2 | 30% | In-class labs |
| Final Practice Test | 1 | 50% | Comprehensive project |
- Total topic GPA >= 7.0/10
- Completed 100% Online video, Quiz, Assignments, Labs, Final Test
- Successful demo of the Final Project (RAG system with evaluation and tracing)
| Role | Responsibility/Criteria |
|---|
| Trainees | Must be familiar with basic RAG. |
| Trainer | Certified AI Engineer or higher. |
| Training | Daily offline training and labs. |
| Re-Test | Allowed 1 time for Failed status. |
| Marking | Direct grading of Practice tests. |
| Waiver Criteria | Equivalent industry certificates. |
| Others | Need personal laptop for labs. |