Fine – Tuning AI Early Release – Laurence Moroney

Grouped Query Attention In this simple example, where there are 6 heads, instead of each Q having its own dedicated KV, we have a situation where Q1 and Q2 share a KV, Q3 and Q4 share one etc. This gives us a 50% reduction in KV Cache size. In LLama 3 8B, there are 32 query heads grouped into 8 groups of 4, and each shares 8 key-value heads. This reduces the KV cache by 75% compared to standard multi-head attention, dramatically improving inference speed and reducing memory requirements—especially for long contexts.

So, when it comes to fine tuning, it’s good to consider models that use GQA because they can give you better inference performance without sacrificing much quality. Continued Innovation: Mistral and Mixtral Continuing the story of innovation being driven by smaller models, and the needs for efficiency that can be generally forgotten in larger models, let’s now explore Mistral and Mixtral. Mistral 7B, introduced in September 2023 demonstrated that a relatively small model—only 7B—could compete with much larger models on benchmarks. It did with several key innovations.

Sliding Window Attention As we pay attention to previous tokens, the longer the sequence of tokens, the more the scale of being able to manage it explodes in size. So, if every token has to look at every other token, including itself, we end up with a quadratic number of lookups. For 2 tokens, there would be 2 comparisons of 2 tokens each, so 4 lookups. With 10 tokens, this becomes 100.

With 1,000 tokens we now have 1 Million lookups! How large would this get for a context window of 128k tokens? Mistral introduced a sliding window, where each token only directly attends to the previous 4,096 tokens. Your initial thoughts on this might be that information outside the context window might be lost, but think about it a little. If the context is 128k tokens, and you are using a sliding window of only 4k, it may feel like the majority of tokens are ignored.

But in a multi-layer model, this isn’t necessarily true. Let’s explore this. Figure 2-9. Sliding Window Context movement. Imagine 10 tokens, with a 4 token context window. So, for example for tokens 7 through 10, the output of our sliding window will be the ‘rightmost’ token in Layer 2.

Customizing Large Language Models With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles. Laurence Moroney OceanofPDF.com Fine-Tuning AI by Laurence Moroney Copyright © 2027 Laurence Moroney. All rights reserved. Published by O’Reilly Media, Inc., 141 Stony Circle, Suite 195, Santa Rosa, CA 95401. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (https://oreilly.com).

For more information, contact our corporate/institutional sales department: 800-998-9938 or [email protected]. Acquisitions Editor: Nicole Butterfield Development Editor: Michele Cronin Production Editor: Katherine Tozer Interior Designer: David Futato Interior Illustrator: Kate Dullea July 2027: First Edition Revision History for the Early Release 2026-03-06: First Release See https://oreilly.com/catalog/errata.csp?isbn=9798341673373 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Fine- Tuning AI, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.

The views expressed in this work are those of the author and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk.

If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. 979-8-341-67333-5 [LSI] OceanofPDF.com Brief Table of Contents (Not Yet Final) Chapter 1: From Generalist to Specialist (available) Chapter 2: Model Architectures: The DNA of AI (available) Chapter 3: The Fine-Tuner’s Workshop (available) Chapter 4: Excavating Data (unavailable) Chapter 5: Synthetic Data (unavailable) Chapter 6: Formatting for Thought (unavailable) Chapter 7: The Mechanics of Adaptation (unavailable) Chapter 8: The Training Loop (unavailable) Chapter 9: Alignment (unavailable) Chapter 10: Case Study (unavailable) Chapter 11: Vision Language Models (unavailable) Chapter 12: Controlling the Canvas (unavailable) Chapter 13: Style Injection (unavailable) Chapter 14: Evaluation (unavailable) Chapter 15: Optimization and Quantization (unavailable) Chapter 16: Serving the Expert (unavailable) OceanofPDF.com Chapter 1.

From Generalist to Specialist A NOTE FOR EARLY RELEASE READERS With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles. This will be the 1st chapter of the final book. Please note that the GitHub repo will be made active later on. If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [email protected].

The honeymoon is over.

This is a short excerpt from the opening of “” by Unknown, quoted for review and introduction purposes. All rights belong to the copyright holders.

Book Information

• Unique ID: 5f799524f30128c4
• File Extension: .pdf
• File Size: 1,496,206 bytes (1.427 MB)
• Title: –
• Author: Unknown
• ISBN: 9798341673373, 9798341673335
• Pages: 87
• Language: English (en)

Reading & Word Statistics

• Estimated Reading Time: 89.82 minutes
• Total Words: 17,964
• Total Characters: 109,198
• Average Words per Page: 206.48
• Average Characters per Page: 1255.15

Most Frequent Words

model (139), models (93), like (78), training (64), gpu (64), it’s (62), tokens (61), attention (57), one (53), use (51), data (44), chapter (43), using (43), vram (43), need (42), see (41), memory (41), figure (37), get (36), also (35), fine (34), available (31), gpus (31), cuda (31), token (30), words (30), context (29), different (29), don’t (29), many (28), look (28), that’s (28), weights (28), work (27), example (27), window (27), tuning (26), you’re (26), cloud (26), much (26), you’ll (25), output (25), time (25), size (24), fine-tuning (23), code (23), book (23), language (22), first (22), single (22), system (22), hardware (22), now (21), lora (21), pytorch (21), nvidia (21), without (20), better (20), every (20), inference (20), layer (20), prompt (19), per (19), version (19), ultimately (18), think (18), architecture (18), understand (18), make (18), parameters (18), machine (18), layers (18), run (18), word (18), position (18), heads (18), bit (18), across (17), release (17), information (17), used (17), good (17), note (17), consider (17), doesn’t (17), step (17), smaller (17), sequence (17), multiple (17), start (16), llama (16), embedding (16), between (16), learning (16), stack (16), gradients (16), colab (16), qlora (16), com (15), take (15).

PDF Download