LLM notes

llm.md

Notebook

Theory

LLM

• Build LLM from scratch in Python using createllm package - https://pythonscholar.com/build-a-large-language-model-from-scratch/
• Private LLM using databricks-dolly-15k (approximately 15,000 instruction/response fine-tuning records) - https://www.leewayhertz.com/build-private-llm/
• The Emergence Of Large Language Model (LLM) API Build Frameworks - https://cobusgreyling.medium.com/the-emergence-of-large-language-model-llm-api-build-frameworks-78d83d68eeda
• Corpus size and LLM - https://genai.stackexchange.com/q/613/2269
• Restrict LLM responses to specific dataset - https://genai.stackexchange.com/q/167/2269
• Fine tuning the LLaVA Vision LLM on AWS - https://medium.com/@mr.sean.ryan/fine-tuning-the-llava-vision-llm-on-aws-2ba46b7dcec9
• Time-LLM: Reprogram an LLM for Time Series Forecasting - https://towardsdatascience.com/time-llm-reprogram-an-llm-for-time-series-forecasting-e2558087b8ac
• Running Your Very Own Local LLM - https://yc.prosetech.com/running-your-very-own-local-llm-6d4db99c0611
• What are 1-bit LLMs? - https://medium.com/data-science-in-your-pocket/what-are-1-bit-llms-3f2ae4b40fdf
• Implementing the Transformer Encoder from Scratch in TensorFlow and Keras - https://machinelearningmastery.com/implementing-the-transformer-encoder-from-scratch-in-tensorflow-and-keras/
• Unleashing the Power of Language Models: A Deep Dive into Language Foundation Model Tuning Strategies - https://ai.plainenglish.io/unleashing-the-power-of-language-models-a-deep-dive-into-language-foundation-model-tuning-4f1e96be7ddf
• Understanding Large Language Models - Words vs Tokens - https://kelvin.legal/understanding-large-language-models-words-versus-tokens/
• Hands-On LangChain for LLM Applications Development: Output Parsing - https://pub.towardsai.net/hands-on-langchain-for-llm-applications-development-output-parsing-876354434462
• What are foundation models? - https://research.ibm.com/blog/what-are-foundation-models
• Your Guide to the LLM Ecosystem - https://pub.aimind.so/your-guide-to-the-llm-ecosystem-f67826c84be8
• Large Language Model (LLM) Stack — Version 5 - https://cobusgreyling.medium.com/large-language-model-llm-stack-version-5-5a9306870e7f
• Building a Million-Parameter LLM from Scratch Using Python - https://levelup.gitconnected.com/building-a-million-parameter-llm-from-scratch-using-python-f612398f06c2
• Deploy your own Open-Source Language Model: A Comprehensive Guide - https://blog.zhaw.ch/artificial-intelligence/2023/04/20/deploy-your-own-open-source-language-model/
• Best practices for building LLMs - https://stackoverflow.blog/2024/02/07/best-practices-for-building-llms/
• Understanding Encoder And Decoder LLMs - https://magazine.sebastianraschka.com/p/understanding-encoder-and-decoder
• How does the (decoder-only) transformer architecture work? - https://ai.stackexchange.com/q/40179/75530
• LLM Agents - https://www.promptingguide.ai/research/llm-agents
• https://python.langchain.com/docs/use_cases/sql/
• Building an LLM Stack, Part 1: Implementing Encoders and Decoders - https://deepgram.com/learn/building-an-llm-stack-1-implementing-encoders-and-decoders

LLM metrics

• The Secrets of Large Language Models Parameters
• Model Cards for Model Reporting
• A simple way to create ML Model Cards in Python
• HF Model Card writing tool
• Model Card Guidebook

LLM Customizations

• Mastering LLM Techniques: Customization
• The architecture of today’s LLM applications

LLM Evaluation

• How to choose your LLM - https://community.aws/posts/how-to-choose-your-llm
• How I selected my GenAI and Large Language Model (LLM) Platform - https://medium.com/@nayan.j.paul/how-i-selected-my-genai-and-large-language-model-llm-platform-cfe6da358b25
• LLM Evaluation: Benchmarking Performance and Metrics - https://aisera.com/blog/llm-evaluation/
• LLM Evaluation: Everything You Need To Run, Benchmark LLM Evals - https://arize.com/blog-course/llm-evaluation-the-definitive-guide/
• W&B Prompts - https://docs.wandb.ai/guides/prompts
• Evaluating Large Language Model (LLM) systems: Metrics, challenges, and best practices - https://medium.com/data-science-at-microsoft/evaluating-llm-systems-metrics-challenges-and-best-practices-664ac25be7e5
• Decoding LLM Performance: A Guide to Evaluating LLM Applications - https://amagastya.medium.com/decoding-llm-performance-a-guide-to-evaluating-llm-applications-e8d7939cafce
• Amazon Bedrock Model Evaluation - https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation.html
• How to Evaluate LLMs: A Complete Metric Framework - https://www.microsoft.com/en-us/research/group/experimentation-platform-exp/articles/how-to-evaluate-llms-a-complete-metric-framework/
• MLflow LLM Evaluate - https://mlflow.org/docs/latest/llms/llm-evaluate/index.html

LLM fine-tuning

• Comprehensive Guide for Finetuning - https://genai.stackexchange.com/q/564/2269
• Fine tuning pipeline for open-source LLMs - https://paulabartabajo.substack.com/p/lets-fine-tune-an-open-source-llm
• When Should You Fine-Tune LLMs? - https://towardsdatascience.com/when-should-you-fine-tune-llms-2dddc09a404a
• Large Language Models: to Fine-tune or not to Fine-tune? - https://www.ml6.eu/blogpost/fine-tuning-large-language-models
• Fine tuning: what is it good for? - https://community.openai.com/t/fine-tuning-what-is-it-good-for/428080?u=nsubrahm

RAG

• What is RAG? - https://aws.amazon.com/what-is/retrieval-augmented-generation/
• Retrieval Augmented Generation (RAG) - https://docs.aws.amazon.com/sagemaker/latest/dg/jumpstart-foundation-models-customize-rag.html
• Retrieval augmented generation: Keeping LLMs relevant and current - https://stackoverflow.blog/2023/10/18/retrieval-augmented-generation-keeping-llms-relevant-and-current/
• Retrieval Augmented Generation (RAG) - https://www.promptingguide.ai/techniques/rag
• Build Industry-Specific LLMs Using Retrieval Augmented Generation - https://towardsdatascience.com/build-industry-specific-llms-using-retrieval-augmented-generation-af9e98bb6f68
• RAG Vs Fine tuning Vs Both - https://medium.com/@ramprasathsee/rag-vs-fine-tuning-vs-both-3cb25857d921
• GraphRAG: Unlocking LLM discovery on narrative private data - https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
• What is a Vector Database? - https://zilliz.com/learn/what-is-vector-database
• Knowledge Bases for Amazon Bedrock - https://aws.amazon.com/bedrock/knowledge-bases/
• Retrieval Augmented Generation (RAG) Architecture based on AWS - https://shabarish033.medium.com/retrieval-augmented-generation-rag-architecture-based-on-aws-fc449b708b04

GenAI stack

• What every investor should know about the GenAI tech stack - https://raphaelledornano.medium.com/what-every-investor-should-know-about-the-genai-tech-stack-813cc04a5249
• Understanding the GenAI Tech Stack : Part 3 — Foundation Models - https://raphaelledornano.medium.com/understanding-the-genai-tech-stack-part-3-foundation-models-8c3a9ad2c49b
• Understanding the GenAI Tech Stack : Part 4 — Application Models - https://raphaelledornano.medium.com/understanding-the-genai-tech-stack-part-4-application-models-8a92fc30e3ef
• GenAI Dev Stack, LLMOps & Vector Databases! - https://www.linkedin.com/pulse/genai-dev-stack-llmops-vector-databases-pavan-belagatti-wmcmc/
• GenAI Stack Walkthrough: Behind the Scenes With Neo4j, LangChain, and Ollama in Docker - https://neo4j.com/developer-blog/genai-app-how-to-build/
• Generative AI Tech Stack: A Complete Guide - https://flyaps.com/blog/generative-ai-tech-stack-a-complete-guide/
• Understanding Generative AI: A Tech Stack Breakdown - https://www.orioninc.com/blog/understanding-generative-ai-a-tech-stack-breakdown/
• The LLM App Stack — 2024 - https://medium.com/plain-simple-software/the-llm-app-stack-2024-eac28b9dc1e7

Prompt Engineering

• What is Prompt Engineering? - https://aws.amazon.com/what-is/prompt-engineering/
• Prompt engineering for foundation models - https://docs.aws.amazon.com/sagemaker/latest/dg/jumpstart-foundation-models-customize-prompt-engineering.html
• What is prompt engineering? - https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-prompt-engineering.html
• Prompt engineering guidelines - https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-engineering-guidelines.html
• Best Practices for Prompt Engineering with Amazon CodeWhisperer - https://aws.amazon.com/blogs/devops/best-practices-for-prompt-engineering-with-amazon-codewhisperer/

Hugging Face

• Total noob’s intro to Hugging Face Transformers
• Hugging Face Transformers - Hugging Face Transformers is an open-source Python library that provides access to thousands of pre-trained Transformers models for natural language processing (NLP), computer vision, audio tasks, and more. It simplifies the process of implementing Transformer models by abstracting away the complexity of training or deploying models in lower level ML frameworks like PyTorch, TensorFlow and JAX.
• Hugging Face Hub - The Hugging Face Hub is a collaboration platform that hosts a huge collection of open-source models and datasets for machine learning, think of it being like Github for ML. The hub facilitates sharing and collaborating by making it easy for you to discover, learn, and interact with useful ML assets from the open-source community. The hub integrates with, and is used in conjunction with the Transformers library, as models deployed using the Transformers library are downloaded from the hub.
• Hugging Face Spaces - Spaces from Hugging Face is a service available on the Hugging Face Hub that provides an easy to use GUI for building and deploying web hosted ML demos and apps. The service allows you to quickly build ML demos, upload your own apps to be hosted, or even select a number of pre-configured ML applications to deploy instantly.

Miscellaneous

According to Alireza Goudarzi, senior researcher of machine learning (ML) for GitHub Copilot: “LLMs are not trained to reason. They’re not trying to understand science, literature, code, or anything else. They’re simply trained to predict the next token in the text.” Source

StackOverflow

and for top tier prompts, you find on www.godtierprompts.com 🪄