Skip to main content
Ctrl+K
Omniverse - Home Omniverse - Home
  • Omniverse

Notations

  • Machine Learning Notations

Influential Ideas and Papers

  • Generative Pre-trained Transformers
    • Notations
    • The Concept of Generative Pre-trained Transformers (GPT)
    • The Implementation of Generative Pre-trained Transformers (GPT)
    • Training a Mini-GPT to Learn Two-Digit Addition
  • Low-Rank Adaptation Of Large Language Models
    • Concept
    • Implementation
  • Empirical Risk Minimization
    • Concept: Empirical Risk Minimization
    • Bayes Optimal Classifier
  • Is The Learning Problem Solvable?
    • Concept: Learning Theory
  • Lloyd’s K-Means Clustering Algorithm
    • Concept: K-Means Clustering
    • Implementation: K-Means (Lloyd)
    • Application: Image Compression and Segmentation
    • Conceptual Questions
  • Naive Bayes
    • Concept
    • Naives Bayes Implementation
    • Naive Bayes Application: Penguins
    • Naive Bayes Application (MNIST)
  • Mixture Models
    • Concept
    • Gaussian Mixture Models Implementation
  • Linear Regression
    • Concept
    • Implementation

Playbook

  • Training Dynamics And Tricks
    • How to Calculate the Number of FLOPs in Transformer Based Models?
    • Why Does Cosine Annealing With Warmup Stabilize Training?
    • How To Fine-Tune Decoder-Only Models For Sequence Classification Using Last Token Pooling?
    • How To Fine-Tune Decoder-Only Models For Sequence Classification With Cross-Attention?
    • How To Do Teacher-Student Knowledge Distillation?
  • Softmax Preserves Order, Is Translation Invariant But Not Invariant Under Scaling.
  • How to Inspect Function and Class Signatures in Python?

Probability Theory

  • Chapter 1. Mathematical Preliminaries
    • Permutations and Combinations
    • Calculus
    • Contour Maps
    • Exercises
  • Chapter 2. Probability
    • Probability Space
    • Probability Axioms
    • Conditional Probability
    • Independence
    • Baye’s Theorem and the Law of Total Probability
    • Summary
  • Chapter 3. Discrete Random Variables
    • Random Variables
    • Discrete Random Variables
    • Probability Mass Function
    • Cumulative Distribution Function
    • Expectation
    • Moments and Variance
    • Discrete Uniform Distribution
      • Concept
      • Application
    • Bernoulli Distribution
      • Concept
      • Application
    • Independent and Identically Distributed (IID)
    • Binomial Distribution
      • Concept
      • Implementation
      • Real World Examples
    • Geometric Distribution
      • Concept
    • Poisson Distribution
      • Concept
      • Implementation
    • Important
    • Exercises
  • Chapter 4. Continuous Random Variables
    • From Discrete to Continuous
    • Continuous Random Variables
    • Probability Density Function
    • Expectation
    • Moments and Variance
    • Cumulative Distribution Function
    • Mean, Median and Mode
    • Continuous Uniform Distribution
    • Exponential Distribution
    • Gaussian Distribution
    • Skewness and Kurtosis
    • Convolution and Sum of Random Variables
    • Functions of Random Variables
  • Chapter 5. Joint Distributions
    • From Single Variable to Joint Distributions
    • Joint PMF and PDF
      • Concept
    • Joint Expectation and Correlation
      • Concept
    • Conditional PMF and PDF
      • Concept
      • Application
    • Conditional Expectation and Variance
      • Concept
      • Exercises
    • Sum of Random Variables
      • Concept
    • Random Vectors
      • Concept
    • Multivariate Gaussian Distribution
      • Concept
      • Application: Plots and Transformations
      • Covariance Matrix is Positive Semi-Definite
      • Eigendecomposition and Covariance Matrix
      • The Geometry of Multivariate Gaussians
  • Chapter 6. Sample Statistics
    • Moment Generating and Characteristic Functions
      • Moment Generating Function
      • Application: Moment Generating Function and the Sum of Random Variables
      • Characteristic Function
    • Probability Inequalities
      • Probability Inequalities
      • Application: Learning Theory
    • Law of Large Numbers
      • Concept
      • Convergence of Sample Average
      • Application: Learning Theory
  • Chapter 8. Estimation Theory
    • Maximum Likelihood Estimation
      • Concept

Operations

  • Distributed Systems
    • Notations
    • Basics Of Distributed Data Parallelism
    • How to Setup SLURM and ParallelCluster in AWS
    • Ablations
  • Profiling
    • Synchronize CUDA To Time CUDA Operations
    • Profiling Code With Timeit
    • PyTorch’s Event And Profiler
    • Profile GPT Small Time And Memory
    • CUDA Memory Allocations
  • The Lifecycle of an AIOps System
    • Stage 1. Problem Formulation
    • Stage 2. Project Scoping And Framing The Problem
    • Stage 3. Data Pipeline (Data Engineering and DataOps)
      • Stage 3.1. Data Source and Formats
      • Stage 3.2. Data Model and Storage
      • Stage 3.3. Extract, Transform, Load (ETL)
    • Stage 4. Data Extraction (MLOps), Data Analysis (Data Science), Data Preparation (Data Science)
    • Stage 5. Model Development and Training (MLOps)
      • Stage 5.1. Model Selection
      • Stage 5.2. Metric Selection
      • Stage 5.3. Experiment Tracking And Versioning
      • Stage 5.4. Model Testing
    • Stage 6. Model Evaluation (MLOps)
    • Stage 7. Model Validation, Registry and Pushing Model to Production (MLOps)
    • Stage 8. Model Serving (MLOps)
    • Stage 9. Model Monitoring (MLOps)
    • Stage 10. Continuous Integration, Deployment, Learning and Training (DevOps, DataOps, MLOps)

Software Engineering

  • Config, State, Metadata Management
    • Configuration Management
    • Pydantic And Hydra
    • State And Metadata Management
  • Design Patterns
    • Dependency Inversion Principle
    • Named Constructor
    • Strategy
    • Registry
    • Context Object Pattern (God Object)
    • Factory Method
    • Singleton
  • Python
    • Init vs New
    • Global Interpreter Lock (GIL)
    • The Iterator Protocol
    • Decorator
    • Generators Over Lists For Memory Efficiency
    • Pydantic Is All You Need - Jason Liu
    • Do Not Use Mutable Default Arguments
    • Set Over List For Frequent Membership Tests
    • Late Binding Closures
    • Is vs Equality
  • Concurrency, Parallelism and Asynchronous Programming
    • Overview Of Concurrency, Parallelism, and Asynchronous Execution
    • Thread Safety
    • A Rudimentary Introduction to Generator and Yield in Python

Computer Science

  • Type Theory, A Very Rudimentary Introduction
    • Subtypes
    • Type Safety
    • Subsumption
    • Generics and Type Variables
    • Bound and Constraint in Generics and Type Variables
    • Invariance, Covariance and Contravariance
    • Function Overloading
    • Sentinel Types

Data Structures and Algorithms

  • Complexity Analysis
    • Master Theorem
  • List/Array
    • Concept
    • Questions
      • Two Sum
  • Hash Map
    • Concept
    • Questions
      • Two Sum
      • Group Anagrams
  • Two Pointers And Sliding Window
    • Two Pointers
    • Sliding Window
    • Questions
      • Two Pointers
        • Remove Duplicates from Sorted Array
        • Two Sum II - Input Array Is Sorted
      • Sliding Window
        • Find All Anagrams in a String
  • Stack
    • Concept
    • Questions
      • Valid Parentheses
      • Min Stack
      • Implement Queue using Stacks
      • Reverse String
  • Queue
    • Concept
    • Double Ended Queue
      • Easy - Hot Potatoes
      • Palindrome Checker
  • Linear Search
    • Concept
  • Binary Search
    • Concept
    • Koko Eating Bananas

Linear Algebra

  • Preliminaries
    • Fields
    • Systems of Linear Equations
  • Vectors
    • Vector and Its Definition
    • Vector and Its Operations
    • Vector Norm and Distance
    • A First Look at Vector Products

References, Resources and Roadmap

  • Bibliography
  • IEEE (Style) Citations
  • Repository
  • Open issue

Index

By Gao Hongnan

© Copyright 2024.