Research topics in ML, AI and Deep Learning

Geometric Deep Learning

Euclidean Geometry

Group Invariance Theorem

Universal Approximation

Player Perceptron, Rosenblatt perceptron

Curse of Dimensionality > neocognitron

LeNet-5

Graph Neural Network, Chemical Precursors, Weisfeiler-Lehman (WL) test

DeepSets

CNN > Symmetry translation, GNN > Symmetry Permutation

CNNs are translation-equivariant; GNN layers are permutation-equivariant, with permutation-invariant graph readouts. https://arxiv.org/abs/2104.13478

5G of GDL - https://arxiv.org/abs/2104.13478

1. Grids
2. Groups
3. Graphs
4. Geodesics
5. Gauges

Supervised ML sources of errors > approximation, estimation, optimization

minimization of empirical risk

Error Decomposition

Total excess error = approximation error + estimation error + optimization error

Double Descent

Nearest Neighbor Classifier

Approximation rates

Soboler Class dimensionality cursed

curse in optimization

benign landscapes

Lipschitz estimation error > too large

Sobolev - approx erro > too small

Graph Foundation Models

knowledge graphs TransE

Message Passing Neural Network

Link Prediction with GNN on Knowledge Graphs

Transferability on Knowledge Graphs

Transformers Research Gaps > Feasible with Less Compute

Quadratic Complexity > LoRA, KV Cache Optimization

Quadratic attention cost > sparse/local/linear attention or FlashAttention; LoRA is for parameter-efficient fine-tuning, while KV-cache optimization targets inference memory. https://arxiv.org/abs/2209.04881

Fixed Context window > Flash Attention, Sliding Window, RAG

Context limits > long-context training, positional scaling and sliding-window attention; RAG supplies external context, while FlashAttention mainly improves efficiency. https://arxiv.org/abs/2205.14135

Lack of Inductive bias > Annotations

Lack of inductive bias > architectural priors such as locality, equivariance, recurrence or structured positional encoding; annotations add supervision. https://arxiv.org/abs/2104.13478

Struggles with Maths

Stochastic Diffusion Models, Deterministic Flow Matching

Diffusion often uses stochastic dynamics, while flow matching usually learns a deterministic ODE vector field; the frameworks can overlap. https://arxiv.org/abs/2210.02747

Applied

• Stochastic
• Differential
• Equations

Recipes for ML success

1. Large and diverse dataset
2. clear performance criterion
3. degenerate solution space
   1. Suitable hypothesis class and tractable optimization

Dataset void training areas:

1. simple arithmetic
2. sort listing
3. thermostat regulation
4. moral reasoning

Elmo RNN of Language Modeling

ELMo: contextual embeddings derived from a deep bidirectional LSTM language model. https://arxiv.org/abs/1802.05365

Limitations of RNN

RNN trained with back propagation through time

unroll computation graph to standard back prop for each

Slow and difficult to parallelize over long sequences.

Transformers

Self-attention uses queries, keys and values: Q, K and V.

Residual Connection, LayerNorm

Positional embeddings in addition to word embeddings

Casual LM forward probability:

• p(x_t | x_<t), predicting token t from preceding tokens.

Transformer decoder

Data: trillions of tokens

Compute: massive parallel training

Outlines Guidance, n grammar:

• Outlines and Guidance support constrained generation using JSON Schema, regular expressions and context-free grammars. https://dottxt-ai.github.io/outlines/1.0.0/features/core/output_types/

Repetition penalty, grammar matching,

instruction tuning SFT, ROUGE-Lsum for summarization

Encoded in params, tokenization meets instruction tuning

• Knowledge is partly encoded in model parameters; tokenization and instruction tuning are separate components.

Reward model training, R (prompt, response option) > score

triplet of prompt, winning, losing (prompt, Ow, Oc) train R such that R (prompt, Ow) > R (prompt, Oc)

Reinforcement learning from Human Feedback

good to maximize reward with staying close to original model

PPO updates - Proximal Policy Optimization, DPO - Direct preference optimization

RL via PPO, RL with Verifiable Rewards (RLVR)

Efficiency Techniques, memory and compute

KV Cache Optimization > Inference

Memory: quantization, 4-bit quantization > 4x memory reduction

• 4-bit weights are theoretically 4x smaller than FP16 weights, but total runtime savings are lower because of metadata, activations and caches. https://arxiv.org/abs/2305.14314

Parameter-efficient fine-tuning: LoRA. https://arxiv.org/abs/2106.09685

vLLM, multimodal: vision + language, LLaVA

spiking NN, state space models,

Model distillation, long context techniques RoPE ALiBi

• Long-context positional methods: RoPE and ALiBi; extending the trained window may still require interpolation, scaling or additional training. https://arxiv.org/abs/2310.13017

Embedding enable generalization

LLM Foundations

ABC scaling, abc-parameterization in μP

Emergent Ability

MMLU tasks few shot prompting:

• MMLU is a 57-subject evaluation benchmark, commonly tested in zero-shot or few-shot settings. https://arxiv.org/pdf/2009.03300

chain of thought prompting Elicit Reasoning in LLM

Reasoning, Question < intermediate steps > answer

Inference time scaling

Agentic AI Recall, LLM augmented system with MRT: Memory Retrieval Tools

RAG, Data ingestion:

• relational deep learning
• Graph MC models
• RDL architectures

User churn model

Graph ML Problem Solving Pipeline

Relational DB > Relational Graph > Graph Problem > Graph ML

RelBench Tasks

Heterogeneous GNN, Temporal GNN, architectures of RDL models

GNN vs Feature Engineering

Relational IGT improvement

KumoRFM and Griffin

• KumoRFM and Griffin are separate relational foundation models; Griffin is explicitly graph-centric. https://arxiv.org/abs/2604.12596

AI Ready Farm

Graph Transformers: ViT/ MLP-Mixer

• Graphormer and Graph ViT/MLP-Mixer; plain ViT and MLP-Mixer originate as vision architectures. https://arxiv.org/abs/2106.05234

Graph positional encoding, Spectral GNN

GCN: most popular technique of GNN

LLM with GNN

LLM Reasoning, LLM sequence to vector, G-retriever, Graph tokens

Graph RAG:

• helps LLM hallucinations
• proposed approach to LLM
• context windows limited

LLM + GNN + Graph RAG = Superior Performance

• LLM + GNN + Graph RAG can improve graph-centric tasks, subject to dataset-specific evaluation against strong baselines. https://arxiv.org/abs/2402.07630

Represent graphs as text

model decay

Medical AI

model centric to system centric AI

data privacy, low end devices, edge computing edge AI

success metric, improve health system performance not algorithmic novelty

Applied use cases in Global Health AI

Leltek, BMGF Apps

AI assisted obstetric ultrasound, Uganda

Hallucinations in LVLMs, Data model uncertainty