We propose COAT, a memory efficient FP8 training method for large language models.

LongVILA is a full-stack solution for long VLM, which incorporates novel training strategies, dataset, and the Multi-Modal Sequence Parallelism (MM-SP) system to efficiently handle long video understanding, achieving significant scalability, accuracy, and speed improvements on multi-modal benchmarks.

LongLoRA takes advantage of shifted sparse attention to greatly reduce the finetuning cost of long context LLMs.

This TinyML project aims to enable efficient AI computing on the edge by innovating model compression techniques as well as high-performance system design.

We discuss the definition, challenges, and applications of TinyML.

This project introduce PockEngine: a tiny, sparse and efficient engine to enable fine-tuning on various edge devices. PockEngine supports sparse backpropagation: it prunes the backward graph and sparsely updates the model with measured memory saving and latency reduction while maintaining the model quality.