TinyML

Projects

AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

MLSys 2024

(

)

Low-bit weight-only quantization for LLMs.

Tiny Machine Learning Projects

NeurIPS 2020/2021/2022, MICRO 2023, ICML 2023, MLSys 2024, IEEE CAS Magazine 2023

(

Feature

)

This TinyML project aims to enable efficient AI computing on the edge by innovating model compression techniques as well as high-performance system design.

TorchSparse++: Efficient Training and Inference Framework for Sparse Convolution on GPUs

MICRO 2023

(

)

TorchSparse++ is a high-performance computing library for efficient 3D sparse convolution. It offers significant performance improvement over TorchSparse++ by overlapping computation with memory access. It also searches for the best execution strategy for sparse workloads within a large design space through auto-tuning.

PockEngine: Sparse and Efficient Fine-tuning in a Pocket

MICRO 2023

(

)

This project introduce PockEngine: a tiny, sparse and efficient engine to enable fine-tuning on various edge devices. PockEngine supports sparse backpropagation: it prunes the backward graph and sparsely updates the model with measured memory saving and latency reduction while maintaining the model quality.

FlatFormer: Flattened Window Attention for Efficient Point Cloud Transformer

CVPR 2023

(

)

We present FlatFormer, an efficient ViT architecture for large-scale point cloud analysis.

SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformer

CVPR 2023

(

)

Vision transformer on high-resolution images can learn richer visual representation. However, the improved performance comes at the cost of huge computation complexity. Thus, we present SparseViT, which accelerates high-resolution visual processing by skipping less important regions during computation.

Blog Posts

On-Device Training Under 256KB Memory

November 28, 2022

In MCUNetV3, we enable on-device training under 256KB SRAM and 1MB Flash, using less than 1/1000 memory of PyTorch while matching the accuracy on the visual wake words application. It enables the model to adapt to newly collected sensor data and users can enjoy customized services without uploading the data to the cloud thus protecting privacy.

Reducing the carbon footprint of AI using the Once-for-All network

July 3, 2020

“The aim is smaller, greener neural networks,” says Song Han, an assistant professor in the Department of Electrical Engineering and Computer Science. “Searching efficient neural network architectures has until now had a huge carbon footprint. But we reduced that footprint by orders of magnitude with these new methods.”

Auto Hardware-Aware Neural Network Specialization on ImageNet in Minutes

July 2, 2020

This tutorial introduces how to use the Once-for-All (OFA) Network to get specialized ImageNet models for the target hardware in minutes with only your laptop.

Efficiently Understanding Videos, Point Cloud and Natural Language on NVIDIA Jetson Xavier NX

May 22, 2020

Thanks to NVIDIA’s amazing deep learning eco-system, we are able to deploy three applications on Jetson Xavier NX soon after we receive the kit, including efficient video understanding with Temporal Shift Module (TSM, ICCV’19), efficient 3D deep learning with Point-Voxel CNN (PVCNN, NeurIPS’19), and efficient machine translation with hardware-aware transformer (HAT, ACL’20).