ncnn

ncnn is a neural network inference framework optimized for mobile, embedded, and desktop deployment.

ncnn is a high-performance neural network inference framework optimized for mobile, embedded, and desktop deployment. It has no third-party runtime dependencies and runs across CPU and Vulkan GPU backends. It provides tools such as pnnx for converting PyTorch and ONNX models to ncnn. Developers can deploy deep learning models efficiently on phones, PCs, browsers, and edge devices. It supports classification, detection, segmentation, pose, and OCR workloads.

ncnn at a glance

Best for: Software engineer
Runs on: Linux, Windows, macOS, iOS, Android, Web, open source
Pricing: Free
Released: June 2017
Category: AI coding tool
Website: github.com/Tencent/ncnn

Key features

No third-party runtime dependencies
Pure C++ implementation with C API and Python binding
Optimized CPU inference for mobile and embedded processors
Vulkan GPU acceleration for supported platforms
Low memory footprint with explicit blob allocator design
PyTorch and ONNX model conversion through pnnx
Supports fp16 storage and int8 quantized inference

Use it for

Deploying image classification models on mobile devices
Running object detection models on edge hardware
Converting PyTorch and ONNX models for efficient inference
Executing speech recognition on embedded devices

Alternatives to ncnn

MNN — It delivers high-performance inference and training with a lightweight footprint.
TensorRT — Speeds up inference by 36X compared to CPU-only platforms.
fal — Access 1,000+ generative models and high-performance serverless GPUs.
BitNet — Runs 1-bit models on CPUs and GPUs with significantly reduced energy consumption.
ort — Provides an ergonomic Rust API for running high-performance ONNX models.
Xinference — Supports a wide range of model types and hardware, including multi-node setups.
LMDeploy — Accelerates inference with high-performance engines and quantization tools.
llm-d — Deploys production-grade LLM serving on existing hardware clusters.
Strata — Runs large AI models locally on consumer gaming PCs without server reliance.
AirLLM — Runs massive 70B+ parameter models on small 4GB consumer GPUs.
Narwhal — It automatically hot-swaps prefill and decode roles without reloading model weights.
SpawnDev.ILGPU.ML — Runs in-browser neural network inference using 6 backends without servers

All ncnn alternatives