Inference with Torch-TensorRT Deep Learning Prediction for Beginners - CPU vs CUDA vs TensorRT

This beginner-friendly tutorial demonstrates deep learning inference with a pretrained ResNet50 image classifier in PyTorch. It sets up Torch-TensorRT in Docker, converts a traced model, and compares prediction speed on CPU, CUDA, and TensorRT.

Syllabus

- intro
- clone Torch-TensorRT
- install and setup Docker
- install Nvidia Container Toolkit & Nvidia Docker 2
- Torch-TensorRT container option #1
- Torch-TensorRT Nvidia NGC container option #2
- import Pytorch
- load ResNet50
- load sample image
- sample image transforms
- batch size
- prediction with ResNet50
- softmax function
- ImageNet class number to name mapping
- predict top 5 classes of sample image topk
- speed test benchmark function
- CPU benchmarks
- CUDA benchmarks
- trace model
- convert traced model into a Torch-TensorRT model
- TensorRT benchmarks
- download Jupyter Notebook
- HOW DID I MISS THIS???
- thanks for watching!