quantizing-models-bitsandbytes

SKILL.md

bitsandbytes reduces LLM memory by 50% (8-bit) or 75% (4-bit) with <1% accuracy loss.

| 8 GB | 3B | 4-bit | | 12 GB | 7B | 4-bit | | 16 GB | 7B | 8-bit or 4-bit | | 24 GB | 13B | 8-bit or 70B 4-bit | | 40+ GB | 70B | 8-bit |

QLoRA training guide: See references/qlora-training.md for complete fine-tuning workflows, hyperparameter tuning, and multi-GPU training.

Quantizes LLMs to 8-bit or 4-bit for 50-75% memory reduction with minimal accuracy loss. Use when GPU memory is limited, need to fit larger models, or want faster inference. Supports INT8, NF4, FP4 formats, QLoRA training, and 8-bit optimizers. Works with HuggingFace Transformers. Source: orchestra-research/ai-research-skills.

View raw