Together AI Sets New Benchmark with Fastest Inference for Open-Source Models

Together AI has announced a groundbreaking achievement in the realm of open-source model inference, delivering up to twice the speed compared to previous benchmarks. This leap in performance is attributed to advancements in GPU optimization, speculative decoding, and low-bit quantization formats, according to Together AI.

Technological Innovations Driving Performance

Central to this achievement is the integration of next-generation GPU hardware, notably the NVIDIA Blackwell architecture. Together AI has re-engineered its inference engine to maximize the potential of these GPUs, employing optimized kernels and advanced quantization techniques such as FP4. This comprehensive overhaul allows the system to function as a high-efficiency unit, optimizing compute kernels, memory layout, and execution graphs.

Quantization and Speculative Decoding

Together AI’s quantization strategy plays a crucial role in its performance gains. By converting large model weights to low-bit formats, the company maintains high accuracy while significantly enhancing speed. Their speculative decoding algorithms further boost efficiency, ensuring high output speed while maintaining quality across various data domains.

Benchmark Results

Independent benchmarks from Artificial Analysis confirm Together AI’s platform as the fastest among GPU-based providers for demanding open-source models, including GPT-OSS and Qwen series. The platform’s output speed surpasses competitors, with some models achieving up to 2.75 times faster inference.

Future Developments

Looking ahead, Together AI is focused on expanding its capabilities, including faster generation for downstream applications and enhanced support for hybrid quantization. The company is committed to advancing the performance and scalability of open-source AI models.

For more information, you can visit the Together AI website.

Image source: Shutterstock