SN1’s validator upgrade: How integrating Llama’s 70bn parameters improves performance
Subnet 1's adoption of Llama 3's 70bn model reveals the challenges of harnessing the latest technology - and the significant benefits that successful integration yields.
By Dr. Steffen Cruz and Will Squires
Broadly defined, Subnet 1 is a benchmarking system for measuring the intelligence of LLMs. Yet intelligence itself is both contextually-dependent, and constantly-evolving. While there is no single measure of intelligence, there are a number of metrics measuring the power, capability, and usability of the underlying models which generate intelligence.
Now, SN1’s validators have been upgraded to incorporate the Llama 3 70B model, significantly enhancing their overall capability. It extends SN1’s validation capabilities, enhancing the challenge generation by enabling more complex and varied tasks - sharpening our ability to refine our selectiveness and extract improvements from miners.
It also expands the perimeter of agent-based task creation and performance, unlocking new potential for miners capable of identifying and exploiting them.
Leveraging the latest Llama
Recently released by Meta, integrating the state-of-the-art Llama 3 70B model into the validators demonstrates both the challenge of harnessing the latest models and the flexibility of SN1’s design to seamlessly incorporate them. While Bittensor’s design enables the adoption of the latest open source technology, doing so seamlessly demands technical expertise and skillful implementation.
The fundamental challenge was to align miners’ incentives with the new model, encouraging them to enhance their intelligence to Llama 3’s 70bn standard. To ensure best fit with our existing system, we tested multiple state-of-the-art models, including Solar 10.7B full-precision, Llama3 70B AWQ (4bit), Llama3 70B full-precision, and Llama3 8B AWQ (4bit). We sought the best trade-off between quality and throughput cost - delivered by Llama3 70B AWQ’s quantized model.
Intelligence with impact
Already, the improvements are clear. Across query, challenge, and reference tasks, throughput increases by an average of 5.31 tokens per second, or 15.61%. This indicates that Llama3’s 70bn parameters are widening the number of predictions a model can make in a given time, compounding into significant efficiencies across the subnet. Moreover, the new model has improved our quality evaluation score by 12% - reflecting higher quality responses and enhanced capability for complex tasks.
In a fast-moving competitive landscape, technical improvements must yield advantages for miners, users, and validators alike. SN1’s upgrade unlocks efficiencies and benefits across the network. By increasing the overall intelligence of the network, the upgrade facilitates the development of more sophisticated AI tasks and solutions, leading to higher quality output.
It opens possibilities for new and innovative use cases, and miners who can leverage the advanced capabilities of Llama 3 70B stand to benefit. Overall, it enhances the scalability of shared intelligence across the network, enabling SN1 to remain competitive.