Subnet 9: Scaling up parameters to further Bittensor’s founding mission

There’s a growing awareness that we must democratize access to trained AI models. But as well as access, we must also democratize the training process itself.

by Dr. Steffen Cruz and Will Squires

Generative AI is changing the world. As it becomes increasingly integrated across business and society, there’s a growing unease about the monopolistic control of these entities - after all, the firms that govern access to the best state-of-the-art AI wield unprecedented power. Left unchecked, these firms could become increasingly misaligned with the common good, further entrenching inequalities within society.

In the face of the growing power of AI, there’s a growing awareness that we must democratize access to trained models. But as well as access, we must also democratize the training process itself. Closed-source AI, developed in private and provided as a service behind gated APIs, are trained on massive amounts of web data - often including our own personal information, which is sold back to us. The lack of transparency around both process and product makes it impossible to trust or understand the outputs and the people selling it. 

Deploying models which cannot be explained, inspected, or audited is ultimately irresponsible. That’s why Bittensor is committed to decentralizing AI. Our ability to harness at-scale computational power via a large distributed network puts us in a unique position to open access to AI and training models. 

Pre-training, decentralized

The pre-training subnet (SN9), is at the heart of Bittensor’s vision. Pre-training is the crucial first step in the creation of all modern AI models, where enormous amounts of data and computational power are consumed by the model in order for it to begin to understand the world. It is usually the most expensive step, too, often costing millions of dollars (if not tens of millions) for large models. Moreover, the quality of model pre-training dramatically shapes both the performance and the security of downstream applications.

Bittensor allows a decentralized network of participants to bring a new scale of computational resources together to compete with these centralized entities. Just as the price of bitcoin is often seen to be bounded by its root production cost, the resource cost to train competitive models and provide comprehensive digital commodities will begin to correlate with the price of TAO as the capabilities of Bittensor are extended. On Subnet 1, we are seeing the vision of LLM as judge exponded in papers such as (LLM as judge) born out at scale. As the volume of synthetic data generated by Bittensor increases, and additional subnets such as SN13 and SN22 begin to scrape and bring traceable, transparent data into the Bittensor ecosystem, the path to a fully contained AI ecosystem begins to become clearer. 

Today on SN9, the pre-training process is distributed between highly skilled teams which compete to continuously improve models and get paid to do so. As a result SN9 produces world class large language models (LLMs) that are truly open: open source, open data, open models, open competition. Open everything. Its track record is outstanding; it has beaten the biggest companies in the world at their own game by creating models that are pound-for-pound better. Expanding this process to support truly decentralized training is a key step, but a drive towards state of the art training is critical to demonstrate Bittensor’s utility as the heart of the decentralized AI ecosystem. 

The time has come for us to scale up SN9 and train a new generation of world class open models. Today, we’re beginning the next stage of our journey and scaling up our LLMs to 7B (billion) parameters, which is a 10x increase. We’re also reconfiguring the model design to incorporate recent advances in the field, which will further improve model quality. Reaching 7 billion is critical: it unlocks many model capabilities which are inaccessible at lower parameters.

SN9 ensures that participants are rewarded equitably for their input to the model training process, while contributing to the democratization of AI. That’s how we’re creating the future of AI: together in the open, not alone behind closed doors.