More oversight, better visibility: The relaunched SN9 Leaderboards

We're expanding our reporting and benchmarking for Subnet 9, bringing more granular data to the Bittensor community.

We’re speeding up our dashboard development at Macrocosmos. After receiving feedback from the Bittensor community, we’ve relaunched the leaderboard for Subnet 9 (SN9) on Hugging Face.

Since launching SN9, we’ve heard from miners and validators that they need greater visibility and transparency over their performance and what they stand to gain by building the best performing model on the subnet. From our perspective at Macrocosmos, we want to be feeding back information to the community that captures how the network is improving over time. We’ve relaunched this first dashboard to meet that need. 

SN9 is our pre-training subnet, designed to incentivize the development of foundation models within the Bittensor ecosystem - which can then be fine-tuned for specific tasks on other subnets. SN9 rewards miners for pretraining models against the Falcon Refined Web dataset, which we implemented last month. As miners will know, SN9 operates a winner-takes-all compensation mechanism which accrues to the top-performing model on the subnet, another reason we want to openly share how models are performing on the subnet.

At launch we have five components to the dashboard: 

1. Model improvement. Captures loss rates over time on SN9 - our measure of accuracy for models on the subnet. See screenshot above.

2. Earnings. The number of Tao that have currently accrued to the best performing miner on the subnet. 

3. Leaderboard. All recent submissions from miners, broken down into how models are performing across every step involved in improving the network.

4. Benchmarks. How the top-performing model on SN9 compares to state-of-the-art models across number of parameters and performance on the Wikitext103 and Falcon Refined Web datasets.

5. Validator board.  Shows how robust the validator pool is, with validator UIDs compared across their staked Tao and vtrust scores.

Why we’re relaunching subnet dashboards 

Our goal at Macrocosmos is to be as transparent as possible with the community about the performance of each subnet, and how their performance compares to other players in the ecosystem. 

We’re not just looking to design the right incentives for each subnet we manage. We also want to design the right tools and data sources to encourage competition on the subnet. It’s another way that we can bring openness to building and training AI models: open source, open competition, open reporting.

What does the future hold?

The SN9 Leaderboard is a first step. Next, we want to expand the reporting we’re delivering across all the subnets we manage at Macrocosmos, as well as deepen our subnet level reporting in response to the community needs. We’ve also upgraded the datasets for SN9 to the FineWeb dataset to drive further improvements in the subnet’s performance, which will all flow through to our dashboards and reporting for the subnet.

So what additional data would you like to see? Let us know in the comments, or on Discord.