The foundations of function: How Global Job Pools productionize subnets
SN25 has recently upgraded to Global Job Pools - so how do they work, and what do they show about the way back-end subnet structures can support front-end product?
Building a functional subnet is hard. Competition is woven into Bittensor’s DNA: each activity is a fight to outperform everybody else. Yet functionality is just one side of the equation- to be successful in this ecosystem you need to be profitable, most likely by offering a product or service. Whatever you develop, it must be consistent, effective, and reliable - adding a whole new dimension of difficulty.
Not only must a subnet be proficient at its goal, but it needs a market-fit that works on Bittensor and beyond. dTAO increases the exposure of subnets market dynamics, increasing the urgency even further.
As owners of five subnets, we’ve learned a lot about the relationship between form, function, and product, and how improving the structure of subnet operations can lay the foundations for the consistent results necessary to build products. Global job pools are a prime example. These have now been implemented on SN25, Protein-Folding, but the concept is applicable to a range of subnets - and by understanding their underlying principles, subnets can pave a path towards products of their own.
What is a global job pool?
A global job pool is a system where validators can submit specific tasks to a subnet, and miners can actively choose which of these look ideal. Rather than the subnet overall thrusting tasks upon miners without giving them much choice, global job pools add more autonomy, handing miners greater control over their performance.
Jobs can have different rewards, depending on how hard they’re perceived to be, or how much the validator wants it completed. Miners can assess the requirements before starting, to see if it suits their computational setups. Not only does it harmonize these two groups, but it also allows users to optimize their computing to tackle specific tasks, which utilizes more of the miner community.
How do global job pools work?
No two job pools are the same, as every subnet has its own unique features, but they do have a general structure. For SN25, our job pool pipeline works like this:
A job task is created (and its files uploaded to Digital Ocean S3, a cloud-based storage service)
The validator receives those S3 file links and places them in the Global Job Pool through our own API.
The information gets then gets submitted via RQLite, which is a lightweight version of SQLite.
The job and its information becomes accessible to miners immediately.
The job ID is sent to the miner working on that job so they can find it in the pool.
The miners use that information from the pool to download the files and start working.
After a certain period, which is set as a parameter in the job, the subnet pings the miner to check if they’ve completed the work. If they have, it gets moved to the validator to evaluate it.
Incentivizing work via organic queries
One of the biggest challenges subnets face is building a system where tasks from real people co-exist with synthetic tasks. Introducing organic queries can prove troublesome, as you need to find a way for them to get added among a sea of synthetics, ensure the tasks are within the scope of the subnet, and find a way for them to stand out via incentivisation.
This is what global job pools aim to fix. They catalog all the tasks available, including synthetics, meaning organic queries can slip into the pool and be seen by miners. They also allow validators to set a bounty on them, making them more enticing for miners, and increasing the competitiveness of the subnet. In turn, this should boost performance, which can help subnets turn their output into product.
Productization on Bittensor
Building products that rest on top of subnets is hard. That’s why only a handful of projects are currently able to do so. While most subnets have products coming up ahead their roadmaps, the reality is that reliably incentivising the right behavior is a significant challenge.
Global job pools help resolve this. They create a streamlined and systematized process for building new revenue streams that connect to preexisting incentives within a subnet. We’re currently implementing this global job pool design with SN25, and SN13, Data-Universe runs an adjacent system for its dynamic desirability/gravity features. We’re now investigating how this could work for SN9, Pre-Training, and SN37, Fine-Tuning. So that global job pools continue to support product launches across Macrocosmos - and ultimately across Bittensor.
If you’re interested in contributing to innovation Bittensor, join our Telegram and Discord communities and have your say.