Kili Technology manages a queue for each project, which can be customized and redefined all along a project. This is helpful for advanced annotation processes: business expectations, priorities, active learning, production supervision, real time annotation.
3 strategies are possible:
- First In First Out
- Assigning assets to specific labelers
- Asset prioritization
First In First Out (FiFo)
This is the default queue orchestration. Assets are loaded with a time stamp (creation date). The first assets will be distributed to be annotated first.
Assign assets to identified labelers
The queue can be customised in order to assign a specific list of assets to specific labelers. Assignation can be done for the whole queue or a specific subset of the complete dataset
toBeLabeledBy. See here on Kili Playground.
To unset the attribution, call
update_properties_in_asset with an empty list (
The 3rd approach allows to set a specific priority at the asset level. You can either set a priority to each asset or set a priority to a limited number of assets within the dataset. The higher the number the higher the priority is. If 2 assets have the same priority, Fifo will apply.
In other terms:
- If asset 1 has priority N_1 and asset 2 has priority N_2, and N_1 > N_2, then asset 1 has a higher priority than asset 2 and will be distributed before.
- If N_1 = N_2, priority follows the default strategy (FiFo).
What can you expect ?
Thanks to active learning, you can expect a reduction in the number of samples to label to reach the same performance by up to 50%. This will depend on the dataset and the task of course. For a demo use case of medical image classification (see this link on Kili Active Learning), we experienced an increase from 78% to 85% of the accuracy with the same number of samples, or a 30% reduction in the number of samples needed to reach 77% accuracy.