Kili Technology is designed to facilitate quality control, while scaling up.
This is done through :
Workload distribution, to ensure a dynamic distribution of the workload to labeling team, and prevent double work.
Instructions, allowing you to illustrate the annotation task with concrete examples. These instructions are made available in the annotation interface, so that annotators can easily refer to them while labelling.
Consensus, which allows you to decide to have all or part of the dataset annotated several times, in order to measure a consensus score between the annotators. Consensus is to be set at the beginning of a project. If it is updated later, the KPIs will be updated but only the remaining assets (with status
TODO) will be distributed.
Honeypot, or Golden Standard, which helps to set a ground truth and compare it to annotations in order to audit the annotators' work quality throughout the annotation process.
These indicators are calculated to allow you to monitor the quality of labels created, to quickly prevent errors and to be able to iterate with annotators helping them to produce a better dataset.