Kili Technology allows annotation workflow management, and when required, a fine-grained data validation process to analyze and validate the quality of production.
This structured process facilitates collaboration within the team and helps to produce better quality annotated data.
The combination of quality scores, which makes it possible to filter quickly on the labels that raise questions, and the process of review, are key to producing better annotations.
The review interfaces
The review interfaces are accessed from the Audit Labelers tab.
There are two possible types of review:
- The pull review can be accessed by clicking on "Explore". It allows you to explore the annotated assets, select them, update the labels and possibly send them back to the queue.
- The push review can be accessed by clicking on "Start Reviewing". In that case, you are pushed assets to review from the review queue (see below).
In both cases, the interface, similar to the annotation interface, allows you to review annotated assets, correct labels if necessary, and validate the labels of an asset.
The review queue
There are two ways to push assets to the review queue.
Randomly select assets to be reviewed
You can select a random portion of assets to be reviewed along the project. A quality parameter in Settings > Quality management defines the percentage of assets that will be randomly sent to review. These randomly chosen assets will automatically fuel the review queue.
Manually select assets to be reviewed
You can also filter on assets and manually send them to the review queue. This can be done in Dataset by going to the thumbnail view and manually picking the assets.
When clicking on a thumbnail, you select one asset. When clicking on a thumbnail and maintaining shift, you select a range of assets. You can send them to the review queue by clicking the button Add review.
When assets are labeled with consensus, or after predictions are uploaded, a score of agreement is computed for classification tasks. It is a percentage of agreement computed with the intersection of all the classes selected over the union of all the classes. In the example below, the labeler 1 selected only the English language, and the labeler 2 French and English, leading to an agreement of 0% and 50% a the classes level. For the second task, that is nested, it is the same principle : here, everything is similar, we have a 100% agreement.
Search labels & assets
To search and filter labels before reviewing, you can play with the panel size to get a better view on the list.
To combine filters, just add a space between two fields. For instance
- Consensus =
- Creation Date =
- Duration = filter not yet available in search
- External ID =
- Honeypot =
- JSON response (ie. the content of the label) =
contains: Object A
- Label Honeypot =
- Labeled by =
- Presence of open/solved issues on the asset:
- Priority = filter not yet available in search
- Skipped =
- Status =
- Type =
You can easily navigate between labels and assets with arrow keyboards. Jump from one asset to the other with the up and down arrows : you will see the latest label each time. You can view the full history of labels with the right arrow, navigate through the list, and collapse it with the left arrow.
Send back to labeling
You can send an asset back to the labeler by clicking on the issue button.
Then you can click on
Send back to queue.
This makes the asset have the top priority and be assigned to the labeler, effectively making the asset the next one to be labeled by the labeler.