Review process · Kili Docs

Kili Technology allows annotation workflow management, and when required, a fine-grained data validation process to analyze and validate the quality of production.

This structured process facilitates collaboration within the team and helps to produce better quality annotated data.

The combination of quality scores, which makes it possible to filter quickly on the labels that raise questions, and the process of review, are key to producing better annotations.

The review interfaces

The review interfaces are accessed from the Audit Labelers tab.

There are two possible types of review:

The pull review can be accessed by clicking on "Explore". It allows you to explore the annotated assets, select them, update the labels and possibly send them back to the queue.
The push review can be accessed by clicking on "Start Reviewing". In that case, you are pushed assets to review from the review queue (see below).

In both cases, the interface, similar to the annotation interface, allows you to review annotated assets, correct labels if necessary, and validate the labels of an asset.

The review queue

There are two ways to push assets to the review queue.

Randomly select assets to be reviewed

You can select a random portion of assets to be reviewed along the project. A quality parameter in Settings > Quality management defines the percentage of assets that will be randomly sent to review. These randomly chosen assets will automatically fuel the review queue.

Manually select assets to be reviewed

You can also filter on assets and manually send them to the review queue. This can be done in Dataset by going to the thumbnail view and manually picking the assets.

When clicking on a thumbnail, you select one asset. When clicking on a thumbnail and maintaining shift, you select a range of assets. You can send them to the review queue by clicking the button Add review.

Agreement

When assets are labeled with consensus, or after predictions are uploaded, a score of agreement is computed for classification tasks. It is a percentage of agreement computed with the intersection of all the classes selected over the union of all the classes. In the example below, the labeler 1 selected only the English language, and the labeler 2 French and English, leading to an agreement of 0% and 50% a the classes level. For the second task, that is nested, it is the same principle : here, everything is similar, we have a 100% agreement.

Search labels & assets

To search and filter labels before reviewing, you can play with the panel size to get a better view on the list. To combine filters, just add a space between two fields. For instance status:labeled consensus<80

Consensus = consensus<90 // consensus>90
Creation Date = creationDate>25/11/2019 // creationDate<25/11/2019
Duration = filter not yet available in search
External ID = externalId:XXX
Honeypot = honeypot<10 // honeypot>10
JSON response (ie. the content of the label) = contains: Object A
Label Honeypot = labelHoneypot>40// labelHoneypot<10
Labeled by = labeledBy:XXX
Presence of open/solved issues on the asset: issue:open // issue:solved
Priority = filter not yet available in search
Skipped = skipped:true / false
Status = status:todo / ongoing / labeled / reviewed
Type = type:prediction / default / review

Navigation shortcuts

You can easily navigate between labels and assets with arrow keyboards. Jump from one asset to the other with the up and down arrows : you will see the latest label each time. You can view the full history of labels with the right arrow, navigate through the list, and collapse it with the left arrow.

Send back to labeling

You can send an asset back to the labeler by clicking on the issue button.

Then you can click on Send back to queue.

This makes the asset have the top priority and be assigned to the labeler, effectively making the asset the next one to be labeled by the labeler.