Review process

Kili Technology allows annotation workflow management, and when required, a fine-grained data validation process to analyze and validate the quality of production.

This structured process facilitates collaboration within the team and helps to produce better quality annotated data.

The combination of quality scores, which makes it possible to filter quickly on the labels that raise questions, and the process of review, are key to producing better annotations.

The review Interface

The review interface is accessed from the Audit Labelers tab, by clicking on Start Reviewing.

The interface, similar to the annotation interface, allows you to review annotated assets, correct labels if necessary, and validate the labels of an asset.

Agreement

When assets are labeled with consensus, or after predictions are uploaded, a score of agreement is computed for classification tasks. It is a percentage of agreement computed with the intersection of all the classes selected over the union of all the classes. In the example below, the labeler 1 selected only the English language, and the labeler 2 French and English, leading to an agreement of 0% and 50% a the classes level. For the second task, that is nested, it is the same principle : here, everything is similar, we have a 100% agreement.

Search labels & assets

To search and filter labels before reviewing, you can play with the panel size to get a better view on the list. To combine filters, just add a space between two fields. For instance status:labeled consensus<80

• Consensus = consensus<90 // consensus>90
• Creation Date = creationDate>25/11/2019 // creationDate<25/11/2019
• Duration = filter not yet available in search
• External ID = externalId:XXX
• Honeypot = honeypot<10 // honeypot>10
• JSON response (ie. the content of the label) = contains: Object A
• Label Honeypot = labelHoneypot>40// labelHoneypot<10
• Labeled by = labeledBy:XXX
• Presence of open/solved issues on the asset: issue:open // issue:solved
• Priority = filter not yet available in search
• Skipped = skipped:true / false
• Status = status:todo / ongoing / labeled / reviewed
• Type = type:prediction / default / review

Navigation shortcuts

You can easily navigate between labels and assets with arrow keyboards. Jump from one asset to the other with the up and down arrows : you will see the latest label each time. You can view the full history of labels with the right arrow, navigate through the list, and collapse it with the left arrow.

Send back to labeling

You can send an asset back to the labeler by clicking on the issue button.

Then you can click on Send back to queue.

This makes the asset have the top priority and be assigned to the labeler, effectively making the asset the next one to be labeled by the labeler.