Kili Docs

Kili Docs

›Quality management

Introduction to Kili Technology

  • Introduction to Kili Technology
  • Kili Technology allows
  • Compatible browser

Getting Started

  • Getting started with Kili - Classification

Hosting

  • SaaS
  • On-Premise Data
  • On-Premise Entreprise

Concepts

  • Definitions
  • Status Lifecycle
  • Architecture

Users and roles

  • Roles by project
  • Users
  • Users and roles management

Projects

  • Audit labelers
  • Customize interface
  • Dataset
  • New project
  • Project overview
  • Projects
  • Projects list
  • Settings
  • Shortcuts

Image interfaces

  • Bounding Box
  • Classification
  • Point
  • Polygon
  • Polyline
  • Segmentation
  • Simple and intuitive interfaces

Text & PDF interfaces

  • Classification
  • Image transcription / OCR
  • Named entities recognition
  • Relations extraction

Video interfaces

  • Classification
  • Multi-frames classification
  • Multi-frames object detection
  • Transcription

Audio interfaces

  • Voice transcription / Speech to text

Data ingestion

  • Data ingestion made easy
  • Load data from a workstation
  • Load data from a public cloud
  • Data on premise or on private cloud
  • How to generate non-expiring signed URLs on AWS

Quality management

  • Consensus
  • Honeypot or Gold Standard
  • Instructions
  • Quality KPIs
  • Quality management
  • Questions and Issues
  • Review Process
  • Workload distribution

Automation

  • Human in the loop
  • Model based preannotation
  • Online learning
  • Queue prioritisation

Data export

  • Data export
  • Data format
  • Example

Python - GraphQL API

  • GraphQL API
  • Python API

Code snippets

  • Authentication
  • Create a Honeypot
  • Create a user
  • Creating Consensus
  • Delete the data
  • Export data
  • Export labels
  • Import data
  • Import labels
  • Prioritize assets
  • See the Consensus of an annotation
  • See the Honeypot of an annotation
  • Throttling

Recipes

  • AutoML for faster labeling with Kili Technology
  • Create a project
  • Exporting a training set
  • Importing medical data into a frame project
  • Importing assets
  • Import rich-text assets
  • Importing predictions
  • Reading and uploading dicom image data
  • How to query using the API
  • Labelled Image Data & Transfer Learning
  • Webhooks

Change log

  • Change log

Review process

Kili Technology allows annotation workflow management, and when required, a fine-grained data validation process to analyze and validate the quality of production.

This structured process facilitates collaboration within the team and helps to produce better quality annotated data.

The combination of quality scores, which makes it possible to filter quickly on the labels that raise questions, and the process of review, are key to producing better annotations.

The review Interface

The review interface is accessed from the Audit Labelers tab, by clicking on Start Reviewing.

The interface, similar to the annotation interface, allows you to review annotated assets, correct labels if necessary, and validate the labels of an asset.

Agreement

When assets are labeled with consensus, or after predictions are uploaded, a score of agreement is computed for classification tasks. It is a percentage of agreement computed with the intersection of all the classes selected over the union of all the classes. In the example below, the labeler 1 selected only the English language, and the labeler 2 French and English, leading to an agreement of 0% and 50% a the classes level. For the second task, that is nested, it is the same principle : here, everything is similar, we have a 100% agreement.

Search labels & assets

To search and filter labels before reviewing, you can play with the panel size to get a better view on the list. To combine filters, just add a space between two fields. For instance status:labeled consensus<80

  • Consensus = consensus<90 // consensus>90
  • Creation Date = creationDate>25/11/2019 // creationDate<25/11/2019
  • Duration = filter not yet available in search
  • External ID = externalId:XXX
  • Honeypot = honeypot<10 // honeypot>10
  • JSON response (ie. the content of the label) = contains: Object A
  • Label Honeypot = labelHoneypot>40// labelHoneypot<10
  • Labeled by = labeledBy:XXX
  • Presence of open/solved issues on the asset: issue:open // issue:solved
  • Priority = filter not yet available in search
  • Skipped = skipped:true / false
  • Status = status:todo / ongoing / labeled / reviewed
  • Type = type:prediction / default / review

Navigation shortcuts

You can easily navigate between labels and assets with arrow keyboards. Jump from one asset to the other with the up and down arrows : you will see the latest label each time. You can view the full history of labels with the right arrow, navigate through the list, and collapse it with the left arrow.

Send back to labeling

You can send an asset back to the labeler by clicking on the issue button.

Then you can click on Send back to queue.

This makes the asset have the top priority and be assigned to the labeler, effectively making the asset the next one to be labeled by the labeler.

← Questions and IssuesWorkload distribution →
  • The review Interface
  • Agreement
    • Search labels & assets
    • Navigation shortcuts
    • Send back to labeling