Kili Docs

Kili Docs

›Quality management

Introduction to Kili Technology

  • Introduction to Kili Technology
  • Kili Technology allows
  • Compatible browser

Getting Started

  • Getting started with Kili - Classification

Hosting

  • SaaS
  • On-Premise Data
  • On-Premise Entreprise

Concepts

  • Definitions
  • Status Lifecycle
  • Architecture

Users and roles

  • Roles by project
  • Users
  • Users and roles management

Projects

  • Audit labelers
  • Customize interface
  • Dataset
  • New project
  • Project overview
  • Projects
  • Projects list
  • Settings
  • Shortcuts

Image interfaces

  • Bounding Box
  • Classification
  • Point
  • Polygon
  • Polyline
  • Segmentation
  • Simple and intuitive interfaces

Text & PDF interfaces

  • Classification
  • Image transcription / OCR
  • Named entities recognition
  • Relations extraction

Video interfaces

  • Classification
  • Multi-frames classification
  • Multi-frames object detection
  • Transcription

Audio interfaces

  • Voice transcription / Speech to text

Data ingestion

  • Data ingestion made easy
  • Load data from a workstation
  • Load data from a public cloud
  • Data on premise or on private cloud
  • How to generate non-expiring signed URLs on AWS

Quality management

  • Consensus
  • Honeypot or Gold Standard
  • Instructions
  • Quality KPIs
  • Quality management
  • Questions and Issues
  • Review Process
  • Workload distribution

Automation

  • Human in the loop
  • Model based preannotation
  • Online learning
  • Queue prioritisation

Data export

  • Data export
  • Data format
  • Example

Python - GraphQL API

  • GraphQL API
  • Python API

Code snippets

  • Authentication
  • Create a Honeypot
  • Create a user
  • Creating Consensus
  • Delete the data
  • Export data
  • Export labels
  • Import data
  • Import labels
  • Prioritize assets
  • See the Consensus of an annotation
  • See the Honeypot of an annotation
  • Throttling

Recipes

  • AutoML for faster labeling with Kili Technology
  • Create a project
  • Exporting a training set
  • Importing medical data into a frame project
  • Importing assets
  • Import rich-text assets
  • Importing predictions
  • Reading and uploading dicom image data
  • How to query using the API
  • Labelled Image Data & Transfer Learning
  • Webhooks

Change log

  • Change log

Questions and Issues

Kili Technology is built to increase collaboration and improve annotation quality. To serve this purpose we allow labelers and reviewers to communicate using questions and issues.

Questions

Questions are created by labelers from the labeling interface. When a labeler is unsure about a label he can ask a question. When a question is asked, reviewers are notified on the review where they then answer the question.

The number of open questions appears as a yellow notification on the issue button.

Ask a question

To ask a question click on the issue button and ADD QUESTION.

You can now type your question in the text area and press enter to validate.

You can then submit or skip the asset. Your question will be available to reviewers and yourself in the review.

Answer a question

Reviewers and admin can answer questions in the review. Click on the issue button to access open questions. Click on a question to answer it. After typing your answer click on SEND BACK TO QUEUE to send the asset back to the labeler with the answer. Once an asset is sent back it is marked with a higher priotity to ensure a fast correction of the label.

Issues

Issues are used by reviewers to make a feedback to labeler regarding an annotation. The number of open questions appears as an orange notification on the issue button.

The number of open issues in a project is also available on the project overview.

Create an issue

Issues are created from the review. Click on the issue button and ADD ISSUE. After typing a comment to explain the issue, click on SEND BACK TO QUEUE to send the asset back to the labeler. Once an asset is sent back it is marked with a higher priotity to ensure a quick correction of the label.

After beeing sent back the status of the asset is changed from LABELED to ONGOING.

Solve an issue

Once the labeler has corrected its annotation and re-submited the asset, reviewers can solve an issue by either using the check box at top right of the issue or by reviewing the asset.

Quick filtering open questions / issues

You can filter assets displayed in the review to show only those that have open questions/issues linked to them.

Click on the issue button and then on see open issues or see open questions to filter the assets of the review accordingly.

← Quality managementReview Process →
  • Questions
    • Ask a question
    • Answer a question
  • Issues
    • Create an issue
    • Solve an issue
    • Quick filtering open questions / issues