Questions and Issues

Kili Technology is built to increase collaboration and improve annotation quality. To serve this purpose we allow labelers and reviewers to communicate using questions and issues.

Questions

Questions are created by labelers from the labeling interface. When a labeler is unsure about a label he can ask a question. When a question is asked, reviewers are notified on the review where they then answer the question.

The number of open questions appears as a yellow notification on the issue button.

Ask a question

To ask a question click on the issue button and ADD QUESTION.

You can now type your question in the text area and press enter to validate.

You can then submit or skip the asset. Your question will be available to reviewers and yourself in the review.

Answer a question

Reviewers and admin can answer questions in the review. Click on the issue button to access open questions. Click on a question to answer it. After typing your answer click on SEND BACK TO QUEUE to send the asset back to the labeler with the answer. Once an asset is sent back it is marked with a higher priotity to ensure a fast correction of the label.

Issues

Issues are used by reviewers to make a feedback to labeler regarding an annotation. The number of open questions appears as an orange notification on the issue button.

The number of open issues in a project is also available on the project overview.

Create an issue

Issues are created from the review. Click on the issue button and ADD ISSUE. After typing a comment to explain the issue, click on SEND BACK TO QUEUE to send the asset back to the labeler. Once an asset is sent back it is marked with a higher priotity to ensure a quick correction of the label.

After beeing sent back the status of the asset is changed from LABELED to ONGOING.

Solve an issue

Once the labeler has corrected its annotation and re-submited the asset, reviewers can solve an issue by either using the check box at top right of the issue or by reviewing the asset.

Quick filtering open questions / issues

You can filter assets displayed in the review to show only those that have open questions/issues linked to them.

Click on the issue button and then on see open issues or see open questions to filter the assets of the review accordingly.