Kili Technology is built to increase collaboration and improve annotation quality. To serve this purpose we allow labelers and reviewers to communicate using questions and issues.
Questions are created by labelers from the labeling interface. When a labeler is unsure about a label he can ask a question. When a question is asked, reviewers are notified on the review where they then answer the question.
The number of open questions appears as a yellow notification on the issue button.
Ask a question
To ask a question click on the issue button and
You can now type your question in the text area and press enter to validate.
You can then submit or skip the asset. Your question will be available to reviewers and yourself in the review.
Answer a question
Reviewers and admin can answer questions in the review. Click on the issue button to access open questions. Click on a question to answer it. After typing your answer click on
SEND BACK TO QUEUE to send the asset back to the labeler with the answer. Once an asset is sent back it is marked with a higher priotity to ensure a fast correction of the label.
Issues are used by reviewers to make a feedback to labeler regarding an annotation. The number of open questions appears as an orange notification on the issue button.
The number of open issues in a project is also available on the
Create an issue
Issues are created from the review. Click on the issue button and
ADD ISSUE. After typing a comment to explain the issue, click on
SEND BACK TO QUEUE to send the asset back to the labeler. Once an asset is sent back it is marked with a higher priotity to ensure a quick correction of the label.
After beeing sent back the status of the asset is changed from
Solve an issue
Once the labeler has corrected its annotation and re-submited the asset, reviewers can solve an issue by either using the check box at top right of the issue or by reviewing the asset.
Quick filtering open questions / issues
You can filter assets displayed in the review to show only those that have open questions/issues linked to them.
Click on the issue button and then on
see open issues or
see open questions to filter the assets of the review accordingly.