Consensus

Consensus works by having more than one annotator annotating the same asset. Once the asset is annotated, a consensus score is calculated to measure the agreement level between the different annotations for a given asset. It is a key lever for controlling production quality.

Set consensus

To activate consensus, go to your project settings, tab "Quality" and tick the box to activate it. You will need then to set the following parameters:

  • Total Asset Coverage: it is the % of the dataset that will be annotated several times. If the dataset contains 10,000 assets, and the Total Asset Coverage is at 10%, it means that 1,000 assets will be annotated several times. The assets to be labelled multiple times will be chosen randomly.

  • Minimum Consensus Size: this is the number of annotators that will have to annotate the same asset. It must be set to a number less than or equal to the total number of annotators.

The assets that have been selected by the algorithm to participate in the consensus remain at the top of the stack until they have been annotated by Minimum Consensus Size annotators.

Consensus calculation

Depending on your task, the consensus will be computed differently.

Single-class classification

  • At the asset level: 1 / number_of_selected_categories
  • At the annotator level: we take the average of the consensuses for the assets annotated by the annotator as a simple and intuitive estimator of the consensus level of an annotator.
  • At the project level : Fleiss kappa method with assets on each row of the Fleiss matrix and classes on each column :
    • N: images to be classified
    • k : possible categories
    • n : number of annotators

In this example, two different classes have been selected (number_of_selected_categories = 2), so the consensus at the asset level is: consensus = 1 / 2 = 50%

Multi-class classification

  • At the asset level: 1 / number_of_selected_categories * sum_on_selected_categories( number_of_annotators_that_selected_this_category / number_of_annotators )
  • At the annotator level: we take the average of the consensuses for the assets annotated by the annotator as a simple and intuitive estimator of the consensus level of an annotator.
  • At the project level: the average of the Fleiss kappas with the assets on each row of the Fleiss matrix and each class and its opposite on each column :
    • N: images to be classified
    • k : possible categories
    • n : number of annotators

In this example, two different classes have been selected (number_of_selected_categories = 2) with two annotators (number_of_annotators = 2), so the consensus at the asset level is:

consensus = 1/2 * ( 2/2 + 0/2 + 1/2 ) = 1/2 * 3/2 = 75 %

Object detection

  • At the asset level: Each image is considered as a set of pixels to be classified. A pixel can be classified into several non-exclusive categories representing different objects. Therefore we calculate for each image a score based on the Fleiss Kappa method for each class appearing on the image:
    • N : pixels to classify
    • k : 2 possible categories, the selected class or the complementary
    • n : number of annotators that have labeled the image
  • At the annotator level : we take the average of the consensuses for the assets annotated by the annotator as a simple and intuitive estimator of the consensus level of an annotator.
  • At the project level: we take the average of the assets consensuses (for the assets with consensus) as a simple and intuitive estimator of project level of consensus.

Character recognition (OCR)

  • At the asset level: we interpret the OCR task as the composition of an object detection task (selecting a box) and a text entry task (text contained in a box). Consensus is thus defined as the following average:
    • the consensus on object detection (as described in the paragraph above),
    • consensus on the text entered: the contents of the boxes are compared from top to bottom and from left to right with the distance difflib.SequenceMatcher. This distance compares two strings, with 0 indicating that the strings are equal.
  • At the annotator level: we take the average of the consensuses for the assets annotated by the annotator as a simple and intuitive estimator of the consensus level of an annotator.
  • At the project level: we take the average of the assets consensuses (for the assets with consensus) as a simple and intuitive estimator of project level of consensus.
← InstructionsHoneypot or Gold Standard →