The Honeypot (or gold standard) is a tool to measure the accuracy of annotations. It helps to audit the work of annotators throughout the project. The Honeypot works by interspersing data instances for which you have defined a groundtruth label in the annotation queue so that you can measure the disagreement level between your gold standards and the annotations made by annotators.
The assets used for the Honeypot are given to be annotated to all project annotators. They are intelligently distributed throughout the annotation process.
Activate the Honeypot
You can set the Honeypot in the Settings tab, Quality section, below the consensus. Check the box to activate it.
Set the Honeypot
Honeypot should be set up by project managers before labelers start working, so that you get immediate return when they submit labels. At the highest granularity, Honeypot is defined on a label. Honeypot can be computed on a label under two conditions :
- the asset of the label is marked as a Honeypot ("isHoneypot" property is set to true, i.e. it can be used for Honeypot computation).
- there is a label of type "REVIEW" on this asset : this label will be used as the ground truth.
- If there are multiple "REVIEW" labels, the last one will be used as the ground truth.
Meeting those conditions
You can activate the "isHoneypot" property of an asset either from the interface or from the API.
- From the interface, by clicking on the Honeypot button, next to the Review button. Clicking on the button simultaneously submits a review and switches the "isHoneypot" property. When the Honeypot is activated for an asset, the button is yellow.
- From the API, you can update the "isHoneypot" property of assets using the playground method
kili.update_properties_in_assets(assets_ids=['id-1', 'id-2'], is_honeypot_array=[True, True])
If you activated the Honeypot of an asset from the interface, you already created a review label. Else, if you created it from the API, you can easily upload labels of type "REVIEW" with the playground method
Once Honeypot is defined on a label, it is defined on the asset of the label, on the member of the project (i.e. project user) and on the project.
If the previous two conditons are met, then :
- If the asset is not already labeled (it only contains the "REVIEW" Honeypot label), then all other members of the project will receive the asset from the queue of annotation, even though the asset has status "REVIEWED". When a labeler submits a label, the Honeypot metrics are updated.
- If the asset was already labeled when the author of the review label created the Honeypot : metrics are updated every 12 hours, so you can wait to see the Honeypot value updated for that first label. If you want to immediately get feedback (i.e. compute the Honeypot from a particular label), you can do it from the API :
kili.labels(label_id="id_of_the_default_label", fields=['honeypotMarkCompute']) # Asset level kili.assets(asset_id="id_of_the_asset", fields=['honeypotMarkCompute']) # Project user level kili.project_users(id="id_of_the_asset", fields=['honeypotMark']) # Project level kili.projects(project_id="id_of_the_asset", fields=['honeypotMark'])
The Honeypot score, expressed as a %, is calculated as follows:
At the label level
For NER: two annotations are equal if they refer to the same class, and if they cover the same portion of text.
For IMAGES: two annotations are equal if they refer to the same class, and if they cover the same pixels.
At the asset level: this is the average of the Honeypots at the label level.
At the annotator level: this is the average of the annotator labels Honeypots.
- For each project user (member of the project) x label, the Honeypot mark visible on the interface is the one of the last submitted label of the project user. We define this value as "honeypot-label/author-last"
- The Honeypot at the asset level is the mean of "honeypot-label/author-last", over all authors of a label on the asset.
- The Honeypot at the project user (member) level is the mean of "honeypot-label/author-last", over all labels (labeled by this project user).
- The Honeypot of the project is the mean of "honeypot-label/author-last" over all "DEFAULT" or "REVIEW" labels.