Labelled Image Data & Transfer Learning

Kili Tutorial: Labelled Image Data & Transfer Learning

This recipe shows you how you can leverage Kili to rapidly create your own custom image detection model by transfer learning. Even if the pre-trained model had a wide training set, it might not be directly adapted to your use case. But it stil learned very important techniques to make sense of a set of RGB pixels, and thus can speed-up your annotation process by Transfer Learning. We'll then try to provide labeled data to help the model improve its analysis, a process that is called fine-tuning.

The main ingredients we are going to use are :

  • The data you have just labelled via Kili as training set
  • The excellent yolov3 library to craft our model

Steps

Those are the steps we are going to detail in this recipe :

  1. Install kili-playground (see the project repository)
  2. Install yolov3 by following their requirements, and download the pretrained weights.
  3. Make the same for our recipe by installing our additional requirements : pip install -U -r requirements.txt
  4. Our main.py script will require you to provide your Kili credentials, endpoint and your yolov3 repository location path as argument. You can as well provide them by setting the following environment variables (they will be used by default by the recipe) : EMAIL, PASSWORD, PROJECT_ID, API_ENDPOINT, YOLO_PATH.
  5. Get a first labeling of our unlabeled data with Yolo
  6. After labeling some data, we can let the model run another time to let it improve.
import os

# !pip install kili # uncomment if you don't have kili installed already
from kili.client import Kili

api_key = os.getenv('KILI_USER_API_KEY')
api_endpoint = os.getenv('KILI_API_ENDPOINT') # If you use Kili SaaS, use 'https://cloud.kili-technology.com/api/label/v2/graphql'

kili = Kili(api_key=api_key, api_endpoint=api_endpoint)

Below is an example of jsonInterface for the 80 class model, but it can be easily adapted for a different number of class.

job_id = 'JOB_0'
json_interface = {
  "jobs": {
    job_id: {
      "mlTask": "OBJECT_DETECTION",
      "content": {
        "input": "radio",
        "categories": {
          "0": {"name": "person"},
          "1": {"name": "bicycle"},
          "2": {"name": "car"},
          "3": {"name": "motorcycle"},
          "4": {"name": "airplane"},
          "5": {"name": "bus"},
          "6": {"name": "train"},
          "7": {"name": "truck"},
          "8": {"name": "boat"},
          "9": {"name": "traffic light"},
          "10": {"name": "fire hydrant"},
          "11": {"name": "stop sign"},
          "12": {"name": "parking meter"},
          "13": {"name": "bench"},
          "14": {"name": "bird"},
          "15": {"name": "cat"},
          "16": {"name": "dog"},
          "17": {"name": "horse"},
          "18": {"name": "sheep"},
          "19": {"name": "cow"},
          "20": {"name": "elephant"},
          "21": {"name": "bear"},
          "22": {"name": "zebra"},
          "23": {"name": "giraffe"},
          "24": {"name": "backpack"},
          "25": {"name": "umbrella"},
          "26": {"name": "handbag"},
          "27": {"name": "tie"},
          "28": {"name": "suitcase"},
          "29": {"name": "frisbee"},
          "30": {"name": "skis"},
          "31": {"name": "snowboard"},
          "32": {"name": "sports ball"},
          "33": {"name": "kite"},
          "34": {"name": "baseball bat"},
          "35": {"name": "baseball glove"},
          "36": {"name": "skateboard"},
          "37": {"name": "surfboard"},
          "38": {"name": "tennis racket"},
          "39": {"name": "bottle"},
          "40": {"name": "wine glass"},
          "41": {"name": "cup"},
          "42": {"name": "fork"},
          "43": {"name": "knife"},
          "44": {"name": "spoon"},
          "45": {"name": "bowl"},
          "46": {"name": "banana"},
          "47": {"name": "apple"},
          "48": {"name": "sandwich"},
          "49": {"name": "orange"},
          "50": {"name": "broccoli"},
          "51": {"name": "carrot"},
          "52": {"name": "hot dog"},
          "53": {"name": "pizza"},
          "54": {"name": "donut"},
          "55": {"name": "cake"},
          "56": {"name": "chair"},
          "57": {"name": "couch"},
          "58": {"name": "potted plant"},
          "59": {"name": "bed"},
          "60": {"name": "dining table"},
          "61": {"name": "toilet"},
          "62": {"name": "tv"},
          "63": {"name": "laptop"},
          "64": {"name": "mouse"},
          "65": {"name": "remote"},
          "66": {"name": "keyboard"},
          "67": {"name": "cell phone"},
          "68": {"name": "microwave"},
          "69": {"name": "oven"},
          "70": {"name": "toaster"},
          "71": {"name": "sink"},
          "72": {"name": "refrigerator"},
          "73": {"name": "book"},
          "74": {"name": "clock"},
          "75": {"name": "vase"},
          "76": {"name": "scissors"},
          "77": {"name": "teddy bear"},
          "78": {"name": "hair drier"},
          "79": {"name": "toothbrush"}
        }
      },
      "required": True,
      "tools": ["rectangle"],
      "instruction": "Detect following objects"
    }
  }
}

Let us create the project in Kili :

title = 'YOLOv3 with Kili Technology'
description = 'Ceci est un projet test'
input_type = 'IMAGE'

project = kili.create_project(
    title=title,
    description=description,
    input_type=input_type,
    json_interface=json_interface
)
project_id = project['id']

and add some collaborators to help us for the labeling task.

emails = ['fx@kili-technology.com']
for user_email in emails:
    kili.append_to_roles(project_id=project_id, user_email=user_email, role='ADMIN')

Add images from the airplane dataset, and let's see how YOLO performs on it :

Man on a bike

content_array = ['https://images.unsplash.com/photo-1532973497172-04b34d604825',
                 'https://images.unsplash.com/photo-1528629297340-d1d466945dc5']
external_id_array = ['airplane',
                     'man-on-bike']
kili.append_many_to_dataset(project_id=project_id,
                                  content_array=content_array,
                                  external_id_array=external_id_array)

We will now clone yolov3 framework from Github, and download the pretrained weights yolov3.pt from Google drive

current_path = !pwd
assert len(current_path) == 1
current_path = current_path[0]
yolo_path = os.path.join(current_path, '../../yolov3')
!git clone --branch v7 https://github.com/ultralytics/yolov3.git {yolo_path}

weights_id = '1SHNFyoe5Ni8DajDNEqgB2oVKBb_NoEad'
weights = os.path.join(current_path, '../../yolov3.pt')

import requests

def download_file_from_google_drive(id, destination):
    URL = "https://docs.google.com/uc?export=download"

    session = requests.Session()

    response = session.get(URL, params = { 'id' : id }, stream = True)
    token = get_confirm_token(response)

    if token:
        params = { 'id' : id, 'confirm' : token }
        response = session.get(URL, params = params, stream = True)

    save_response_content(response, destination)    

def get_confirm_token(response):
    for key, value in response.cookies.items():
        if key.startswith('download_warning'):
            return value

    return None

def save_response_content(response, destination):
    CHUNK_SIZE = 32768
    with open(destination, "wb") as f:
        for chunk in response.iter_content(CHUNK_SIZE):
            if chunk:
                f.write(chunk)

download_file_from_google_drive(weights_id, weights)

As explained above, let's install all the dependencies to train Yolov3 :

!wget -O /tmp/yolov3-requirements.txt https://raw.githubusercontent.com/ultralytics/yolov3/master/requirements.txt
!pip install -r /tmp/yolov3-requirements.txt

!pip install torchvision==0.6.0
!pip install tensorflow==2.1.0
!pip install opencv-python==4.3.0.36

#https://github.com/ultralytics/yolov3/issues/1465
!pip uninstall -y thop 

# Force MKL environment variable
import os
os.environ['MKL_SERVICE_FORCE_INTEL'] = 'true'

Everything is ready now, we'll just launch YOLOv3, leveraging the script we provide, with

  • The API endpoint is the SAAS or on-premise link
  • your Kili credentials
  • the desired number of inferences
  • our project's id
  • the pre-trained weights
  • the installation path of Yolo
import requests
import sys

base_url = 'https://raw.githubusercontent.com/kili-technology/kili-playground/master/recipes/image-object-detection-with-yolo/'
for filename in ['coco.template.data', 'main.py', 'yolov3.template.cfg']:
  with open(filename, 'wb') as f:
    f.write(requests.get(base_url + filename).content)

command = f'''python ./main.py \
    --api_endpoint {api_endpoint} \
    --api_key {api_key} \
    --job_id {job_id} \
    --number_of_inferences 1 \
    --project_id {project_id} \
    --weights {weights} \
    --yolo_path {yolo_path}
'''

!{command}

# Update label references to assets
kili.labels(project_id=project_id, fields=['assetIdCompute'])

# Check pre-annotations were saved in Kili
asset = kili.assets(project_id=project_id,
                                             external_id_contains=['man-on-bike'])

assert len(asset[0]['labels']) > 0

Here is the result for our cyclist, well identified by Yolo, but there can be some improvements that will be brought by the Transfer learning

Yolo Result

Launching your training

After having labelled a few data instances, you are ready to launch a new training using transfer learning.

Next launch the following command to keep tuning the weights according to your data's specificity. The number of inferences will determine the number of full passes over your labeled data :

number_of_inferences = 5
command = f'''python ./main.py \
    --api_endpoint {api_endpoint} \
    --api_key {api_key} \
    --job_id {job_id} \
    --number_of_inferences {number_of_inferences} \
    --project_id {project_id} \
    --weights {weights} \
    --yolo_path {yolo_path}
'''
#!{command}

Conclusion

In this short recipe, we provide a script to help you accelerate your labeling task, leveraging both the powerful Kili interface and the State-of-the-art Yolo pre-trained model.

Feel free to keep modifying this recipe to your needs, and to check out our other recipes or the Kili documentation

← How to query using the APIWebhooks →