We’re happy to announce that we’re starting a new rubric in our blog, namely software reviews. As we work in the high-tech industry, it’s fitting to learn more about the tools that can help us achieve our goals. And for our first choice, we’ll be covering CVAT, the data annotation software that we at Label Your Data know, use, and love.
Before we begin with the review, just a few words about why we chose to start with a data annotation tool. Data annotation is one of the most time-consuming and labor-intensive processes in machine learning. However, it is also an essential part of creating an AI algorithm. While there are ways to avoid data annotation, they only fit certain niche cases. The largest part of AI projects relies heavily on data annotation as an instrument to succeed in the model training process.
And for data annotation, you need professional software. The CVAT annotation tool is one of the best examples of simple, adaptable, easily adjustable software options that are indispensable in data annotation work. Let’s take a look at the major features of CVAT and evaluate the pros and cons of this tool.
Data Annotation: Why You Need It and How the CVAT Annotation Tool Can Help
Let’s first talk about annotation. When you collect the data, your set exists in a raw form. It is unprocessed, usually uncleaned, and, most importantly, unlabeled. This means that, if you feed it into your beautiful algorithm, the computer won’t be able to understand it. And what you need to do is to label your data.
Labeling or annotation is the process of adding meaningful tags or labels to the pieces of data in your set. These tags are then used by your algorithm to make sense of the data and to train. We won’t go into too much detail about the process of annotation since we’ve already talked about it in our blog. Just visit our article on data labeling if you’re interested.
What you get as a result of the annotation process is labeled data. Now, this is the data set that your algorithm will understand. It is prepared to be used to train your algorithm by the example of the labels you’ve assigned. That’s why there are not a lot of projects that can skip the step of data annotation.
What’s the CVAT Annotation Tool and How Does It Work?
Computer Vision Annotation Tool (CVAT) is the software created for the annotation of photo and video data. While there is a possibility to add labeling tasks for other types of data (such as text and audio), CVAT was built to deal primarily with the visual format. As follows, CVAT is being used for computer vision (CV) tasks that are based on image and video data sets.
The CVAT labeling tool was written on Python and JS with the goal of covering all the major CV tasks: image classification, object detection, and image segmentation. It has an intuitive enough interface and a set of functions that you don’t require a lot of training to master. Let’s take a closer look at the CVAT annotation.
How to Start Working with the CVAT Annotation Tool: The Steps of a New Project Creation
- First of all, you create a Project in CVAT that contains a name and a list of labels (or tags) that you will use to annotate your data. There are also other features you can specify during this step, such as bug tracker for detailed labeling specification, segment size to divide big data sets, image quality, etc.
- Then you create Tasks that contain annotation frames. For this, you need to upload the data set from your storage to CVAT. You can also specify the number of jobs, which are subtasks, which allow dividing the tasks between several annotators.
- You can start annotating by using one of the four shapes available in CVAT:
- Bounding boxes (2D boxes to frame the object of interest)
- 3D cuboids (similar to bounding boxes but with the added third dimension)
- Polygons (irregular lines to track the shape of the object)
- Polylines (lines generally used for road marks labeling)
- Key points (a set of meaningful points mostly used for natural objects annotation, e.g., faces, poses, and so on)
The additional CVAT features like interpolation and automatic annotation can help facilitate the labeling process significantly. Visit the shortcut menu to see some treats to make your annotation life easier.
- Once finished, you can download the annotated data from the CVAT annotation tool and use it further to train your model. It’s that simple!
Annotation Tool Competition: Is CVAT Your Best Choice?
There are quite a few competitors of the CVAT annotation tool that we can name, SuperAnnotate and Supervisely standing out among them. All of these tools have similar functionality with CVAT with a few distinctive features and different interfaces. SuperAnnotate supports smart segmentation, which is a cool feature that allows you to save a lot of time. Supervisely can boast in-tool statistics that shows, for example, the time spent on the job and the time spent in the interface. CVAT has all those features, too, and it additionally supports more upload formats. Besides, it’s open-source, unlike SuperAnnotate or Supervisely.
We’ve created the lists of advantages and disadvantages of the CVAT annotation tool based on our own experience of working with the tool. Keep in mind that we are not the developers of CVAT and might not know about all the existing possibilities of the tool.
CVAT Pros: Open-Source and Friendly
We’ll start with the good stuff since CVAT is one of the best open-source annotation tools out there:
- CVAT is an open-source professionally developed code under the license of MIT.
- CVAT is web-based. You don’t need to download anything; just open the web page and start annotating.
- Possibility to collaborate inside CVAT. A whole team can be working on an annotation project in CVAT, which allows to split the tasks and helps finish the job faster.
- Easy to use and deploy. The CVAT annotation tool has an understandable interface with no need for additional training or expert knowledge.
- Wide range of applications. CVAT covers all major CV annotation tasks and can be used for a variety of annotation scenarios.
- CVAT supports more upload formats than most of its competitors.
- Integrative and flexible. The CVAT annotation tool supports several useful tools like TensorFlow Object Detection API, ELK analytic system, or NVIDIA CUDA Toolkit.
- Support of certain types of text annotation. It’s very limited and exists for certain attribute annotation, entity recognition, and OCR annotation only. Besides, the text needs to be uploaded as an image into CVAT so it’s not entirely convenient. But hey, it’s there and we’ll take it.
CVAT Cons: Limited Workflow Functionality
Now, to the flaws of CVAT. We’ve based them mostly on our own user experience rather than engineering issues:
- Limited support of browsers. The CVAT annotation tool was only tested on Google Chrome and Mozilla. It might also work with Opera and Yandex browsers but do it at your own risk. We also tested it on Safari, which didn’t give any positive results.
- Limited source code documentation. If you want to get your hand on the development of CVAT, it won’t be easy.
- CVAT performance problems. If the CVAT server fails, an annotator can keep working on the task without the possibility to save it, which can lead to losing a lot of time on the job that cannot be saved.
- Lack of automatic checking in CVAT. The testing must be done manually, which extends the time spent on the job.
- Lack of statistics inside a CVAT task. You can only see the statistics of a job, which may be okay for small tasks that include around five jobs but turns into pain for large tasks with 50+ jobs.
- No possibility to transfer annotation between layers in CVAT. If one object covers the other, they need to be labeled on different layers, which is a problem for complicated tasks with multiple objects to detect.
- The CVAT annotation tool doesn’t support PDF files. You need to convert them into photo format to work with them.
- No CVAT notifications for finished tasks. When an annotator finishes their task, there’s no way to notify a supervisor or a QA inside the software, which creates unnecessary procedural difficulties and extends the downtime.
- No annotator-based filter in CVAT. The search for tasks done by a specific person needs to be done by hand. It’s fine for smaller projects with 30 tasks but is very time-consuming for 100+ task projects.
Features and Use Cases of CVAT
As you can see, the CVAT annotation tool is not without its flaws. But it still is a great piece of software for any company from any industry when it has a computer vision project on its hands. We’d like to share with you a couple of exciting use cases that we completed using the CVAT annotation tool to demonstrate the versatility and usability of this piece of software.
Insurance: Bounding Boxes for Damage Evaluation in CVAT
Automation is in full swing today, and why not add insurance to the list of the tasks that machines help humans do? Our Client came to us with a large annotation project where we were asked to label the car damages. The goal was to train the algorithm to evaluate the damages after car crashes and accidents and to optimize the process of determination of insurance payments.
The damages were of varying degrees of severity and located on different parts of the cars, which added to the complexity of the project. For this job, we used bounding boxes in CVAT to tag the location, size, and severity of the damages.
Construction: The Two CVAT Annotation Tool Cases for Planning and Security Improvement
Architecture is a time-consuming and demanding subject because of the complex planning involved. But an AI can do the job better and faster than an architect. We got a job of annotating the architectural blueprints with the goal of estimating the level of complexity of the future building project. This helped the algorithm to calculate the time and resources needed for the construction.
Our second construction case was connected to video annotation. We used the CVAT annotation tool to label videos from construction sights to track the presence of helmets and reflective safety clothing. The goal of this project was to improve the security of the construction sights, which are on top of the list of notoriously hazardous working spaces.
Health Care: Semantic Segmentation in CVAT for Skin Disease Detection
Medicine and health care are among the top adopters of AI, and we completed our fair share of health-related labeling projects using the CVAT annotation tool. One of the most challenging was the labeling of skin diseases. This project required a high level of expertise in health care and initial anonymization of the data, which is not uncommon for health care annotation tasks. For the project itself, we used semantic segmentation to tag the areas of problematic skin.
Summary: So Is CVAT Worth the Time?
The short answer is: yes, CVAT is definitely worth the time. That’s why we at Label Your Data preferred this annotation tool after trying out several other software options.
For a longer answer, we think the CVAT annotation tool is a great piece of software if you have a lot of annotation tasks but cannot spend a fortune on developing your own state-of-the-art software. There probably are annotation tools that allow easier and faster annotation of data sets that are on the smaller side. However, CVAT has a ton of useful features that simplify and facilitate the process of annotation if you work with it on a regular basis. These features also help to improve the pace of work for larger projects and save you some precious time.
Overall, the CVAT annotation tool is a nice all-around solution for computer vision tasks. Its open-source code allows starting the annotation work without any additional spending. For a more professional approach, it would definitely be simpler and probably less expensive to buy a license of one of the many prepaid solutions. However, if you need to customize the tool for your many needs, you might find CVAT is the better choice with the addition of developers that will provide for all your highly specific or unique desires.
In our next software review, we’ll be covering an entirely different tool, TensorFlow. It is one of the most popular choices for AI engineers who seek a professional platform to design, build, and deploy ML applications. Sign up for our blog to stay tuned!
Table of Contents
Get Notified ⤵
Receive weekly email each time we publish something new:
