Table of Contents

  1. Building an in-house data labeling team
    1. Outsourcing data labeling team
    2. Quality, speed and price — the golden triangle of data labeling

For many companies and researchers looking to develop AI or machine learning algorithms, there's a choice that they must make. Since self-learning models require plenty of annotated information to train before they go live, the question poses: create an in-house team or outsource data labeling. Many think that providing data labeling in-house increases security. Others may consider building one's own team a costly job, and see outsourcing as an ultimate time and resource saver. In reality, there are benefits and hidden pitfalls in both of these approaches. Choosing the right one is a strategic decision, which may affect the whole developing process.

Building an in-house data labeling team

Having an in-house data labeling team can be beneficial to the project in many ways. First and foremost, for many companies in-house data labeling teams are about direct oversight of the whole data annotation process. Therefore, having a team in a physical proximity is an idea that works well. Security is the second reason to choose building one's own data annotation team. When the tasks are concerned highly sensitive topics and images which cannot be transmitted over the internet, or when files to be annotated may potentially breach the security standards if sent to the third party, hiring on-site teams is the way to go. Finally, as a rule of thumb, it is more common to have an in-house team for long-term AI projects, where data flow is continuous and you need individuals to annotate it during the prolonged periods of time.

The cons of creating your own data labeling team are quite obvious. The tremendous amount of time and resources needed to hire and train a professional team, provide a secure environment for their work, develop software with the right tools, compile instructions are a cost at which a good team is developed. Constant management of a new team is time-consuming and requires HR professionals at your side. And if the workload is seasonal, or project-to-project, you'll create a constant staff turnover.

Outsourcing data labeling team

Label Your Data annotators (office #2, Kyiv, Ukraine)

Nowadays many companies who provide outsourced services understand the potential shortcomings and align their processes to meet the requirements of security certificates and provide on-demand personalized services to clients. Third party data annotation tools are usually more sophisticated and allow your company to work with experienced and specialized annotators.

Outsourcing data labeling is saving you time and stress of doing it all by yourself. The pros of this approach lie, primarily, in speed and cost efficiency. On the contrary to on-site data labeling which would be most suitable for high volumes of changing data, third-party solutions are more suitable for high volume data for short-term projects or projects that require data annotators to work for short periods of time regularly (for instance, to update model training data once a month). It is better to have a clear set of instructions about how data should be handled or, at least, a very clear idea in mind about what your task is about.

Labelers are very important elements of data annotation process, especially when it comes to specific data labeling tasks, such as ones relating to agriculture, medicine, sports or technology. Diverse expertise and availability of specialists in various fields is among the benefits of outsourced services, which do their best to accommodate their clients. Management of people all together is off your hands when you decide to outsource. There is a surprising amount of work that goes into hiring and training employees, connecting and securing the tech, communicating instructions and feedback. Related closely to this, are the more flexible possibilities of scaling when outsourcing. Scaling up or down along with your project needs isn't a problem, since labelers can be reassigned to different projects, instead of being completely laid off.

Label Your Data annotators (office #2, Kyiv, Ukraine)

One last thing to consider is the software. When making your own team, you'll need to put additional time into modifying an existing software for data labeling or even develop your own entirely. Whilst when hiring a team to do data labeling for you, they can help you in developing or choosing the right tools for the task you need.

Among the few things that may be frustrating for managers looking into outsourcing options are less control over the processes and need for trust third parties to handle your data. Even if outsourced, a team of labelers remains an extension of your own team. This mindset should be maintained for a smoother collaboration through continuous communication and clear instructions.

Quality, speed and price — the golden triangle of data labeling

There is a primary set of measures that define a good data annotation team. It's the quality of their work, how fast they perform their tasks and the funds needed to cover all the spending.

In our previous articles, we have discussed the importance of high quality data labeling and best practices to measure data labeling quality. Indeed, datasets themselves matter — they must be balanced and include a variety of data points. However, the quality of employees' annotation is of essence too. Labeled dataset quality is defined by how accurately points are placed on each of the pictures and also how consistently these points are accurate. Various manual and automated QA methods are introduced to measure both of these, including consensus algorithm, benchmarking and gold standard, Cronbach's alpha test or a combination of these.
Speed and its variables
There are plenty of points to consider when approximating the time required to label a dataset. These include:
  • People. How many data annotators are needed in order to complete the task in a certain timeframe.
  • Usability of the software. UI and UX are the key! Software must be designed in a utilitarian way, most comfortable for the annotators to work with. A simple example: if getting to an essential feature requires four clicks, it's going to add up time-wise when annotating thousands of images or objects per day.
  • Possibility of process automation. Automation is a huge time saver, in this field too! Instance segmentation, interpolation, automatic detection of key points — many processes may be found in so-called model zoos.
  • Quality assurance. Often neglected in the initial calculations, the QA process may also be lengthy and require extra time to be accounted for.
How data labeling price is formed
Data labeling price is formed based on the client's needs and wants. If you decide to outsource, most of the companies, including Label Your Data, have a ballpark rate for the simplest tasks they provide. All additional costs vary. Majorly, the factors that play a significant role in this are:
  • Annotators. The number of people working on your project is, perhaps, the largest spending you would have to account for in the data labeling process.
  • Computer software. Data annotation programs and whether they need to be customized to fit a specific task is costly both time- and money-wise, whether you decide to outsource or develop your own team.
  • Pilot. For specialized outsourcing companies, a major step for determining further work, timeframes and costs are a pilot project, during which you can look into the quality and speed of annotators' work and decide whether they are a right fit for the company or project.
Interested in collaborating with us? Request Your Quote →
Veronika Gladchuk by Veronika Gladchuk
on July 28, 2020.
Infinite Community

Free Infinity Membership.
Subscribe for updates:

Count Me In
✔︎ Congrats! You are on the list.

Build Your AI App Faster – Outsource Data Annotation

High Quality + Certified Security

PCI DSS Level 1 Compliance   ISO 27001:213 Security Certification
Get Your Quote →