iMerit vs. Humans in the Loop: Comparing Data Labeling Providers

Table of Contents

As new machine learning projects continue to emerge, the importance of preparing accurate data for them is higher than ever before. Data annotation services deal with this task, making a critical contribution to AI development.

That's why we have prepared for you a comparative analysis of iMerit and Humans in the Loop, two leading data labeling companies in the market. We've analyzed their main data annotation services, pricing, and more to save you time on research. Check if they have what you're looking for.

iMerit vs. Humans in the Loop: Company Profiles

Feature	iMerit	Humans in the Loop
Founded	2012	2017
Headquarters	San Jose, California	Sofia, Bulgaria
Market Focus	Autonomous vehicles Medical AI Geospatial technology Financial services Commerce Government Agricultural AI	Geospatial technology Automotive Medical Industrial Agricultural Retail

iMerit Company

iMerit was founded in 2012 by Dipak Basu. Today with over 5000 employees, the company offers full end-to-end services for data annotation in a wide array of industries. The company is headquartered in San Jose, California, but also has offices in New Orleans, LA, Kolkata, and Bengaluru, India.

Humans in the Loop Company

Humans in the Loop is a social enterprise which provides ethical dataset annotation and AI model validation services. Its data labeling services are provided by over 250 specialists located in Sofia, Bulgaria, their headquarters, but also Malaysia and Ukraine, Turkey, and the Middle East. One of its missions is to help people from countries that experience armed conflicts and those forced to displace.

Services and Products

iMerit Services and Products

In comparison to other data labeling companies that offer a limited number of services, the iMerit company works with both texts and images. It provides data labeling for machine learning, computer vision, natural language processing (NLP), as well as content services.

Their main services include:

Reinforcement learning from human feedback (RLHF) for LLMS and LVMS
Image annotation
Video annotation
Text annotation
Audio transcription
LiDAR annotation
Sentiment analysis
Content moderation
Product categorization
Image segmentation

Besides, you can receive data labeling and annotation services in various languages. In addition to English, their annotators based in Latin America, Germany, US, and Bhutan, can support other languages on request.

Humans in the Loop Services and Products

Humans in the Loop is a data labeling company that mostly specializes in computer vision. Their main services include:

Dataset annotation
Dataset collection
Human-in-the-loop for active learning
Real-time edge case handling
Reinforcement learning with human feedback

If your project is based on computer vision, Humans in the Loop can not only annotate the data for you. They can also collect it, review your final models, and assist with their testing before final deployment.

Pricing Models

Feature	iMerit	Humans in the Loop
Pricing Structure	Subscription-based, per task, discounts for high volume	Depending on the task, per task or per hour
Pricing Tiers	Monthly subscription for Ango Hub platform Universal annotation task pricing varies by dataset volume Discounts for more than 100k objects per month One-time charge for custom output export	Individual calculation of price Dependent on volume, timelines, and difficulty of the project
Free pilot	Yes	Yes
Additional Notes	Discounts based on annotator's language and location Pricing flexibility depending on the scope of annotation	Available for express/urgent annotation

Among many data annotation companies in the market, Humans in the Loop's free trial is counted per hour and not volume. With 2 hours of free annotation, both you and the company will be ready to estimate further timelines and cost of the annotation accomplishment.

Dataset Types

iMerit Dataset Types

iMerit technology services include various dataset types. They range from datasets for text annotation to video and image annotation. Besides, they transform audio data into text formats that can be further processed by computers. The supported formats are then used for named entity recognition and conversation categorization tasks.

Humans in the Loop Dataset Types

While labeling data for machine learning projects, Humans in the Loop work with the majority of acceptable dataset types. Among the most popular ones they mention JSON, Yolo, COCO, and Pascal Voc XML. However, they can also consider and find solutions for your specific format, both for importing and exporting data.

Data Annotation Tools

	iMerit	Humans in the Loop
Tool (ownership)	Proprietary tool	Open source, third-party tools
Name	Ango Hub	Alegion, Diffgram, Human Lambdas, Hasty, Kili Technology, Lightly, V7, Manthano, Superannotate, Segments.ai
Other notes	Used for image, video, and text annotation	Can also work on your own tool

It's important to note that both companies adhere to a humans-in-the-loop approach, which means that behind every annotation there is a human. Even though both use various tools for annotation, it's a human annotator who processes data, annotates, and validates it.

Integrations

Both iMerit and Humans in the Loop offer no-code integrations. The final annotated data can be deployed to you through APIs. While iMerit technology also considers working with plugins and proceeds with custom requests, Humans in the Loop offers you the possibility to indicate the URLs where you want to receive final annotated datasets.

Annotation Process

iMerit Annotation Process

iMerit's annotation process is comprehensive and consistent, where you can participate at any stage. The main annotation steps include:

Consultation with an expert and registration on the platform
Trial and annotators’ training
Workflow customization
Feedback cycle
Evaluation

iMerit is one of a few AI data labeling companies that offers a trial with their Ango Hub platform.

Humans in the Loop Annotation Process

If you choose Humans in the Loop for your ML journey, the annotation with them will go through the following steps:

Introduction call
Free trial
Fine-tuning
Contract conclusion
Project delivery

Quality Assurance

iMerit QA

iMerit uses multiple layers of quality assurance along the whole data annotation process. To better notice the gaps, identify the efficiency of the accomplishment, and track KPI metrics, they use various dashboards, reports, and tracking systems. Their techniques for ensuring data annotation quality include:

Setting a gold standard during mini-sets
Using an annotator consensus
Using scientific methods for label consistency
Subsampling

Around 5-10% of random annotated data is additionally checked by a solution architect. Besides, data annotators review the labeled datasets multiple times before the final submission of the project.

Humans in the Loop QA

As the majority of data annotation companies, Humans in the Loop checks the annotated datasets multiple times. Their two levels of QA consist of:

Supervisor dedicated to every project
QA team

While the supervisor checks every piece of the annotated data and checks it against previously defined acceptance rates, the QA team checks the whole project at the end before sending it to the client.

Security and Data Compliance

Feature	iMerit	Humans in the Loop
Data Encryption	-	All datasets are encrypted at rest. All applications encrypt in transit with TLS/SSL only.
Access Controls	Security Manager sets and monitors security measures Role-based access	Role-based access Quarterly access reviews SSO and 2FA access to cloud services
Worker Screening	Security Manager trains employees	Background checks All workers sign security and NDA agreements
Compliance	SOC 2 ISO 27001 ISO 9001:2015 GDPR TISAX	GDPR Data Protection Policy, in accordance with the Regulation (EU) 2016/679 SOC 2

iMerit vs. Humans in the Loop: Pros and Cons

To let you make your own decision whether the two data annotation providers fall under your top data labeling companies, here is the main summary:

Depending on the nature and size of your project, both iMerits' and Humans in the Loop AI solutions may be helpful:

Aspect	iMerit Pros	iMerit Cons	Humans in the Loop Pros	Humans in the Loop Cons
Services	Fuller range of services for images, videos, audio, and texts	The majority of services are provided in English. Other languages are possible on demand only, and not all	In addition to data annotation, there are also data collection, testing, and training services for ML models	Focus on computer vision
Tools	Own tool, which influences also security	Subscription-based tool	Possibility to manage multiple formats and datasets	Annotating with third-party and open-source tools
QA	More techniques for QA	Time-consuming QA processes	Quick turnaround of project tasks	Less stringent QA and potential errors

TL;DR

To summarize, both iMerit and Humans in the Loop offer competitive solutions. If the first vendor may be interesting to those who consider comprehensive annotation, the second one may suffice to those focusing on computer vision projects.

iMerit may be suitable to those who want to have extensive cooperation with regular subscription to their platform. Humans in the Loop is a good option for smaller projects that consider annotation, but also testing and training.

And if you're looking for someone who has experience with other industries (e.g. insurance, academia) or provides annotation in multiple languages across the globe, try a free pilot with us.

FAQ

For data annotation, which company offers a better combination of experience and cost: iMerit or Humans in the Loop?

Both iMerit and Humans in the Loop have extensive experience across various industries. They work with different dataset types and have a tailored approach to every client. While Humans in the Loop have had experience with geospatial, automotive, medical, retail industries, to name just a few, iMerit additionally covers commerce and finances. Both have individual pricing per project, but iMerit also considers subscriptions for their Ango Hub platform.

How do iMerit and Humans in the Loop compare in terms of data labeling accuracy and turnaround times?

iMerit has an elaborated QA that consists of multiple quality check techniques. This ensures the final annotated dataset is checked several times and has a high level of accuracy. Humans in the Loop, in their turn, have dedicated specialists and teams that check the annotated data after every task and at the end of the project.

If my project requires a significant increase in labeling volume, which company, iMerit or Humans in the Loop, can better scale their workforce to meet my needs?

iMerit counts over 5000 specialists, while Humans in the Loop have only around 250. This means that iMerit has more capacity for scaling and considering high-volume projects. They also offer discounts, creating attractive conditions. At the same time, Humans in the Loop are also ready to undertake express requests, although their price may increase.

Which company offers more responsive customer support?

Humans in the Loop assign you a project coordinator who serves as an intermediary between you and the annotators. Not only does he take your requests, but also ensures the quality of annotation. Besides, you can intervene at any stage and ask for additional reports from the team. iMerit also has a responsive team. However, although you can also comment and respond on various annotation stages, you communicate through the platform, which may delay the communication time.

by Maria Shepeta
on June 6, 2024.