Automotive companies seeking to improve features such as vehicle autonomy, safety and advanced driver-assistance systems require sizable quantities of high-quality annotated data that contains both common and edge-case scenarios. Automated annotations help immensely in the scaling and efficiency of such large data labeling projects. However, to ensure the training data is of high quality, automotive builders and original equipment manufacturers rely on the skills of expert annotators to incorporate details and nuances into the complex data annotation scenarios.
The challenge
Our client, a U.S.-based luxury electric car manufacturer renowned for their innovative design and exceptional range, faced a critical annotation challenge. They required high-volume datasets consisting of 2D camera and 3D lidar data captured by their vehicles in diverse real-world conditions to be annotated for perception model building.
The annotation process was highly demanding, requiring precise labeling of longer driving sequences with lane markings and 2D–3D object linking. This necessitated a highly-skilled and experienced workforce capable of accurately identifying and annotating relevant objects and features in various scenarios, including driving at nighttime and in adverse weather conditions that make object identification more difficult.
Managing the large annotation project from end-to-end, including workforce logistics, proved challenging for the client and consumed significant internal resources. As a result, they turned to TELUS Digital for our proven expertise in 2D and 3D annotations by our trained in-house annotators.
The TELUS Digital solution
In order to meet our client's needs, we implemented a comprehensive project-planning workflow that included the following components:
Expert workforce management and proactive training
We deployed a team of 63 highly-skilled annotators and quality-control experts to work on the client’s proprietary platform. A long-term client, they relied on our annotators not only for their expertise, but also as a power user group to evaluate and enhance their internal tools, offering continuous, actionable feedback.
To ensure the annotators understood the project's guidelines, we conducted comprehensive training sessions, using quizzes to reinforce learning. Based on quiz results and task performance, we also provided targeted training to address gaps and enhance overall skills. We further enhanced task approval rates by implementing iterative feedback cycles in collaboration with the client.
Project management and operations streamlining
TELUS Digital overtook key aspects of the project’s management, including job creation, ticketing, task assignment and scheduling. By synchronizing task input batch timelines with job composition, we optimized resource use and task-flow management. We also implemented an always-on performance monitoring system. Contributors were evaluated on the following three key parameters:
- Availability: Ensures consistent accessibility and responsiveness to meet project needs
- Productivity: Demonstrates efficient output, meeting objectives within set timelines
- Quality: Delivers work that consistently meets or exceeds established standards
Regularly assessing the annotators allowed us to identify gaps and areas for improvement, resulting in an increasingly agile turnaround time and improved work quality across pipelines. With user-specific statistics, we provided tailored feedback to contributors that optimized overall performance. We also designed incentive programs to increase productivity and motivate the annotators, including weekly recognition and rewards for top performers.
Quality assurance measures
The client relied on us to build a quality rubric that aligned with the project’s requirements. Feedback from prior projects helped us shape a comprehensive reference document that became an essential resource for the team. Our quality process included the following:
- Regular shadowing sessions with the client's quality team to ensure alignment
- Internal feedback loops with our in-house quality experts to refine outputs further
- Periodic guideline updates to ensure consistency in attributes and handling of new edge cases
Overall, this helped us achieve more than 98% accuracy across all pipelines.
The results
We successfully delivered precise 2D and 3D annotations, including detailed object identification, lane markings and other critical features necessary for training and validating our client's perception models. Our annotators were located in a time zone more than 10 hours ahead of the client, which provided an advantage to streamline the project flow, ensuring a continuous workflow and maximized output, clocking over 30,000 work hours at 99% workforce efficiency. Additionally, we implemented our strict in-facility security and privacy protocols to safeguard the client’s proprietary data, following industry best practices and privacy regulations.
By effectively managing workforce sourcing, training and project operations, TELUS Digital saved considerable business bandwidth for the client, allowing them to focus on their core tech developments. Additionally, our ability to ramp up workforce support as required helped us become a single-vendor workforce provider supporting their future model development roadmap.
