Deep North, which uses AI to track people from camera footage, raises $16.7M

Correction Note: An earlier version of this article stated that Deep North uses biometric data. However, biometric data is not used as part of Deep North’s solution. Further, Deep North’s solution for the SVIP initiative is still in development and is not currently tested in any airports. Additionally, Deep North has no project or contract with the Detroit Metropolitan Wayne County Airport. Originally, the story stated that the company’s software can understand emotions, but Deep North does not have the ability to understand or track emotions. The original story stated facial recognition capabilities from this company, but this is not part of Deep North’s offering or capabilities.

We regret these errors.

Deep North, a Foster City, California-based startup applying computer vision to security camera footage, today announced that it raised $16.7 million in a series A-1 round. Led by Celesta Capital and Yobi Partners, with participation from Conviction Investment Partners, Deep North plans to use the funds to make hires and expand its services “at scale,” according to CEO Rohan Sanil.

Deep North, previously known as Vmaxx, claims its platform can help brick-and-mortar retailers “embrace digital” and protect against COVID-19 by retrofitting security systems to track purchases and ensure compliance with masking rules. But the company’s system, which relies on algorithms with potential flaws, raises concerns about both privacy and bias.

“Even before a global pandemic forced retailers to close their doors … businesses were struggling to compete with a rapidly growing online consumer base,” Sanil said in a statement. “As stores open again, retailers must embrace creative digital solutions with data-driven, outcome-based computer vision and AI solutions, to better compete with online retailers and, at the same time, accommodate COVID-safe practices.”

AI-powered monitoring

Deep North was founded in 2016 by Sanil and Jinjun Wang, an expert in multimedia signal processing, pattern recognition, computer vision, and analytics. Wang — now a professor at Xi’an Jiaotong University in Xi’an, China — was previously a research scientist at NEC before joining Epson’s R&D division as a member of the senior technical staff. Sanil founded a number of companies prior to Deep North, including Akirra Media Systems, where Wang was once employed as a research scientist.

“In 2016, I pioneered object detection technology to help drive targeted advertising from online videos. When a major brand saw this, they challenged me to create a means of identifying, analyzing, and sorting objects captured on their security video cameras in their theme parks,” Sanil told VentureBeat via email. “My exploration inspired development that would unlock the potential of installed CCTV and security video cameras within the customer’s physical environment and apply object detection and analysis in any form of video.”

After opening offices in China and Sweden and rebranding in 2018, Deep North expanded the availability of its computer vision and video analytics products, which offer object and people detection capabilities. The company says its real-time, AI-powered and hardware-agnostic software can understand customers’ preferences, actions, interactions, and reactions “in virtually any physical setting” across “a variety of markets,” including retailers, grocers, airports, drive-thrus, shopping malls, restaurants, and events.

Deep North says that retailers, malls, and restaurants, in particular, can use its solution to analyze customer “hotspots,” seating, occupancy, dwell times, gaze direction, and wait times, leveraging these insights to figure out where to assign store associates or kitchen staff. Stores can predict conversion by correlating tracking data with the time of day, location, marketing events, weather, and more, while shopping centers can draw on tenant statistics to understand trends and identify “synergies” between tenants, optimizing for store placement and cross-tenant promotions.

“Our algorithms are trained to detect objects in motion and generate rich metadata about physical environments such as engagement, pathing, and dwelling. Our inference pipeline brings together camera feeds and algorithms for real-time processing,” Deep North explains on its website. “[We] can deploy both via cloud and on-premise and go live within a matter of hours. Our scalable GPU edge appliance enables businesses to bring data processing directly to their environments and convert their property into a digital AI property. Video assets never leave the premise, ensuring the highest level of security and privacy.”

Beyond these solutions, Deep North developed products for particular use cases like social distancing and sanitation. The company offers products that monitor for hand-washing and estimate wait times at airport check-in counters, for example, as well as detect the presence of masks and track the status of maintenance workers on tarmacs.

“With Deep North’s mask detection capability, retailers can easily monitor large crowds and receive real-time alerts,” Deep North explains about its social distancing products. “In addition, Deep North … monitors schedules and coverage of sanitization measures as well as the total time taken for each cleaning activity … Using Deep North’s extensive data, [malls can] create tenant compliance scorecards to benchmark efforts, track overall progress, course-correct as necessary. [They] can also ensure occupancy limits are adhered to across several properties, both locally and region-wide, by monitoring real-time occupancy on our dashboard and mobile apps.”

Bias concerns

Like most computer vision systems, Deep North’s were trained on datasets of images and videos showing examples of people, places, and things. Poor representation within these datasets can result in harm — particularly given that the AI field generally lacks clear descriptions of bias.

Previous research has found that ImageNet and Open Images — two large, publicly available image datasets — are U.S.- and Euro-centric, encoding humanlike biases about race, ethnicity, gender, weight, and more. Models trained on these datasets perform worse on images from Global South countries. For example, images of grooms are classified with lower accuracy when they come from Ethiopia and Pakistan, compared to images of grooms from the United States. And because of how images of words like “wedding” or “spices” are presented in distinctly different cultures, object recognition systems can fail to classify many of these objects when they come from the Global South.

Bias can arise from other sources, like differences in the sun path between the northern and southern hemispheres and variations in background scenery. Studies show that even differences between camera models — e.g., resolution and aspect ratio — can cause an algorithm to be less effective in classifying the objects it was trained to detect.

Tech companies have historically deployed flawed models into production. ST Technologies’ facial recognition and weapon-detecting platform was found to misidentify black children at a higher rate and frequently mistook broom handles for guns. Meanwhile, Walmart’s AI- and camera-based anti-shoplifting technology, which is provided by Everseen, came under scrutiny last May over its reportedly poor detection rates.

Deep North doesn’t disclose on its website how it trained its computer vision algorithms, including whether it used synthetic data (which has its own flaws) to supplement real-world datasets. The company also declines to say to what extent it takes into account accessibility and users with major mobility issues.

In an email, Sanil claimed that Deep North “has one of the largest training datasets in the world,” derived from real-world deployments and scenarios. “Our human object detection and analysis algorithms have been trained with more than 130 million detections, thousands of camera feeds, and various environmental conditions while providing accurate insights for our customers,” he said. “Our automated and semi-supervised training methodology helps us build new machine learning models rapidly, with the least amount of training data and human intervention.”

In a follow-up email, Sanil added: “Our platform detects humans, including those with unique gaits, and those that use mobility aids and assistive devices. We don’t do any biometric analysis, and therefore there is no resulting bias in our system … In the simplest terms, the platform interprets everything as an object whether it’s a human or a shopping cart or a vehicle. We provide object counts entering or exiting a location. Our object counting and reporting is not influenced by specific characteristics.” He continued: “We have a large set of labeled data. For new data to be labeled, we need to classify some of the unlabeled data using the labeled information set. With the semi-supervised process we can now expedite the labeling process for new datasets. This saves time and cost for us. We don’t need annotators, or expensive and slow processes.”

Privacy and controversy

While the purported goal of products like Deep North’s are health, safety, and analytics, the technology could be co-opted for other, less humanitarian intents. Many privacy experts worry that they’ll normalize greater levels of surveillance, capturing data about workers’ movements and allowing managers to chastise employees in the name of productivity.

Deep North is no stranger to controversy, having reportedly worked with school districts and universities in Texas, Florida, Massachusetts, and California to pilot a security system that uses AI and cameras to detect threats. Deep North claims that the system, which it has since discontinued, worked with cameras with resolutions as low as 320p and could interpret people’s behavior while identifying objects like unattended bags and potential weapons.

Deep North is also testing systems in partnership with the U.S. Transportation Security Administration, which furnished it with a grant last March. The company received close to $200,000 in funding to provide metrics like passenger throughput, social distancing compliance, agent interactions, and bottleneck zones as well as reporting of unattended baggage, movement in the wrong direction, or occupying restricted areas.

“We are humbled and excited to be able to apply our innovations to help TSA realize its vision of improving passenger experience and safety throughout the airport,” Sanil said in a statement. “We are committed to providing the U.S. Department of Homeland Security and other government entities with the best AI technologies to build a safer and better homeland through continued investment and innovation.”

Deep North admitted in an interview with Swedish publication Breakit that it offers facial characterization services to some customers to estimate age range. And on its website, the startup touts its technologies’ ability to personalize marketing materials depending on a person’s demographics, like gender. But Deep North is adamant that its internal protections prevent it from ascertaining the identity of any person captured via on-camera footage.

“We have no capability to link the metadata to any single individual. Further, Deep North does not capture personally identifiable information (PII) and was developed to govern and preserve the integrity of each and every individual by the highest possible standards of anonymization,” Sanil told TechCrunch in March 2020. “Deep North does not retain any PII whatsoever, and only stores derived metadata that produces metrics such as number of entries, number of exits, etc. Deep North strives to stay compliant with all existing privacy policies including GDPR and the California Consumer Privacy Act.”

To date, 47-employee Deep North has raised $42.3 million in venture capital.