Alibaba Cloud's EMR Serverless Spark now processes images and video in plain SQL, no Python needed

Alibaba Cloud added multimodal support to its E-MapReduce (EMR) Serverless Spark AI Function. Images, video frames, and binary blobs now work directly inside SQL queries. The update arrived with the esr-4.7.0 release, based on Spark 3.5.2. Data engineers can stop shuffling data between Python notebooks and ML serving endpoints. Vision inference runs as a standard SQL function alongside existing ETL work.

Three ways to feed images into SQL

The AI Function exposes two core capabilities: ai_query handles image understanding and classification; ai_embedding_multimodal generates vector embeddings from visual data. Both support three input modes:

• DLF Blob field: store binary image data directly in a table column and pass it to the function.
• URI path: reference image files stored in OSS, OSS-HDFS, or other Hadoop-compatible file systems. The function reads the file internally.
• read_files + binary: batch-read images from a directory without pre-creating a table, using the suffix parameter for file type filtering.

This solves a common pain point for teams that already warehouse image paths in structured tables but need to run inference without rewriting their pipelines.

Autonomous driving case: from manual annotation to SQL ETL

Alibaba Cloud demonstrated the capability with an autonomous driving data preprocessing scenario. Traditional dashcam analysis relies on manual annotation teams. It’s costly, slow, and error-prone on rare vehicles or complex road structures.

Using ai_query with Qwen 3.6-plus, the team built a standard ETL pipeline. It reads raw images from OSS, applies a prompt describing five visual categories (roundabout, police car, pickup truck, crane/lift truck, none of the above), and outputs structured JSON results, all in a single SQL statement.

The task completed in seconds across five high-resolution test images, correctly identifying vehicles and road features. Edge cases worked: when given an image that matched no defined category, the system returned an empty array instead of hallucinating a label.

Models and integration paths

EMR Serverless Spark ships with several built-in models callable without registration: Qwen 3.6-plus, Qwen 3.5-plus, Qwen-plus, text-embedding-v4, and tongyi-embedding-vision-plus. For enterprise work, the platform supports integration with PAI-EAS for secure VPC-deployed models and Alibaba Cloud Model Studio for rapid proof-of-concept testing.

The architecture also supports external LLMs including DeepSeek, KIMI, GLM, and MiniMax through a unified model service registration mechanism.

Vector embeddings for multimodal retrieval

The ai_embedding_multimodal function converts images into vector embeddings that can be written directly into Milvus or other vector databases. This enables cross-modal text-image search and multimodal retrieval.

Alibaba Cloud positions this update as an enabler for the autonomous driving data closed loop, from raw camera footage to structured labels, embeddings, and training-ready datasets, all within a single Serverless Spark pipeline.