Multimodal LLMs Enable Scalable, Structured Analysis of Real-World Smartphone Screen Behavior
The Media Content Atlas (MCA) is a pipeline that applies multimodal LLMs to large-scale smartphone screenshot data, enabling content-level analysis far beyond traditional screen-time metrics like app usage or duration. Evaluated on 1.12 million screenshots from 112 adults over one month, MCA supports moment-by-moment content analysis, clustering, topic modeling, and image retrieval at a scale previously infeasible. The system bridges inductive (exploratory) and deductive (hypothesis-driven) research workflows, with expert evaluators rating clustering relevance at 96% and content descriptions at 83% accuracy — suggesting MLLM-based pipelines are now viable for behavioral media research.