Researchers at the University of Pennsylvania and the Allen Institute for Artificial Intelligence have developed a groundbreaking tool that allows open-source AI systems to match or surpass the visual understanding capabilities of proprietary models like GPT-4V and Gemini 1.5 Flash, potentially reshaping the competitive landscape between open and closed AI development.
The tool, called CoSyn (Code-Guided Synthesis), addresses a critical bottleneck in AI development: the scarcity of high-quality training data for teaching machines to understand complex visual information like scientific charts, medical diagrams, and financial documents. Rather than scraping millions of images from the internet — a practice fraught with copyright and ethical concerns — CoSyn leverages the coding abilities of existing language models to generate synthetic training data.
“We have, we lack of such data to train the model. We lack of data, like documents, charts with rich annotations to train a vision language model to do question answering over those images,” explained Yue Yang, a recent Penn Engineering Ph.D. graduate and co-first author of the research, during an exclusive interview with VentureBeat. “Those images actually are more challenging to annotate, compared to natural photos, like a picture of a dog of a cat of a house.”
The breakthrough comes as enterprises increasingly seek AI systems capable of understanding and reasoning about complex visual information — capabilities essential for everything from automated document processing to AI agents that can navigate digital interfaces independently. The work was conducted during Yang’s internship with the PRIOR team at the Allen Institute for AI and supported by the Office of the Director of National Intelligence, Intelligence Advanced Research Projects Activity, and the Defense Advanced Research Projects Agency.
How synthetic data generation solves AI’s biggest training challenge
The challenge of training AI to understand text-rich images has long plagued the field. Unlike natural photographs, scientific figures, charts, and d …