In today’s increasingly AI-driven landscape, one essential habit to cultivate is taking numerous screenshots. This simple act of capturing digital content can be incredibly valuable, especially amidst advancements in voice recognition, camera technology, and other multimodal interfaces.
Screenshots have emerged as a universal means of saving digital information. Nearly anything can be captured with just a few clicks—except, arguably, content protected against screenshots, like that from Netflix. According to Johnny Bree, founder of the digital storage app Fabric, screenshots serve as a highly portable format that can be shared across various platforms and devices.
A screenshot is rich with information, including its source, contents, and even timestamps. More importantly, it conveys a meaningful message: I care about this. As numerous new AI tools analyze our activities and interactions, many struggle with effectively discerning what truly matters. By taking a screenshot, users essentially assign significance to that content, signaling to the system that it warrants attention.
Furthermore, screenshots empower users by putting them in control of their information. Mattias Deserti, head of smartphone marketing at Nothing, points out the overwhelming noise created by unrestricted access to emails and messages. Instead of allowing a system like Microsoft Recall to access everything, individuals can selectively share information through screenshots, honing in on what they wish to convey about themselves.
Screenshots, traditionally, have been somewhat rudimentary tools. Typically, they are saved to a camera roll where they may be forgotten and difficult to locate later on. At best, users can search for text within the images, but more commonly, they find themselves scrolling through endless files to recover what they need.
The first step in making screenshots more useful is to figure out what’s actually in them.
Enhancing the utility of screenshots begins with understanding their content. With advancements in optical character recognition (OCR) technology, extracting text from images has become easier. AI models take this further, enabling users to search by title or keywords, thereby finding specific screenshots among a myriad of digital captures. Shenaz Zack, a product manager at Google involved with the Pixel Screenshots app, explains that the process utilizes an OCR model combined with an entity-detection model and context analysis.
Beyond mere text, a sophisticated AI can discern nuances such as the app from which a screenshot originated, the specific colors associated with certain platforms, or the context of a listing from e-commerce sites. This capability enables screenshot applications to organize captured images intelligently, enhancing user experience.
However, simply creating an app for browsing screenshots might not entice users, as it could become one more chore to manage. The more intriguing potential lies in the ability of devices to leverage these images actively, assisting users in recalling captured information or taking action based on it.
For instance, Nothing’s new Essential Space app can generate reminders based on screenshots. If users capture a concert they want to attend, the app will remind them as the event approaches. The Pixel Screenshots application adds another layer of interactivity, prompting users to listen to a band after saving a concert listing. Similarly, if you capture an ID card or boarding pass, the Pixel phone may suggest adding it to a virtual wallet. According to Zack, the goal is to transform screenshots into a dynamic input mechanism for an array of functions.
Indie developer Mike Choi created an app called Camp to better utilize his own screenshots. The app transforms each screenshot into a “card” that displays relevant information alongside the image. “You have a screenshot, and at the bottom there’s a button that flips the card over,” Choi explains. “It reveals a map if it pertains to a location or a preview of a song if it’s musical.” The goal is to leverage the diverse range of screenshots and enable AI to generate the optimal user interface on-the-fly.
This concept relates to a broader trend referred to as agentic AI, where tech companies are seeking to build systems that operate autonomously on users’ behalf. Notably, in this case, users can simply capture information by taking screenshots rather than composing lengthy prompts. Deserti is enthusiastic about developing systems that would enable users to screenshot concert dates and automatically receive ticket purchase prompts when they become available.
Making sense of screenshots isn’t always so straightforward.
However, identifying what each screenshot signifies can be complex. Some screenshots, like essential ID cards, may need to be retained indefinitely, while others, such as temporary concert posters, become obsolete quickly. It raises the question of how an application can differentiate between a frequently used parking pass and one used only once. Users often accumulate screenshots from diverse sources, making it challenging for an app to categorize them effectively. Many screenshot applications aim to encourage users to add notes or organize their captures to enhance system understanding, yet doing so may complicate the simple utility that makes screenshots appealing.
To enhance the utility of screenshots further, it’s advantageous to collect additional context from devices. Companies like Google and Nothing have a unique advantage as they manufacture the devices and thus can monitor all activities occurring during screenshot capture. They can save relevant links, note the time, and even record weather data—all of which add context but can also contribute to unnecessary noise if not handled correctly.
Nonetheless, the input system is effective. Most users take screenshots regularly, using them as markers for important information. Gaining access to relevant, personalized data is crucial for building an effective AI assistant. While future technology will integrate various modalities, utilizing simple screenshots may provide one of the best initial methods for engaging AI.
