Over the past year or so, Google has been previewing a lot of Gemini-branded and other AI features across its consumer apps. Here’s a look at all the features announced and when they’ll be available.
pixel
At the end of “Made by Google 2023,” Google teased a zoom enhancement for the Pixel 8 Pro that “intelligently fills gaps between pixels and predicts fine details,” leveraging on-device “custom generative AI image models” to help when you forget to zoom.
It’s an incredible application of generative AI, and opens up a whole host of possibilities for framing and editing your images — essentially putting the zoom enhancements you’ve seen in sci-fi right in the phone’s hands.
Google said in October that the feature would be “coming at a later date.” It’s been three Pixel Feature Drops and it still hasn’t arrived. It’s unclear if Google is referring to the Gemini Nano with multi-modality. For now, it could debut as a flagship photography feature on the Pixel 9 Pro.
Google Home
The Google Home app will use generative AI to condense events into “a concise view of what’s happened recently.” This “quick and easy summary” uses bullet points, and you can also conversationally “Ask about your home” to find video history clips and automate things. This “experimental feature” will be available to Nest Aware subscribers in 2024.
Fitbit
Fitbit Labs plans to allow Fitbit Premium users to test experimental AI features and provide feedback.
One such feature is a chatbot that lets you ask questions about your Fitbit data in a natural, conversational way. Taking your fitness goals into account, this “personalized coaching” aims to generate “actionable messaging and guidance,” and responses may even include custom charts.
- “For example, we can look more closely at the correlation between how many Active Zone Minutes (AZM) you have and how restorative your sleep is.”
- “…the model could analyze changes in sleep patterns and quality and potentially provide recommendations on how to change training intensity based on those insights.”
Behind the scenes, it’s being built on Gemini by Fitbit and Google Research’s new Personal Health LLM, which as of March was expected to be released “later this year” to a limited number of Android users who are signed up for the Fitbit Labs program in the Fitbit mobile app.
Google Photos
Ask Photos lets you ask questions about images and videos in your library. It not only searches photos, but also pulls information and gives you a text response. Example queries for this Gemini-powered app include “Show me the best photos of the national parks you’ve visited” and “What was the theme of Lena’s birthday party?” The app can be used to “suggest top photos” and write captions. Ask Photos is an “experimental feature” that will be rolling out soon, but Google has already teased that it will add more features in the future.
Gmail + Google Workspace
Gmail for Android and iOS will have a Gemini button in the top right corner that will bring up the mobile equivalent of the side panel to fill out the full prompt. Gmail will also be introducing Contextual Smart Replies, which will provide more personalized, detailed and nuanced suggestions, and will be rolling out to Workspace Labs in July.
At Cloud Next 2024 in April, Google also previewed a Help me write voice prompt feature in mobile Gmail, while an “Instant Polish” feature will “convert your draft into a full email in one click.”
On desktop web, the side panel is available in Gmail, Google Drive, Docs/Sheets/Slides, and next, Gemini is coming to Google Chat to summarize conversations and answer questions.
Google Maps
In February, Google announced that Maps would use LLM to power its Ask chatbot, which lets you search for places that match a prompt and also supports follow-up questions. The chatbot is powered by detailed information about 250 million places, as well as user-submitted photos, videos, and reviews.
chromium
Gemini Nano will be coming to desktop Chrome, powering browser features like “Help me writing,” and will be available on most modern laptops and desktops.
Google Search
In addition to releasing the AI Overview, Google previewed several upcoming features that will be coming to Search Labs for the first time.
- The original AI summary can now be made “simpler” (a few sentences) or “more granular” (a longer response).
- The multi-step inference feature allows you to ask complex questions all at once, rather than breaking them down into multiple queries.
- Food and travel planning
- AI-organized search results page
- Video Search: Record a video and ask a question about it
Android
Gemini Nano with multi-modality capabilities will be released on the Pixel “later this year” and will bring enhanced features like in-device/offline TalkBack explanations and real-time fraud warnings that listen for distinctive patterns in calls. Google plans to announce more details later this year.
At I/O 2024, Google also previewed that Gemini for Android will be an overlay panel instead of opening a full-screen UI to display results. Not only will the context be preserved, but you’ll be able to drag and drop generated images into conversations. For Gemini Advanced subscribers, the “Ask this video” and “Ask this PDF” buttons will bring up Gemini digest videos and documents, respectively. This is expected to roll out “in the coming months.” Additionally, Dynamic Suggest will use Gemini Nano with its multi-modal capabilities to understand what you’re seeing on the screen.
For example, if you launch Gemini in a conversation where you’re talking about pickleball, you might see suggestions like “Find a pickleball club near you” or “Pickleball rules for beginners.”
Another addition that’s especially useful on mobile is the Gemini extension for Google Calendar, Tasks and Keep, which lets you take a photo of a page with multiple upcoming dates and turn it into a calendar event in Gemini. In the coming months, a “utility” will let you access Android’s clock app from mobile Gemini.
We’re also looking forward to Mobile Gemini coming to Pixel tablets this summer.
Gemini
Live lets you have a two-way conversation with Gemini. To create a more natural experience, Gemini responds with brief responses, allowing users to interrupt to add new information or ask for clarification. With 10 voices to choose from, Google believes Gemini Live will be useful for preparing for an interview or rehearsing a speech. It will be available to Gemini Advanced members in the “coming months.”
“Later this year,” Gemini Live will let you launch a live camera mode where you can simply point at something in the real world and ask questions about it. This is powered by Project Astra.
Gems are customised versions of Gemini that let you have things like “gym buddies, sous chefs, coding partners and creative writing guides.” Gemini Advanced members can create custom versions, but all users have access to pre-made Gems such as the Learning Coach.
Simply describe what you want your Gem to do and how you want it to respond, for example, “You are my running coach. Give me my daily running plan and keep me positive, upbeat and motivated.” Gemini takes these instructions and reinforces them with one click to create a Gem that meets your specific needs.
Gemini Advanced users are also provided with an “immersive planner” that not only suggests activities but actually creates a detailed itinerary taking into account travel times, stops and the user’s interests. Gemini uses flight/trip details from Gmail, Google Maps recommendations for dining and museums near the hotel, and searches for other activities.
FTC: We use automated affiliate links that generate revenue. more.