Apple is currently levelling up Siris onscreen awareness, enabling it to interact with your screen

Date:
Tue, 05 Nov 2024 19:57:27 +0000

Description:
Apple's new onscreen awareness feature could make Siri more intuitive,
helping it interact with whats on your screen for faster, smarter responses.

FULL STORY ======================================================================Apple
is developing an "onscreen awareness" feature to allow Siri to understand and interact with the content currently displayed on your screen Apple will also provide APIs to developers for integrating onscreen awareness into their third-party apps and is currently testing ChatGPT integration, allowing Siri to answer questions based on screenshots While not available in the iOS 18.2 beta, "onscreen awareness" may arrive in time for iOS 18.4 in 2025

Among the digital assistants, Siri has fared fairly well (certainly compared to Cortana, the ill-fated assistant from rival Microsoft), and now Apple is working on making Siri even smarter by giving it a better sense of what youre looking at on your screen, calling it onscreen awareness.

Apple has gone into detail about the development of this feature in an Apple Developer Documentation page , which also notes that its due to be included
in various upcoming Apple operating system (OS) beta releases for testing.

Apple originally showcased onscreen intelligence in June 2024 and this is a pretty solid indication that its still in development.

The core idea of onscreen awareness is pretty straightforward if youre looking at items on your screen, say a document or a browser with a page
open, and you have a question about something youre looking at, you can ask Siri (equipped with Apple Intelligence ). Siri should then be able to respond to your question with relevant information or perform an action asked of it, like sending content to a supported third-party app.

If it works as intended (and thats a big if), it will result in a smarter
Siri that wont need you to describe what you want it to do as extensively as you probably need to right now. For example, you can have a document open and ask for a summary without entering the contents of the document yourself. Apple's plans for Siri's foreseeable future

MacRumors reports that Apple has provided APIs for developers, enabling them to make the content in their apps available to Siri and Apple Intelligence. The idea is to provide developers with this API many months in advance of its release so that you can use onscreen awareness with third-party apps when its officially rolled out.

Currently, we know Apple is trialing out ChatGPT in the latest iOS 18.2 beta version (among other Apple OSs), and ChatGPT integration coupled with Siri will enable you to ask questions about items (like images, PDFs, and videos) on screen. Then, Siri will take a screenshot and pass it on to ChatGPT to reply to your query. This means that, for now, this functionality is limited to screenshots of your screen.

However, onscreen awareness is a little bit different, as pointed out by MacRumors, as onscreen awareness is intended to be integrated in a more
direct way.

Siris onscreen awareness is supposed so be a sort of capacity to directly analyze, interpret, and interact with the content on your screen. If someone messages you their number, and youd like to save it, you can tell Siri to create a new contact without having to add additional instructions or Siri having to perform many intermediate steps. Will Siri survive and even thrive in the AI age?

Apparently, onscreen awareness isnt actually in the iOS 18.2 developer beta just yet, and MacRumors speculates that its one of multiple Siri features
that we wont see for a while, but this news is still promising. One
prediction has onscreen awareness to possibly be included in iOS 18.4, expected to be released in the first half of 2025.

If this pans out, Siri could become a more helpful digital assistant, and
with Apples flair for design, it might become the choice digital assistant
for many. This development reminds me of what Microsoft is aiming for with Copilot . That hasnt been received very well from what weve seen so far, leaving the goal open for Apple. YOU MIGHT ALSO LIKE... Your Macs menu bar will finally get a weather widget in macOS Sequoia 15.2 plus these Apple Intelligence features iOS 18.2 beta shows exactly how Siris ChatGPT upgrade will work on iPhones including new daily limit guide Microsoft could be
about to rebrand its AI into Windows Intelligence rather than Copilot



======================================================================
Link to news story: https://www.techradar.com/computing/artificial-intelligence/apple-is-currently -levelling-u-siris-onscreen-awareness-enabling-it-to-interact-with-your-screen


--- Mystic BBS v1.12 A47 (Linux/64)
* Origin: tqwNet Technology News (1337:1/100)