Breaking Language Barriers, Unleashing Visual Understanding Technology
Have you ever imagined that artificial intelligence would one day be able to recognize the content of images and videos? At present, Azure GPT-4 Turbo with Vision has achieved this powerful capability and is offering a preview version for users to try out. You can now ask artificial intelligence questions about images, and it will identify the images and provide appropriate responses in natural language format. It's hard to imagine, isn't it? The GPT-4 Turbo with Vision model breaks through the limitations of language models that could only accept text inputs in the past. Now, it can accept image inputs, understand the meaning and context of images, provide rich image descriptions, identify objects, extract text from images for data translation, and more. Let's take a closer look at how this can be utilized.
Evolving Azure GPT-4 Turbo with Vision
- Video Prompt Feature
With native integration with Azure AI Vision Video Retrieval, videos can now serve as inputs for GPT-4 Turbo with Vision, allowing the model to understand the context of videos and generate summaries of the content. For reference, you can check out the official Microsoft video examples.
- Azure OpenAI on your data with images
By combining GPT-4 Turbo with Vision with Azure AI Search and Azure AI Vision, new possibilities are created for data retrieval. Users can add images to textual data, and when vector search functionality is set up, such image data can be linked. For example, a company's outdoor equipment website's chatbot, using Azure OpenAI technology, adds image data to text, allowing consumers using the chatbot to directly ask questions using images with accompanying text. The chatbot can respond appropriately. You can refer to the official Microsoft video examples for more details.
- Objects Grounding
Azure AI Vision combined with GPT-4 Turbo with Vision primarily focuses on visual aspects, highlighting objects in input images and taking the integration of image data to the next level. For example, if a user inputs a portrait image and asks what fashion accessories are needed to recreate the look, the model can identify prominent objects and list descriptions of the required fashion accessories, as shown in the image.
Object Grounding
（圖片取自 Microsoft 微軟新聞中心：https://news.microsoft.com/zh-tw/features/gpt-4-turbo-with-vision/）
- Optical Character Recognition（OCR）
Azure AI Vision assists GPT-4 Turbo with Vision in OCR, allowing dense text inputs to be converted. This enables integration with financial documents. For example, if a user inputs several receipt images and requests specific data extraction, the model can transcribe the text from the images into data and present a clear summary of the data, as shown in the image.
Optical Character Recognition（OCR）
（圖片取自 Microsoft 微軟新聞中心：https://news.microsoft.com/zh-tw/features/gpt-4-turbo-with-vision/）
Responsible Principles Ensure Privacy Safety
We all know that when using the GPT-4 Turbo with Vision model, sometimes images containing faces may be uploaded. Based on privacy protection principles, GPT-4 Turbo with Vision blurs faces in input images and then uses other image clues to identify and respond to user requests. You might wonder how the model makes these determinations. During the learning phase, GPT-4 Turbo with Vision matches specific images with their corresponding names, which is why even if a user inputs a photo of a sports star and asks about their identity, the model can respond correctly, even with blurred faces.
Source：
Microsoft 微軟新聞中心 - 〈GPT-4 Turbo with Vision 現已於Azure OpenAI Service 上公開預覽，開放使用〉https://news.microsoft.com/zh-tw/features/gpt-4-turbo-with-vision/