Language
Country/Area
No result found
From pixels to emotions: How does image description technology help us understand happy scenes?
With the advancement of new communication technologies and the widespread application of the Internet in society, audio and video information in digital formats continues to increase. This motivates us to design systems to describe various types of multimedia content in order to facilitate the search and classification of the required information. As a key part of this system, image description technology can effectively improve the search efficiency of audio and video content, and assume the main description task of audio and video files in the context of growing user needs.
Visual descriptors or image descriptors can be thought of as intuitive interpretations of digital image and animation content. They cover basic characteristics such as color, shape, texture and motion.
Standardized systems of these descriptors, such as MPEG-7 (Moving Picture Experts Group-7), are designed for in-depth description of audio and video content. Compared with search engines for textual content, it can be imagined that it is more difficult to search for visual content. For example, if you want to search for a happy person, the emotion of happiness is not directly expressed through shapes, colors, or textures.
Type of image descriptor
Image descriptors are mainly classified into two categories: general information descriptors and domain-specific information descriptors.
General information descriptor
This part includes low-level descriptors, which mainly provide descriptions of basic features such as color, shape, texture, motion and position.
Color
Color is one of the most fundamental characteristics of visual content. Tools for describing color include:
- Dominant Color Descriptor (DCD)
- Extensible Color Descriptor (SCD)
- Color Structure Descriptor (CSD)
- Color Layout Descriptor (CLD)
- Group of Frames (GoF) or Group of Pictures (GoP)
Texture
Texture is crucial in image description. It can describe the regional characteristics of the image. This set of descriptors includes:
- Uniform Texture Descriptor (HTD)
- Texture Browse Descriptor (TBD)
- Edge Histogram Descriptor (EHD)
Shape
Shapes carry important semantic information because humans are able to recognize objects by their shapes. These descriptors can describe the regions, contours and shapes of 2D images or 3D volumes:
- Region Based Shape Descriptor (RSD)
- Contours based on shape descriptors (CSD)
- Three-dimensional shape descriptor (3-D SD)
Sports
Motion is usually defined through four descriptors, including information related to object movement and camera movement:
- Motor Activity Descriptor (MAD)
- Camera Motion Descriptor (CMD)
- Motion Trajectory Descriptor (MTD)
- Deformation and parametric motion descriptors (WMD and PMD)
Location
The position of elements in the image is used to describe the distribution of elements in space and time:
- Region Locator Descriptor (RLD)
- Spatiotemporal Location Descriptor (STLD)
Field-specific information descriptor
These descriptors focus on providing information about objects and events in the scene, and are often not easily extracted automatically, but can be supplemented by manual processing. Facial recognition is a specific example of this type of application.
Application of descriptors
Image descriptors have a wide range of applications, including: multimedia file search engines and classifiers, digital libraries, personalized news services, and monitoring and filtering of audio and video content, etc. For example, image descriptors allow users to precisely search for videos with specific content, such as quickly finding movies in which a certain actor appears.
In the future, how may technology change the way we understand and respond emotionally to image content?
From happy pictures to a deep understanding of people’s emotions, image description technology is gradually leading us to explore the rich connections between emotion and vision. Considering the future of audio and video interaction, image description technology will allow us to experience image content more accurately and richly. Perhaps deeper human emotional communication will be realized in the near future?
