What Is Computer Vision?

Computer vision is a field of artificial intelligence (AI) that enables computers to derive meaningful information from digital images, videos, and other visual inputs. Today, computer systems have access to large volumes of image and video data from smartphones, traffic cameras, security systems, and other devices. Computer vision applications use artificial intelligence and machine learning (AI/ML) to process visual data for such tasks as object identification, optical character recognition, facial recognition, classification, recommendations, and monitoring. Modern computer vision applications include generative AI applications. These applications can understand context, answer questions about visual content, and generate new images from scratch.

Computer vision systems use AI to mimic the human brain's ability to interpret visual data. Computer scientists train computers to recognize visual data by inputting vast amounts of training data. Machine learning (ML) algorithms identify common patterns in these images or videos and apply that knowledge to accurately detect elements in new images.

Building a computer vision tool typically follows a process of data gathering, preprocessing, model selection, and model training.

Data gathering

The first step is collecting a large dataset of visual information. For example, if computers process millions of images of cars, they begin to build up pattern recognition that can help them accurately detect a vehicle in an image. Typically, each image is labeled by a human to indicate whether a vehicle is present, so that the model can learn these associations.

Preprocessing

Raw data in large datasets is rarely ready for immediate training. For example, millions of still images of cars from street cameras. Data preparation involves preprocessing. Preprocessing is the process of cleaning and enhancing images to improve their quality and for example, resizing or adjusting the contrast on them. To make sure that the model is robust, developers can also use data augmentation, which involves modifying existing images to expand the diversity of the dataset. For example, flipping or rotating them.

Model selection

You identify the best machine learning model for your computer vision task. The most commonly used models are convolutional neural networks (CNNs) and visual transformers (ViTs). All these models apply deep learning, a type of machine learning that uses neural networks. Deep learning neural networks are made of many layers of software modules called artificial neurons. They use mathematical operations to automatically process image data to gradually understand visual data.

Convolutional neural networks

Convolutional neural networks (CNNs) learn feature hierarchies to categorize visual data. They process images by applying convolutional filters that slide across the image, processing regions of pixels at a time. A CNN first identifies outlines and textures before predicting more details such as color, internal forms, and complex patterns.

Recurrent neural networks

Recurrent neural networks (RNNs) are similar to CNNs, but can retain information across timesteps in a sequence. That enables them to more effectively identify links between patterns across video frames. That is why, while CNNs are often used for single-image analysis, RNNs were historically used to analyze videos in the 2010s.

RNNs and CNNs are increasingly being replaced by transformer-based AI and multimodal large language models (LLMs), which are much more effective for processing sequential data.

Vision transformers

Instead of scanning images pixel by pixel or sequentially, vision transformers (ViTs) process the entire image at once to capture global-spatial context. For example, they can process many "cars" close together or a "highway" to recognize a "traffic jam."

Model training

During training, the computer vision algorithm performs feature extraction by analyzing labeled car images to identify key attributes such as tires, windshields, and headlights. Through repeated iterations, the model compares its prediction to the correct label. It then follows a process called backpropagation, in which it adjusts its internal parameters based on how accurate its predictions have been over time.

Security and safety

Governments and enterprises use computer vision to improve the security of their facilities and assets. For example, cameras and sensors can monitor public spaces, industrial sites, and high-security venues. They send automatic alerts if they identify something out of the ordinary occurring, such as an unauthorized individual entering a restricted area.

Computer vision can also improve home and workplace safety. For example, recognition technology can monitor many safety-related events. These include at-home real-time streams that detect pet activity and live door cameras that detect visitors or package deliveries. In the workplace, computer vision tools can help identify whether workers are wearing appropriate personal protective equipment (PPE) and generate individual warnings or reports for supervisors.

Operational efficiency

Computer vision can analyze images and extract metadata for business intelligence. This can improve existing operational efficiency and generate new revenue opportunities. For example, it can:

• Assist with quality control by automatically identifying product defects before products leave the factory

• Detect machine maintenance and safety issues

• Analyze social media images to discover previously-undetected patterns in customer behavior

• Assist with access control by authenticating employees using automatic facial recognition

Content creation

Generative AI tools can help create new visual media in addition to performing analyses. For example, in architecture and product engineering, generative AI tools with computer vision capabilities can analyze rough hand-drawn sketches or 2D floor plans and rapidly generate photorealistic 3D renderings or suggest structural variations in real time. This allows teams to visualize finished products in real-world environments before a single prototype is built.

Autonomous vehicles

Autonomous vehicle technology uses computer vision to process real-time images and construct 3D maps from multiple cameras mounted to autonomous vehicles. It can analyze images and identify other road users, road signs, pedestrians, or obstacles.

In driver monitoring systems on board vehicles, computer vision uses machine learning (ML) to monitor driver behavior. For example, it looks for signs of distraction, fatigue, and drowsiness based on the driver's head position, eye tracking, and upper body movement. If the technology detects certain warning signs, it alerts the driver and reduces the risk of a safety issue.

Agriculture

Computer vision applications are used in agriculture. Satellite imaging and UAV footage feed vision tools with data on large tracts of land. These computer vision tools can automate tasks such as monitoring field conditions, identifying crop diseases, assessing soil moisture, and predicting weather and crop yields. Animal monitoring with computer vision is another key strategy of smart farming.

Healthcare

Healthcare is one of the leading industries applying computer vision technology. Notably, medical image analysis provides visualizations of organs and tissues to help medical professionals make rapid and accurate diagnoses, resulting in improved treatment outcomes and patient quality of life. For example:

• Tumor detection by analyzing moles and skin lesions

• Automatic X-ray analysis

• Symptom discovery from MRI scans

Computer vision tools identify information within an image and do something with that information. They analyze visual data to derive meaning, such as identifying a defect or recognizing a face. The input is an image or video, and the output is data, an action, a decision, or another image.

Image processing uses algorithms to alter raw visual data, such as by sharpening details, reducing noise, or adjusting contrast. The input is an image, and the output is a modified version of that image.

In production environments, they often work together. Image processing is often used in the computer vision preprocessing step to clean visual data so the computer vision model can analyze it more accurately. Conversely, a system might use computer vision to detect a sensitive object (such as a face) and initiate processing algorithms to blur it for privacy.

Image classification

Image classification, a form of image recognition, is the process of labelling an image based on its content. Unlike object detection, which locates items within an image or video frame, image classification typically assigns a single label to the image as a whole. For example, you might use an environmental monitoring system to analyze thousands of satellite photos and tag each "Forest," "Desert," or "Urban," or other environment based on identified details.

Object detection

Object detection, a form of object recognition, is a technique that locates and identifies specific items within an image and performs object classification. For example, manufacturing lines use object detection to count specific components on a moving conveyor belt, verify assembly, and identify defective parts.

Object tracking

Object tracking uses deep learning models to identify and track items across a video, belonging to different categories. It has several real-world applications across industries. Object tracking begins with object detection. The vision tools create a bounding box around the detected item, give it an object ID, and track it across a sequence of frames. For example, you can use this for traffic monitoring in urban environments and surveillance.

Segmentation

Segmentation traces an object's precise boundary and classifies every single pixel within an image. This technique is used when knowing the exact shape is vital. For example, a self-driving car needs to know the exact curvature of a lane, or a radiologist measuring the precise volume of a tumor.

Semantic segmentation classifies pixels into broad categories. For example, all pixels belonging to "sky" are blue, and all pixels belonging to "cars" are red. It does not distinguish between two cars, it only identifies them as "car pixels." Instance segmentation identifies separate instances of the same object. It distinguishes "Car A" from "Car B."

AWS provides a broad set of artificial intelligence and machine learning (AI/ML) services for customers at all levels of expertise:

• For customers who lack ML skills, need faster time-to-market, or want to add intelligence to an existing process or application, AWS offers a range of ML-based computer vision services. These services allow you to easily add intelligence to your AI applications through pre-trained APIs. Amazon Rekognition automates your image and video analysis with ML and analyzes millions of images, live streams, and stored videos in seconds. Amazon Textract automatically extracts printed text, handwriting, layout elements, and data from any document.

• For customers who want to create a standard computer vision solution across their business, Amazon SageMaker makes it straightforward to prepare data and build, train, and deploy ML models for any use case with fully managed infrastructure, tools, and workflows, including no-code offerings for business analysts.

• For customers building on frameworks and managing their own infrastructure, AWS Neuron is a developer stack for running deep learning and generative AI workloads on AWS Trainium and AWS Inferentia chips. You can use the most popular deep learning frameworks, including PyTorch and TensorFlow. AWS provides a large portfolio of compute, networking, and storage infrastructure ML services with a choice of processors and accelerators to meet unique performance and budget needs.

Get started with computer vision by creating a free AWS account today.