3.1.1

Ship Detection – Counting as Maritime Activity Indicator

This workflow is designed to provide the user with the number of ships detected in a specified Area Of Interest (AOI) and time interval. Our detection tool is based on YOLOv5¹⁶, a fast object detection model. Our model has been fine-tuned on an annotated dataset of Sentinel-1 images containing ship locations (SSDD)¹⁷. Sentinel-1 is a radar satellite mission developed by the European Space Agency (ESA) that provides high-resolution images of the Earth’s surface. EYE-Sense can automatically detect ships in unseen Sentinel-1 images with a high degree of accuracy (97%). The detection pipeline initiates by acquiring a specific Sentinel-1 image. Next, the image is pre-processed via cropping, calibration, speckle filtering, and terrain correction. Image calibration is necessary to adjust and standardize the satellite image to remove any distortions or anomalies caused by atmospheric conditions and sensor characteristics. Speckle filtering is used to remove noise from the images and enhance the signal-to-noise ratio. Terrain correction is necessary to account for any variations in the Earth’s surface, which can affect the radar signal. The pre-processed image is then passed as input on the fine-tuned model. As an output to the user, EYE-Sense creates and displays time-series data depicting the object counts over a specified time interval.

3.1.2

Aircraft/Trucks / Containers / Umbrellas Detection - Counting

This workflow is designed to detect objects in a Very High-resolution satellite image provided by the user. The user designates the type of object they intend to detect, selecting from four available options: ships, aircrafts, umbrellas, and containers. Depending on the type, the appropriately pre-trained model is selected to perform the detection task (Table 1). Different models are employed for each object type due to performance considerations and the ease of training specialized models, resulting in enhanced efficiency and effectiveness. For the detection of ships, and aircraft, the Faster R-CNN¹⁸ model pre-trained on the DOTA¹⁹ dataset is selected. Faster R-CNN is a state-of-the-art object detection algorithm composed of two elements: 1) a region proposal network which generates candidate object locations, and 2) a classification network which labels each candidate as “object” or “background”. The DOTA dataset contains annotated satellite images with various types of objects, including vehicles, ships, and aircraft. For the detection of umbrellas, a Mask R-CNN²⁰ model is used. Mask R-CNN is an extension to Faster R-CNN which adds a segmentation network to the architecture, allowing it to predict object masks in addition to bounding boxes and class labels. Finally, for the detection of containers, a custom YOLOv5 model is used. YOLOv5 relies on a single neural network to predict both bounding boxes and class probabilities for objects in an image. The custom YOLOv5 model is trained on a dataset of annotated satellite images including containers²². Once the appropriate model has been inferred, the image is parsed to detect the selected object. The results can be provided to the user either as an image with the detected objects highlighted, or as a .txt file in COCO format, a standard format for object detection results which includes bounding boxes, class labels, and confidence scores. Assuming that a user has multiple VHR images of the same region, the described workflow can be used iteratively to construct time-series data.

Table 1:

Datasets and models used for each object’s detection.

3.1.3

Atmospheric quality analysis

The designed workflow aims to provide users with time series data pertaining to various atmospheric indicators, for a specified Area Of Interest (AOI). Users are required to input the AOI and select the desired atmospheric indicators from a list of six options: Sulfur dioxide (SO2), Carbon monoxide (CO), Nitrogen dioxide (NO2), Formaldehyde (HCHO), Aerosol Optical Depth at a wavelength of 550nm (AER_AI), and Ozone (O3). Furthermore, users are given the option to choose the frequency at which the analysis should be conducted, such as on a weekly, biweekly, or monthly basis. Utilizing the Google Earth Engine²⁴ platform, Sentinel-5p satellite data is harnessed and processed to generate the requested time-series information. The Sentinel-5p satellite mission, developed by the European Space Agency (ESA), delivers high-resolution data on atmospheric composition. EYE-Sense extracts pertinent data for the specified atmospheric indicators from the Sentinel-5p satellite dataset. Employing the mean reducer offered by the Google Earth Engine (GEE), a single value (the mean value) is derived for the area of interest concerning each atmospheric indicator chosen by the user. Subsequently, these values are employed to produce a time-series for the atmospheric indicators at the specified frequency of analysis.

3.1.4

Water quality analysis

The Water Quality Analysis workflow provides users with EO parameters that may potentially correlate with the economic activity of a specified area. To initiate the analysis, users provide an AOI and time interval. The relevant multi-band spectral images produced by the Sentinel-2 mission (providing high-quality and frequent observations of the Earth’s surface) are dynamically acquired. Specifically, we retrieve the data through the Google Earth Engine, which allows for efficient processing and analysis of extensive data volumes.

By adopting several literature prescriptions^25,26 – based on the Sentinel-2 bands – we created indicators correlated with water quality parameters, including Chlorophyll-A, Cyanobacteria, Turbidity, Dissolved Organic Carbon, and Color Dissolved Organic Matter. The details of these formulas are presented in Table 2. The Normalized Difference Water Index (NDWI) is utilized to mask land and retain only water areas. The most suitable prescription is applied to compute the indicator value for each time-step, depending on the user’s selection. Google Earth Engine’s (GEE) mean reducer is employed to obtain a single value (the mean value) for the area of interest concerning each water quality indicator chosen by the user. These values are then harnessed to produce time-series data of the water quality indicators for the specified frequency of analysis. Users can visualize the derived time-series on an interactive map or download it in a .csv format.

Table 2:

Water quality analysis information.

3.1.5

Night-light activity analysis

The Nighttime lights (NTL) Analysis workflow providing users with an economic-related parameter indicator for a designated area, is arguably the most valuable EYE-Sense tool, and it is showcased in Section 4. To acquire high-quality NTL data, we rely on Visible Infrared Imaging Radiometer Suite (VIIRS) Day/Night Band (DNB)²⁷ satellite data. Specifically, we access the VIIRS DNB database through the Google Earth Engine platform, allowing us to effectively process and analyze large amounts of data. To initiate the analysis, users first set the Area of Interest (AOI) by uploading a file in GeoJSON format. VIIRS DNB data is subsequently retrieved from the Google Earth Engine data catalog, and preprocessed to remove any faulty values or interference, which is achieved by comparison against a supplementary dataset such as the VIIRS DNB straylight corrected dataset. Temporal aggregation is executed using Google Earth Engine’s reducer functions like mean, max, and min. After completing data pre-processing and temporal aggregation, time-series analysis can be pursued, and the data for the chosen AOI can be either visualized or exported in a .csv format.

3.2.1

Microservices Technology Stack

Let’s now delve into the technology stack employed for the development of each individual microservice. A schematic depiction can be seen in Figure 3. The Ship Detection microservice serves as a REST API that offers ship detection functionality for a specified area. It utilizes the fastAPI backend framework²⁸, a modern Python web framework is renowned for its exceptional performance and user-friendliness. The API calls for the Sentinelsat-API²⁹ to obtain Synthetic Aperture Radar (SAR) images from Sentinel-1, which are subsequently preprocessed via the snappy toolkit³⁰ to furnish the required inputs for a YOLOv5 object detection model implemented in PyTorch. The Very High Resolution (VHR) Object Detection microservice, -functions as a REST API for detection in VHR images. The backend framework employs fastAPI. For umbrella detection in VHR images, we utilized the Mask-RCNN model implemented in PyTorch. A YOLOv5 model, also implemented in PyTorch, was employed for container detection. In the case of aircraft and ship detection, we used a Faster-RCNN model from the MMDetection framework³¹, a widely-used open-source object detection toolbox. NTL Analysis, Atmospheric Analysis, and Water Quality Analysis all exploit Google Earth Engine to acquire data from distinct datasets. Despite their varying data sources, these microservices are structured similarly and constructed using fastAPI. The output results from each microservice are presented as a pandas dataframe in JSON format, offering a flexible and user-friendly format for data analysis and visualization.

Figure 3.

Technology Stack Details.

One of the main design considerations for all these five microservices was scalability. The ability to process large volumes of data quickly and accurately was a top priority, as the datasets used in these microservices can be quite large. We also focused on making the microservices modular and easy to integrate with other applications and platforms. One challenge faced during the development of these microservices was optimizing the data processing pipeline to ensure speed. Additionally, working with large datasets can be computationally intensive, so we had to carefully balance computational resources with the need for accurate results. Moreover, ensuring that the models in the Ship Detection and VHR Object Detection microservices remained accurate while maintaining real-time performance was another challenge that we overcame through model selection, fine-tuning, and optimization. The Frontend is built on top of Streamlit³², a Python library that allows for the creation of interactive web applications. Several visualization libraries were utilized, such as Plotly³³ and LeafletJS³⁴, to create an intuitive and user-friendly UI experience. Plotly allows for the creation of interactive charts and graphs, while LeafletJS provides a flexible mapping library for visualizing geospatial data. The frontend provides a streamlined and intuitive way for users to interact with the data and makes it easy to explore different visualizations and insights. One of the main design considerations for the Frontend was creating a responsive and visually appealing interface that would be easy for users to navigate. We focused on creating a design that would be both functional and aesthetically pleasing, while also ensuring that the interface was intuitive and easy to use. We also had to ensure that the interface was responsive and performant, even when dealing with large datasets or complex visualizations.

3.2.2

Leveraging Serverless Computing

In this section, we discuss the serverless architecture used to efficiently run the computer vision models that are crucial to our platform. Leveraging serverless computing allows us to seamlessly scale the processing of these models, enabling faster and more accurate results for our users. Below, we briefly describe the fundamentals of Serverless Computing and then we delve into the specific benefits and implementation details of our serverless approach. Serverless computing, also known as Function as a Service (FaaS), is a cloud computing model that allows developers to focus solely on coding and deploying their applications without the need to manage underlying servers or infrastructure^35,36. The cloud provider handles server management, capacity planning, and scaling, making it more efficient and cost-effective for developers^{37,38,39,40,41}. Popular commercial FaaS platforms include AWS Lambda, Google Cloud Functions, and Microsoft Azure Functions, while open-source alternatives include Apache OpenWhisk, OpenLambda, and Knative. Serverless computing is well-suited for deploying and executing scientific workloads, as it allows for optimal resource provisioning and function chaining^42,43. We opted for AWS Lambda, as it enables code execution without managing infrastructure components while defining events that activate the functions⁴⁴. AWS Lambda executes each function within a dedicated container, which is then executed on a multi-tenant cluster of machines managed by AWS. The total runtime of a function in AWS Lambda includes the execution of the code itself and the initialization performed by Lambda⁴⁵. The code is executed within a container deployed for this purpose, which will be reused if the function is triggered again within the next 15 minutes, reducing the initialization time⁴⁶. EYE-Sense utilizes AWS Lambda and serverless architecture to develop an object detection system capable of inferring the trained Computer-Vision models. For instance, our YOLOv5 models are implemented through Lambda functions, enabling parallel execution and optimal resource utilization. To handle large files exceeding 1GB in size, we employed AWS Elastic File System (EFS) and integrated it with the API Gateway. The API Gateway serves as an intermediary for securely accessing the EFS files, ensuring optimal performance and scalability while maintaining a high level of security. Our implementation also considers the “cold start” time, which occurs when a new instance of the Lambda function is initiated, and the EFS file system needs to be mounted. We carefully analyzed the cold start time to ensure efficient execution of the Lambda function. Our object detection system utilizes an Amazon S3 bucket as the input for processing images, with the API Gateway managing requests and triggering the YOLOv5 Lambda function. The YOLOv5 Lambda function then processes images, performs object detection, and deposits the resulting output in a designated S3 bucket, ensuring seamless and efficient operations of our Platform. Using the EFS-Lambda model provides several advantages over local computing, including efficient parallelism, scalability, automatic scaling, and a pay-as-you-go pricing model, which significantly reduces operational costs and simplifies infrastructure management. One approach we employ to minimize the costs associated with object detection tasks is to avoid the use of Amazon S3 or Elastic File System (EFS) storage when possible. Specifically, in cases where the available 512 MB of temporary storage provided by Lambda instances is sufficient for the task at hand, we opt not to use S3 or EFS storage. The motivation behind this approach is to reduce costs, as we will argue in the remainder.

We compared the actual cost of using AWS Lambda functions with or without S3 and EFS storage versus using Amazon EC2 with S3 storage. Amazon EC2 are virtual computing environments that allow users to rent virtual servers, also known as instances, to run their applications. We tested these three deployment configurations for a simple scenario. We inferred a YOLOv5 model on 1000 images with an average size of 50 MB each. For the AWS Lambda functions scenarios we allocated 300 MB of memory and we had a duration of 40 seconds per image inference and for the EC2 scenario we have a t3.medium instance with 2 vCPUs and 4 GB RAM.We considered the following factors for cost estimation:

Based on the data presented in Table 3, it is evident that processing 1,000 images using AWS Lambda functions without S3 storage incurs a cost of approximately $0.20, which is significantly lower than the cost of using Amazon S3 for storage and Amazon EC2 instances, which is approximately $1.54. The implementation of AWS Lambda functions can provide cost savings from 12.5% when we use S3 storage up to 85% percent when we do not use S3 storage in our Lambdas. At the same time, they offer the benefits of parallel processing, resulting in higher scalability and throughput when compared to the use of Amazon EC2 instances. While our testing and implementation were focused on the YOLOv5 model, similar performance boost and cost-effectiveness is to be expected when inferring our other models using AWS Lambda with or without temporary storage and parallel processing. By leveraging the benefits of AWS Lambda, EYE-Sense can efficiently parallelize the processing of image datasets, resulting in improved scalability and faster processing times. Additionally, by using temporary storage within the Lambda function, we eliminate the need for additional storage costs, further reducing the overall cost of image inference.

Table 3.

Actual cost estimates for each scenario (data transfer were considered negligible for all the scenarios).

3.3.1

Ship detection workflow – UI

Our platform offers a highly effective workflow for identifying ships in synthetic aperture radar (SAR) images using a YOLOv5 object detection model. Users provide their SentinelAPI credentials and choose a time-frame of interest, upon which our system searches for available Sentinel-1 tiles within the specified timeframe (as illustrated in Figure 5, left). The user can then select their preferred preprocessing steps and initiate the ship detection algorithm (Figure 5, right). Upon completion of the detection algorithm, users receive an email containing a .zip file with the processed images, each displaying bounding boxes that indicate detected ships, as well as multiple coco.txt files containing the results in COCO format. Figure 6 displays the [zoomed-in] result of ship detection on an image. Our user-friendly interface streamlines the entire process, enabling users to effortlessly search for SAR images, customize detection parameters, and obtain comprehensive output files with minimal effort.

Figure 5.

Ship detection workflow UI – area selection.

Figure 6.

Ship detection workflow UI – detection results.

3.3.2

VHR object detection - UI

A potent workflow is available for detecting a broad spectrum of objects within satellite images with very high resolution (VHR). Images can be uploaded by the user and the targets of interest, namely aircrafts, ships, containers, or umbrellas, can be selected, as shown in Figure 7. Once the object detection algorithm has completed, the user is provided with the results both as an image with bounding boxes indicating the detected object(s) and as a .txt file in COCO format, allowing for easy integration with other software or analysis tools. An example of the output images of our object detection models can be seen in Figure 8. The interface simplifies customization of detection parameters, uploading and processing of large data volumes, and generation of detailed output files with minimal effort.

Figure 7.

VHR object detection workflow UI.

Figure 8.

Object detection annotated results for aircrafts, ships and containers.

3.3.3

Atmospheric quality analysis – UI

EYE-Sense provides an efficient and user-friendly workflow for accessing and analyzing atmospheric indicators from Sentinel-5P data using Google Earth Engine. The user simply specifies the area of interest, the time interval for analysis, and the desired atmospheric indicators (Figure 9, top left). After retrieving and processing the pertinent data from the Earth Engine database, EYE-Sense offers users the choice to view the visualized results on an interactive map (Figure 9, top right) or extract raw data in .csv format. For those who prefer visualizing the data, we also offer an interactive plot graph in our UI (Figure 9, bottom). The user-friendly interface provided by our platform simplifies the process of exploring and analyzing atmospheric indicator data, allowing users to customize analysis parameters and generate detailed output files with ease.

Figure 9.

Atmospheric quality analysis - UI.