DeepSeaNet: Improving Underwater Object Detection using EfficientDet

1Why underwater detection is hard

Seawater is a hostile lens. It absorbs light, scatters it back into the camera and fills the frame with drifting particles. A detector has to find a shrimp a few dozen pixels wide in that.

Automated monitoring of marine life is how we count populations, notice invasive species, check the effect of fishing and pollution, and keep ships off the seabed. Cameras are cheap; people watching the footage are not. The catch is the footage itself. Brackish water carries silt and plankton, visibility changes with the tide and the weather, and a stationary camera sees the same rock, rope and calibration board for months while the animals it is meant to watch appear small, blurred and half-hidden.

The paper calls this a natural adversarial environment: the corruption is not crafted by an attacker, but it degrades a network in much the same way. Try it on real frames from the dataset.

Water conditions playground

Illustration Pick a frame, then move the sliders or try a preset.

Turbidity0

Marine snow0

Light falloff0

Sensor noise0

Contrast stretch0

What the network receives

RMS contrast of the frame–

Dashed outline: the frame as recorded. The number is the standard deviation of luminance, a simple proxy for how much edge information survives.

The effects are simple image operations for intuition, not a physical model of light in water. The frames themselves are unedited Brackish frames.

Low contrast

Turbidity pulls every pixel towards the colour of the water. Edges and textures, the cues a CNN relies on, fade first.

Tiny, moving targets

Small fish and shrimp cover a few dozen pixels and move between frames. They are the hardest classes for every detector trained here.

Real time

Monitoring runs on continuous video, often on the edge. That rules out slow two-stage detectors and favours compact one-stage models.

2The Brackish dataset

Three cameras bolted to a pillar of the Limfjord bridge between Aalborg and Nørresundby, nine metres below the surface, recorded by Aalborg University and annotated frame by frame.

The Brackish dataset (Pedersen et al., CVPR Workshops 2019) was filmed in real, murky coastal water rather than on a clear reef or in an aquarium, and its authors release 14,518 annotated frames. Videos are filed into folders by what they mostly show, and every frame carries bounding boxes for six object classes.

15,084frames extracted at 960×540 and paired with label files

logged in 0_Dataset

10,995frames left after dropping those with no box

logged in 0_Dataset

6object classes, from shrimp to large fish

Brackish annotations

70 : 20 : 10train, validation and test split in the released Roboflow version

paper Table 2

What is in the frames

Switch between the three ways the data is counted.

From raw video to training data

Every detector in this project consumes the same frames, but each toolkit wants its labels in a different file format. Step through the preprocessing, then drag the box to see what a single annotation looks like on disk.

Preprocessing and annotation formats

Click a step. Drag the box, or its corner handle.

Download the Kaggle release: AVI videos filed by category, with YOLO label files.archive.zip, unpacked to dataset/videos/<category>/
Extract every frame with ffmpeg, scaled to 960×540 with bicubic resampling.ffmpeg -i {name}.avi -vf scale=960:540 -sws_flags bicubic {name}-%04d.jpg
Match frames to label files by name and rename both to a running index.15,084 images and 15,084 label files
Convert PNG to JPG with ImageMagick.mogrify -path images -format jpg *.png
Remove frames whose label file is empty.10,995 images remain
Normalise box coordinates to the 0–1 range and shift class ids to start at 0.normalize_yolo_coords(), class_map
Split, then export per toolkit: YOLO text, COCO JSON or Pascal VOC XML.Roboflow export: 7,000 / 2,000 / 1,000

crab

Format

3Anatomy of the detector

A one-stage detector is three parts: a backbone that sees, a neck that mixes scales, a head that decides. DeepSeaNet keeps EfficientDet's recipe and proposes changes to each part.

From a frame to a box

Step through it, or click a block.

Figure 1 of the paper: EfficientNet backbone, BiSkFPN bottleneck and prediction head — **Figure 1 of the paper.** The proposed pipeline: backbone features at several scales, the BiSkFPN bottleneck that fuses them, and fully connected class and box networks at the head.

The neck: how scales talk to each other

A small shrimp is only visible in the high-resolution P3 map; a large fish fills the coarse P7 map. The neck decides how information flows between them. Each design below adds paths the previous one lacked. Hover an output node to light up every input it can draw from.

Feature pyramid explorer

Choose a design, then hover or tap a node in the right-hand column.

Hover an output node to trace it.

Figure 2 of the paper: BiFPN feature fusion — **Figure 2.** BiFPN, the neck of the original EfficientDet, repeated as stacked layers.

Figure 7 of the paper: proposed BiSkFPN bottleneck — **Figure 7.** The proposed BiSkFPN, with deconvolutional feature maps and skip connections.

The backbone: doing more with fewer weights

EfficientNet replaces ordinary convolutions with mobile inverted bottleneck blocks (MBConv). A 1×1 conv expands the channels, a depthwise conv filters each channel on its own, and another 1×1 conv projects back down. The paper also argues for the Swish activation over ReLU. Both choices are easy to feel with numbers.

MBConv parameter counter

Input channels80

Output channels80

Expansion t6

Kernel k3

Swish against ReLU

β1.00

Swish(x) = x · σ(βx). It dips slightly below zero and has a non-zero gradient for negative inputs, so weak, noisy activations are damped rather than cut to zero.

The head: one loss, three pulls

For every anchor box the class network predicts species probabilities and the box network predicts offsets. Training minimises a weighted sum: a classification term, a box regression term weighted by α, and an ℓ₂ penalty on the weights. The paper writes it as

L = L_cls + α·L_box + β·L_reg L_box = (1/N) Σᵢ Σⱼ∈{x,y,w,h} smoothL1(tⱼ − t̂ⱼ)

The EfficientDet-Lite0 run in the repository logged each of those terms for all 350 epochs, so the balance can be read straight off the training log. In that implementation α = 50 and β = 1.

Where the training loss comes from

Repository log Hover for values. Drag α to reweight the logged terms.

classification L_cls box α·L_box weight decay β·L_reg

Box weight α50

4Results reported in the paper

Six detectors, one dataset, five repetitions each. This section shows the paper's own tables; the next one shows what the committed runs in the repository logged.

mAP over five runs

Paper, Table 5 Each dot is one run. Hover for the value.

Axis

Class-wise mAP

Paper, Table 6 Click a column header to sort.

Column names follow the paper, which labels classes by Brackish video category. Darker cells mean higher mAP. See the notes in section 7 on how this table relates to Table 5.

98.63%mAP reported for EfficientDet with adversarial learning

paper, abstract

98.04%mAP reported for YOLOv5 with adversarial learning

paper, abstract

350training epochs for every model, weight decay 5×10⁻⁴

paper, Table 4

Experimental setup (paper, Table 4)

Setting	YOLOv5	EfficientDet	Detectron2
Epochs	350	350	350
Backbone	CSP-Darknet53	EfficientNet	ResNet
Neck	PANet	BiFPN	RPN + FPN
Head	YOLOv3-like	YOLOv3-like	RPN + RCNN
Train / val / test	7000 / 2000 / 1000	7000 / 2000 / 1000	7000 / 2000 / 1000
Annotations	YOLO TXT	COCO JSON	COCO JSON
Optimiser	SGD, lr 0.1	SGD + Adam, adaptive	SGD + Adam, adaptive
Activation	Leaky ReLU	Swish (neck), Leaky ReLU (head)	Sigmoid (box), softmax (class)

Hardware listed in the paper: AWS EC2 p3dn instance, NVIDIA V100 GPUs, 96 vCPUs, 31.2 USD per hour.

Where the design choices came from

The related-work study reviews five EfficientDet variants built for other hard imaging domains. Each contributed an idea, or a warning, for water.

Oil tanks from orbit

Attention and residual deformable 3D convolutions damp cloud and haze noise in remote sensing.

100 mAP · Xu et al., 2022

Crop circles in the desert

YOLOv5 against EfficientDet: EfficientDet scores higher, YOLOv5 generalises to more examples.

91 mAP · Mekhalfi et al., 2021

Ultrasonic defects

Anchor aspect ratios found by K-means with Jaccard distance for extreme box shapes.

89.65 mAP · Medak et al., 2021

Military ships

Multilayer attention with deep feature fusion to separate ship types in optical imagery.

97.05 mAP · Qin et al., 2021

Clothing landmarks

Compound-scaled EfficientDet that detects garments and landmarks in 42 ms per image.

68.6 mAP · Kim et al., 2021

Figure 12 of the paper: data, train and infer workflow — **The Data → Train → Infer workflow** followed for every detector, from the final report.

5What the committed runs show

The repository keeps one training run per detector, with logs, weights and saved predictions. Everything in this section is parsed from those files.

Run	Toolkit	Data	Input	Schedule	Params	AP@0.5	AP@[.5:.95]
EfficientDet-Lite0	TFLite Model Maker	Roboflow, test 1,000	320²	350 ep, batch 64	3.24 M	0.898	0.601
YOLOv5s	Ultralytics YOLOv5	own split, val 1,506	416	100 ep, batch 16	7.04 M	0.976	0.748
YOLOv8s	Ultralytics 8.0.20	Roboflow, val 2,000	800	100 ep, batch 16	11.13 M	0.988	0.836
Faster R-CNN X101-FPN	Detectron2	Roboflow, test	default	300 iter, batch 4	—	0.433	0.204

Splits and input sizes differ between runs, so compare shapes and orders of magnitude rather than third decimals. The EfficientDet-Lite0 row is the float model; the exported TFLite file scores 0.863 and 0.561. YOLO rows are the validation scores at the last epoch of the committed run.

Validation accuracy during training

Repository logs Hover to read all runs at one epoch. Click a legend entry to hide a run.

Metric

Which animals are hard

Repository logs AP@[.5:.95] per class. Hover a dot.

EfficientDet-Lite0 YOLOv5s YOLOv8s, 25-epoch notebook run Detectron2 (hollow)

All four detectors find starfish, which rarely move on the seabed, the easiest class. For the three one-stage detectors small fish, which swim through the frame in groups, are the hardest; the short Detectron2 run struggles most with shrimp. The strict metric averages over IoU thresholds from 0.5 to 0.95, so it rewards tight boxes rather than rough hits. The committed 100-epoch YOLOv8s run saved per-class results only as plots, so its dots come from the shorter 25-epoch run whose validation table is printed in the notebook.

Look at the predictions

YOLOv5s against YOLOv8s on the same frame

Pick a frame, then drag across it. Arrow keys step through frames.

The two toolkits number classes differently: YOLOv5 from 0, the Roboflow export used for YOLOv8 from 1. The order is the same.

YOLOv5sYOLOv8s

↔

EfficientDet-Lite0 on ten Brackish frames

Click to enlarge. Labels read class id and confidence.

Diagnostics saved with each YOLO run

Click an image to enlarge.

6Perturbations and explanations

Two further ideas in the paper: train on deliberately perturbed frames so the model shrugs off noise, and open the black box to check it looks at the animal, not the water.

Universal adversarial perturbations

A universal adversarial perturbation (UAP, Moosavi-Dezfooli et al., 2017) is a single, image-agnostic noise pattern, small enough to be nearly invisible, that pushes a network towards wrong predictions on most inputs. The paper adds UAP noise to training frames in a curriculum, a form of adversarial learning, and reports 98.63% mAP for EfficientDet and 98.04% for YOLOv5 trained this way.

The key word is universal: the same pattern is added to every frame. The panel below shows what that means at different strengths.

One pattern, every frame

Illustration Drag the strength. The pattern is random, not optimised against any model.

crab frame + δ

fish school + the same δ

δ itself, amplified

Strength ξ8/255

Where does the detector look?

Class activation maps colour each pixel by how much it drives the network's output. GradCAM++, the method the paper describes, weights the last convolutional feature maps by positive gradients of the class score:

Lᶜ(x, y) = ReLU( Σₖ αₖᶜ · Aₖ(x, y) )

The CAM notebooks in the repository use EigenCAM from pytorch-grad-cam, which needs no gradients: it projects the activations of one layer onto their first principal component. The maps below are the images those notebooks saved.

Class activation map viewer

Repository notebooks Choose a frame and a notebook, then slide between the detections and the map.

Detections ↔ map50%

7Reproducibility and limitations

What a reader should know before building on these numbers.

Paper and repository side by side

Detector	Paper, Table 5 mean	Committed run, AP@0.5	Committed run setup
EfficientDet	98.6 ± 1.0	89.8 (test), 86.3 as TFLite	EfficientDet-Lite0, TFLite Model Maker, 350 epochs
YOLOv8	98.2 ± 0.17	98.8 (val)	YOLOv8s, 100 epochs, 800 px
YOLOv5	97.6 ± 0.61	97.6 (val)	YOLOv5s, 100 epochs, 416 px
Detectron2	95.2 ± 1.4	43.3 (test)	Faster R-CNN X101-FPN, 300 iterations

Scope of the repository.

Each detector has a single committed run. The five repetitions behind Table 5 are not in the repository.
The EfficientDet notebook trains the stock efficientdet_lite0 specification from TFLite Model Maker. Code for the BiSkFPN neck, the Swish changes, UAP generation and adversarial training is not included.
The four CAM notebooks load YOLOv5-format checkpoints through torch.hub and run EigenCAM. They differ in the checkpoint they download and the confidence threshold, not in detector architecture.
Table 6's Detectron2 row (28.1, 14.5, 8.6, 3.8, 26.1, 40.6) closely matches the per-class COCO AP@[.5:.95] logged in 4_Detectron2.ipynb for classes 1–6 (28.8, 14.6, 8.6, 3.9, 25.7, 40.7), while Table 5 reports 95.2 for the same model. The two tables therefore do not report one consistent metric.
The paper and its Table 3 name classes after video folders (fish-school, fish-small…). The boxes use six object classes: fish, small fish, crab, shrimp, jellyfish and starfish.

Limitations that apply to any result on this data

Neighbouring frames leak. Frames are cut from continuous video, and consecutive frames are nearly identical. A random split puts near-duplicates into both training and test sets, which inflates every score. Splitting by video would give a harder and more honest estimate.

AP@0.5 is near its ceiling. Several detectors pass 0.97, where differences shrink to noise. The stricter AP@[.5:.95], and per-class numbers for the rare shrimp and jellyfish classes, separate models far better.

One site, one camera. Every frame comes from the same fixed rig in Limfjorden. Nothing here shows transfer to other water, depths, lighting or cameras.

Classes are imbalanced. Shrimp and jellyfish together make up only 3–4% of the validation boxes. A single mean hides how a model does on exactly the animals that are hardest to monitor.

8Code and citation

Every experiment is a notebook that runs top to bottom on Google Colab or AWS SageMaker.

Repository map

Pick a folder.

github.com/s4nyam/efficientdet-advml

Cite this work

@inproceedings{Jain2024DeepSeaNet,
  author    = {Jain, Sanyam},
  title     = {DeepSeaNet: Improving Underwater Object Detection using EfficientDet},
  booktitle = {2024 4th International Conference on Applied Artificial Intelligence (ICAPAI)},
  year      = {2024},
  pages     = {1--11},
  address   = {Halden, Norway},
  publisher = {IEEE},
  doi       = {10.1109/ICAPAI61893.2024.10541265}
}

Copied Preprint: arXiv:2306.06075