Week 4

I started this week with a meeting with Danielle to talk about what I was able to do last week and some interesting next steps I could take. The rest of the week I spent (1) continuing to make changes to the neural network, (2) looking more in-depth into how well the networks were performing, (3) attempting to use my networks on the small swarm, and (4) labeling bees.

1) The Neural Network:

Below are some more modifications I made to the neural network with the hopes of improving it. About halfway through the week I also was set up with a GPU, so I was able to try some things that were too time-intensive on my computer’s CPU. See last week’s entry for modifications 0-2. Note that all training occurred over 1000 epochs.

Modification 3:

  • given the starting neural network with 2 encoder and 2 decoder layers, I tried adding another layer to create a deeper network
  • This change increased the loss from 3.182e-06 to 1.940e-05 given the training data
  • Given test data (the same cube, cropped in an unseen way), the loss increased from 9.292 to 11.238
  • Training took a total wall time of 21min 44s on the CPU
  • I was quite surprised that making the network deeper decreased performance (at least when trained on one image) image

Modification 4:

  • Given the starting neural network, I added batch normalization to each convolution layer
  • This change increased the loss from 3.182e-06 to 0.001 given the training data
  • Loss decreased from 9.292 to 0.546 when tested on the same cube but with a different crop
  • Training took a total wall time of 21min 33s on the CPU image

Modification 4, but trained on 8 different crops of my cube:

  • Training on 8 different crops of the cube resulted in a training loss of 3.021e-05 and a test loss of 0.781
  • Training took a total wall time of 1h 21min 2s on the CPU image

Modification 5:

  • After gaining access to a GPU I decided to combine previous things that seemed to improve performance
  • First, I trained a network with regular channel numbers and batch normalization on 125 different crops of my cube
  • This resulted in a training loss of 1.058e-11 and a test loss of 1.830
  • Training took a total wall time of 46min 6s on the GPU image

Modification 6:

  • I then trained another network with regular channel numbers on 125 different crops of my cube, but also manually implemented early stopping to try to prevent overfitting
  • Out of all 125 pieces of training data, 20% were set aside for validation data, and a patience of 4 was used (unless specified, this is how all early stopping was set up)
  • After 40 epochs, the training stopped at a training loss of 0.007 and a validation loss of 0.011
  • Training completed after a wall time of 2min 1s on the GPU
  • Given an unseen crop of the cube, the test loss was 0.754 image

Modification 7:

  • I then trained a network with regular channel numbers, batch normalization, and early stopping on 125 crops of my cube
  • After 60 epochs, training stopped at a training loss of 0.0001 and a validation loss of 0.001
  • Training completed after a wall time of 2min 58s on the GPU
  • Given an unseen crop of the cube, the test loss was 0.351 image

Modification 8:

  • Training on 64 crops of the cube with early stopping, architecture has doubled channels and batch normalization (I ran out of available RAM when trying to train on 125 crops)
  • After 60 epochs, training stopped at a training loss of 0.0001 and a validation loss of 0.0006
  • Training completed after a wall time of 4min 8s on the GPU
  • Given an unseen crop of the cube, the test loss was 0.327 image

Testing Modification 0 with Cleaner Data:

  • Given the very first starting architecture, I decided to test it on a cleaner version of the data (less noise present)
  • Given this cleaner test data, the loss increased from 9.292 to 10.787 image

Testing Modification 6 with Cleaner Data:

  • Given the trained model from modification 6, I tested it on a cleaner version of the data
  • Given this cleaner test data, the loss increased from 0.003 to 2.162 image

2) Network Performance Analysis:

Up until this point, the only way I was evaluating how well my networks were doing was through loss and visually comparing the output to my labels, so I decided to try to evaluate some of the outputs more comprehensively. I first started with the original architecture as a benchmark. Similarly to when I was first analyzing my input data, I started by looking at the connected components of the output given different thresholds (0.1-0.9): image image It was apparent that, despite the threshold level, many of the “bees” identified by this network were connected. I visualized the sizes of these components using histograms: image image image So, using this benchmark network, around 25-30 individual bees were able to be identified, and many appear to be connected. Lastly, I visualized the connected components in 3D, and I was able to confirm that a large number of bees were connected: image

I then did the same analysis using the network with halved channels and trained on 27 different crops of the cube (at the time, this was my best-performing network). image From the above image, with a very low threshold, it is clear that some separate bees are connected, as shown by the light green masses. Regardless, a significantly higher number of bees is being identified when compared to the benchmark output. image image With higher thresholds, there are fewer connected components and what appear to be more individual bees, though bees are at higher risk of getting split into two/three separate pieces. image image image Given this network’s output, there also seems to be a larger variety of bee sizes, ranging from 1 to 533 pixels. image I looked at this output in 3D as well, and it looks much more similar to what my original labels looked like. Unlike the previous output, there are no large masses of connected bees and each bee seems fairly well-shaped and separated. image

3) Using the Network(s) on the Small Swarm:

I decided to try to apply one of my networks to the small swarm I’m currently labeling, as I’ve only ever tried applying the networks to small cubes of bees. To apply the network, trained on a small cube size, the network needs to be applied to the swarm in patches. The swarm is cropped into smaller cubes (patches), and the network gets an output for these smaller sizes. Then, the center of the cube is given more “weight” than the edges, as the network is more likely to make a correct prediction at the center of the cube. All parts of the swarm must be evaluated in a patch, and the outputs of all the patches are then combined, using averages.

When I used the neural network with halved channels and trained on 27 different crops of the cube (at the time, this was my best-performing network), the result looked promising (note that the empty black box in the middle of the figure is where a cage containing the queen bee was; this x-ray data was removed from the input): image image image image It appears that, while some of the bees are connected the model is doing a fairly good job at identifying individual bees. While the threshold gets higher though, the bees do become separated, and many individual pixels are considered individual components. This observation can be seen in histograms of component sizes as well: image image image In order to take a better look at the output, I visualized it in Napari using a threshold of 0.5 (after removing all components smaller than 100 pixels).

Untitled video

4) Labelling the Small Swarm:

By the end of this week, I was able to finish labeling about 160 bees. image

Written on June 28, 2024