Fully convolutional networks for 2D segmentation #186

StephanieLarocque · 2017-05-01T20:30:23Z

Only look at doc/fcn_2D_segm.txt for now (other tutorials still under construction).

I know fcn_2D_segm.txt still miss:

Images reference
Whole references
Cleaned code from code/fcn_2D_segm directory (less files, remove absolute paths)

lamblin · 2017-05-01T22:01:37Z

doc/fcn_2D_segm.txt

+*downsampling path* to refer to the network up to *pool5* layer and we will use the term
+*upsampling path* to refer to the network composed of all layers after *pool5*. It is worth
+noting that the 3 FCN architectures share the same downsampling path, but differ in their
+respective upmsapling paths.


"upsampling"

lamblin · 2017-05-01T22:04:21Z

doc/fcn_2D_segm.txt

+
+
+1. **FCN-32** : Directly produces the segmentation map from *pool5*, by using a
+transposed convolution layer with stride 32.


What about conv6-7? Are they the transposed convolution you mention? But if there are two of them, which one has stride 32?
Or is it a transposed convolution after conv7?

Explanations added.

lamblin · 2017-05-01T22:06:19Z

doc/fcn_2D_segm.txt

+1. **FCN-32** : Directly produces the segmentation map from *pool5*, by using a
+transposed convolution layer with stride 32.
+
+2. **FCN-16** : Sums the 2x upsampled prediction from *pool5* with *pool4* and then


How is that 2x upsampling done? Simple duplication of pixels, or another strdied transposed convolution? Or something else?

Explanations added.

lamblin · 2017-05-01T22:22:49Z

code/fcn_2D_segm/dataset_loaders/.gitignore

+*.pyc
+dataset_loaders/datasets
+.cache
+config.ini


I'm not sure we want to add the whole dataset_loaders repo into this one.

I removed it

lamblin · 2017-05-01T22:26:41Z

doc/fcn_2D_segm.txt

+**Fully Convolutional Networks** (FCNs) owe their name to their architecture, which is
+built only from locally connected layers, such as convolution, pooling and upsampling.
+Note that no dense layer is used in this kind of architecture. This reduces the number
+of parameters and computation time. To obtain a segmentation map (output), segmentation


Also, the network can work regardless of the original image size, since all connections are local, there does not need to be any fixed number of units at any stage.

comment added

StephanieLarocque · 2017-05-01T23:49:15Z

Ok je ferai les modifications demain pour - upsampling (erreur de frappe) - explications conv6, conv7 - explications des upsampling utilisés - Pour le dataset_loaders repo, je vais simplement mettre le lien du repo github (je me demandais justement) - Je vais ajouter la dernière annotation (can work regardless of the original image size) Merci beaucoup pour cette réponse rapide

________________________________ De : Pascal Lamblin [notifications@github.com] Envoyé : 1 mai 2017 18:27 À : lisa-lab/DeepLearningTutorials Cc : Larocque Stéphanie; Author Objet : Re: [lisa-lab/DeepLearningTutorials] Fully convoutional networks for 2D segmentation (#186) @lamblin commented on this pull request.

________________________________ In doc/fcn_2D_segm.txt<#186 (comment)>:

+explicitely show the relative size of the feature maps.

+ +.. figure:: images/fcn.png + :align: center + :scale: 50% + + **Figure 2** : FCN architecture (from FCN paper) + +**Difference between the 3 FCN variants** + +As shown below, these 3 different architectures differ in the stride of the last convolution, +and the skip connections used to obtain the output segmentation maps. We will use the term +*downsampling path* to refer to the network up to *pool5* layer and we will use the term +*upsampling path* to refer to the network composed of all layers after *pool5*. It is worth +noting that the 3 FCN architectures share the same downsampling path, but differ in their +respective upmsapling paths. "upsampling"

________________________________ In doc/fcn_2D_segm.txt<#186 (comment)>:

+ :scale: 50%

+ + **Figure 2** : FCN architecture (from FCN paper) + +**Difference between the 3 FCN variants** + +As shown below, these 3 different architectures differ in the stride of the last convolution, +and the skip connections used to obtain the output segmentation maps. We will use the term +*downsampling path* to refer to the network up to *pool5* layer and we will use the term +*upsampling path* to refer to the network composed of all layers after *pool5*. It is worth +noting that the 3 FCN architectures share the same downsampling path, but differ in their +respective upmsapling paths. + + +1. **FCN-32** : Directly produces the segmentation map from *pool5*, by using a +transposed convolution layer with stride 32. What about conv6-7? Are they the transposed convolution you mention? But if there are two of them, which one has stride 32? Or is it a transposed convolution after conv7?

________________________________ In doc/fcn_2D_segm.txt<#186 (comment)>:

+ **Figure 2** : FCN architecture (from FCN paper)

+ +**Difference between the 3 FCN variants** + +As shown below, these 3 different architectures differ in the stride of the last convolution, +and the skip connections used to obtain the output segmentation maps. We will use the term +*downsampling path* to refer to the network up to *pool5* layer and we will use the term +*upsampling path* to refer to the network composed of all layers after *pool5*. It is worth +noting that the 3 FCN architectures share the same downsampling path, but differ in their +respective upmsapling paths. + + +1. **FCN-32** : Directly produces the segmentation map from *pool5*, by using a +transposed convolution layer with stride 32. + +2. **FCN-16** : Sums the 2x upsampled prediction from *pool5* with *pool4* and then How is that 2x upsampling done? Simple duplication of pixels, or another strdied transposed convolution? Or something else?

________________________________ In code/fcn_2D_segm/dataset_loaders/.gitignore<#186 (comment)>:

@@ -0,0 +1,4 @@

+*.pyc +dataset_loaders/datasets +.cache +config.ini I'm not sure we want to add the whole dataset_loaders repo into this one.

________________________________ In doc/fcn_2D_segm.txt<#186 (comment)>:

+Classification needs to understand *what* is in the input (namely, the context). However,

+in order to predict what is in the input for each pixel, segmentation needs to recover +not only *what* is in the input, but also *where*. + +.. figure:: images/cat_segmentation.png + :align: center + :scale: 35% + + **Figure 1** : Segmentation network (from FCN paper) + +TODO : reference de l'image + +**Fully Convolutional Networks** (FCNs) owe their name to their architecture, which is +built only from locally connected layers, such as convolution, pooling and upsampling. +Note that no dense layer is used in this kind of architecture. This reduces the number +of parameters and computation time. To obtain a segmentation map (output), segmentation Also, the network can work regardless of the original image size, since all connections are local, there does not need to be any fixed number of units at any stage. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#186 (review)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AO3-hYa6vYOt-VOFbvPLdPtUPtopOkC5ks5r1lw5gaJpZM4NNZ2w>.

- Except upsampling path (convo 6, 7)

lamblin · 2017-05-04T23:10:42Z

code/fcn_2D_segm/data_loader.py

@@ -0,0 +1,231 @@
+import os


Do we still need that file?

No, I left the two data_loader.py from unet and fcn_2D_segm just in case. But they can be removed

lamblin

Also:

remove code/cnn_1D_segm/data_loader/__init__.pyc
remove code/cnn_1D_segm/data_loader/cortical_layers.pyc
remove code/cnn_1D_segm/data_loader/parallel_loader_1D.pyc
you still need to explain that Lasagne and dataset_loaders are requirements, and should be installed (and link to their doc or repo there, add version number).
Can you add links to your tutorial in the home page (index.txt), maybe as a list of 3 tutorials on segmentation for medical imagery?
Also, I just realized there was something to cite for datasets_loader: Francesco Visin, Adriana Romero - Dataset loaders: a python library to
load and preprocess datasets (BibTeX)

lamblin · 2017-05-04T23:11:53Z

code/fcn_2D_segm/fcn8.py

+    if load_weights:
+        if pascal:
+            path_weights = '/data/lisatmp4/erraqabi/data/att-segm/' + \
+                          'pre_trained_weights/pascal-fcn8s-tvg-dag.mat'


It looks like it is still there.

lamblin · 2017-05-04T23:19:31Z

code/unet/data_loader.py

@@ -0,0 +1,231 @@
+import os


Is that file still needed as well?

No, you can remove it if the test works

Or I can do it.

lamblin · 2017-05-04T23:30:57Z

doc/fcn_2D_segm.txt

+of polyps for each pixel. The other subdirectories (/masks3 and /masks4) are,
+respectively, for a segmentation task with 3 and 4 classes, but will not be
+presented here.
+


It would be good to have a sentence explaining how to set up the configuration file for dataset_loaders to point to the right thing after the data is downloaded.
For instance, data for the other tutorials will be downloaded in DeepLearningTutorials/data/, so maybe use that as an example, and tell to change the config.ini file to have

[general] datasets_local_path = /path/to/DeepLearningTutorials/data [polyps192] shared_path = /path/to/DeepLearningTutorials/data/polyps_split7

lamblin · 2017-05-04T23:32:30Z

doc/unet.txt

++++
+
+The data is from ISBI challenge and can be found `here <http://brainiac2.mit.edu/isbi_challenge/home>`_.
+


I think there should be instructions as well as what to put / add to datasets_loaders's config file.
I guess the section should be [isbi_em_stacks].

Also, there seems to be data augmentation by default during training.

it would be nice to mention that, maybe a sentence or two about what that does and why it matters

also, it uses another dependency, SimpleITK. We should indicate it is an additional dependency.

lamblin · 2017-05-05T00:08:33Z

code/fcn_2D_segm/train_fcn8.py

+                                  batch_size=batch_size[0],
+                                  seq_per_subset=0,
+                                  seq_length=0,
+                                  data_augm_kwargs=train_data_augm_kwargs,


I guess it should be defined to {}, as that was the default in data_loader.py.

This would confirm that data_loader.py is not loaded.

In fact we must keep data_augm_kwargs = data_augmentation here in order to crop all the images to (224,244) and be able to use batch size >1.
Data_augmentation is a default argument for the parser at the bottom of the page. It is set with the right data augmentation to use.

OK, thanks. I'll run the full thing, then.

lamblin · 2017-05-05T00:12:54Z

code/fcn_2D_segm/train_fcn8.py

+                                  return_01c=False,
+                                  overlap=0,
+                                  use_threads=False,
+                                  shuffle_at_each_epoch=shuffle_train,


And shuffle_at_each_epoch=True here (but keep False below).

lamblin · 2017-05-05T00:22:23Z

doc/cnn_1D_segm.txt

+We are interested in the outer part of the brain, the cortex.
+More precisely, we are interested in segmenting the 6 different layers of the cortex in 3D.
+Creating an expertly labelled training dataset with each 2D section (shown in figure 1) is unfeasible. Instead of giving as input a 2D image of one section of the brain, we give as input 1D vectors with information from across the cortex, extracted from smaller portions of manually labelled cortex
+as shown in Figure 2. The actual dataset can be found `here TODO: link the dataset`.


OK. We should probably mention the name of the section for datasets_loader as well (cortical_layers I guess).

lamblin · 2017-05-05T00:57:48Z

doc/cnn_1D_segm.txt

+The FCN implementation can be found in the following file:
+
+* `fcn1D.py <../code/cnn_1D_segm/fcn1D.py>`_ : Main script. Defines the model.
+* `train_fcn1D.py <../code/cnn_1D_segm/train_fcn1D.py>`_ : Training loop


This one seems to be starting fine.
I did not complete the training on a GPU yet, and the small one I had for visualization did not have enough RAM, but it should be OK.

lamblin · 2017-05-05T01:02:29Z

code/unet/train_unet.py

+                                     use_threads=True,
+                                     shuffle_at_each_epoch=shuffle_train,
+                                     return_list=True,
+                                     return_0_255=return_0_255)


Here again, I guess one_hot -> False, shuffle_train -> True, return_0_255 -> False.

Done. (Thanks)

lamblin · 2017-05-05T01:29:07Z

doc/unet.txt

+
+* `Unet.py <https://github.com/Lasagne/Recipes/blob/master/modelzoo/Unet.py>`_ : Main script. Defines the model
+
+* `train_unet.py <../code/unet/train_unet.py>`_ : Training loop.


This one starts correctly as well (needs more memory than I have available now).

Add final results and timing for cnn 1D

lamblin · 2018-06-15T21:08:58Z

Finally finished in #204!

StephanieLarocque and others added 23 commits May 1, 2017 11:45

first commit

22ab392

images for fcn and unet

b4a7abf

small changes

21b8aec

polyps dataset explanation + metrics

d3b111f

old build instructions

215b66e

jaccard visualisation

6ead56b

code for fcn8, miss dataset_loaders

86394c3

small changes

70bfaa9

small changes

4ca76f5

Merge branch 'master' of https://github.com/StephanieLarocque/DeepLea…

e2f05cc

…rningTutorials

fixed details

4880afa

Merge branch 'master' of https://github.com/StephanieLarocque/DeepLea…

3404767

…rningTutorials

fixed fcn8

f473730

cortical

cb59677

ray

cc33e5c

remove file

058bf42

remove dependance from metrics.py

925309d

remove dependance from model_helpers.py

2bd5db2

big brain images

730a06d

added reference captions

1ebb26f

global dataset loader from fvsin

7379e56

first commit for cortical layers segmentation

32e06a7

first commit

751efc3

lamblin reviewed May 1, 2017

View reviewed changes

StephanieLarocque and others added 5 commits May 2, 2017 09:10

cortical layers

cf64352

cortical layers updated

b5aa18a

cortical layers imagse

6621007

Feedback comments from PascalLamblin

d1521b3

- Except upsampling path (convo 6, 7)

dataset explanation addeed

1cfe08e

StephanieLarocque added 4 commits May 4, 2017 14:38

accuracy import fix

8b0f364

small change

f75d174

load data from train file

d8027b0

data link

cca8512

lamblin reviewed May 4, 2017

View reviewed changes

lamblin reviewed May 5, 2017

View reviewed changes

StephanieLarocque and others added 17 commits May 5, 2017 11:40

removed files .pyc

2a62c86

index updated with new links

afc1d79

ref for dataset loaders

655f023

remove loading pretrained weights

b06f3ee

update in loading data parameters

4014ed5

added requirements for lasagne, datasetloaders and simple ITK

4f4ac27

specify that data augmentation is used

ca481e1

data augmentation fixed

10636c6

fix in shared_path for config.ini

c9ce864

More small fixes

01cd675

Fix warnings in doc generation

06a2827

Remove files that are not used any more

efecb56

Formatting

1b735a8

Changes to help unet run

b88aec2

added per class jaccard

bdf617a

Results for unet

22dcc93

Preload data for 1D cortical dataset

6333200

Add final results and timing for cnn 1D

lamblin changed the title ~~Fully convoutional networks for 2D segmentation~~ Fully convolutional networks for 2D segmentation Jul 3, 2017

notoraptor mentioned this pull request Jan 12, 2018

[continuation of #186] Fully convolutional networks for 2D segmentation #200

Closed

notoraptor and others added 2 commits June 14, 2018 16:49

Update segmentation tutorials and .gitignore.

91c95cc

Add final scores and training times for FCN 2D segmentation

d2b2d41

lamblin mentioned this pull request Jun 15, 2018

Fully convolutional networks for 2D segmentation #204

Merged

lamblin closed this Jun 15, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fully convolutional networks for 2D segmentation #186

Fully convolutional networks for 2D segmentation #186

StephanieLarocque commented May 1, 2017

lamblin May 1, 2017

StephanieLarocque May 3, 2017

lamblin May 1, 2017

StephanieLarocque May 3, 2017

lamblin May 1, 2017

StephanieLarocque May 3, 2017

lamblin May 1, 2017

StephanieLarocque May 3, 2017

lamblin May 1, 2017

StephanieLarocque May 3, 2017

StephanieLarocque commented May 1, 2017 via email

lamblin May 4, 2017

StephanieLarocque May 5, 2017

lamblin left a comment

lamblin May 4, 2017

lamblin May 4, 2017

StephanieLarocque May 5, 2017

StephanieLarocque May 5, 2017

lamblin May 4, 2017

lamblin May 4, 2017

lamblin May 5, 2017

lamblin May 5, 2017

lamblin May 5, 2017

StephanieLarocque May 5, 2017

StephanieLarocque May 5, 2017 •

edited

Loading

lamblin May 5, 2017

lamblin May 5, 2017

StephanieLarocque May 5, 2017

lamblin May 5, 2017

lamblin May 5, 2017

lamblin May 5, 2017

StephanieLarocque May 5, 2017

lamblin May 5, 2017

lamblin commented Jun 15, 2018



		1. FCN-32 : Directly produces the segmentation map from pool5, by using a
		transposed convolution layer with stride 32.

		++++

		The data is from ISBI challenge and can be found `here <http://brainiac2.mit.edu/isbi_challenge/home>`_.


		* `Unet.py <https://github.com/Lasagne/Recipes/blob/master/modelzoo/Unet.py>`_ : Main script. Defines the model

		* `train_unet.py <../code/unet/train_unet.py>`_ : Training loop.

Fully convolutional networks for 2D segmentation #186

Fully convolutional networks for 2D segmentation #186

Conversation

StephanieLarocque commented May 1, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

StephanieLarocque commented May 1, 2017 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lamblin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

StephanieLarocque May 5, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lamblin commented Jun 15, 2018

StephanieLarocque May 5, 2017 •

edited

Loading