Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added PILToTensor and ConvertImageDtype classes in reference scripts #4495

Merged

Conversation

prabhat00155
Copy link
Contributor

@prabhat00155 prabhat00155 commented Sep 28, 2021

Detection:

srun -p train --cpus-per-task=16 -t 10:00:00 --gpus-per-node=8 python -m torch.distributed.launch --nproc_per_node=8 --use_env train.py --model fasterrcnn_resnet50_fpn --output-dir logs/to_tensor/ --pretrained --test-only

Without these changes:

IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.369
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.585
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.396
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.212
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.403
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.482
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.307
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.485
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.509
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.317
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.544
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.649

With these changes:

IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.369
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.585
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.396
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.212
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.403
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.482
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.307
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.485
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.509
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.317
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.544
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.649

Segmentation:

srun -p train --cpus-per-task=16 -t 10:00:00 --gpus-per-node=8 python -m torch.distributed.launch --nproc_per_node=8 --use_env train.py --model fcn_resnet50 --output-dir logs/to_tensor/ --pretrained --test-only

Without these changes:

global correct: 91.4
average row correct: ['94.0', '80.9', '73.9', '72.3', '49.9', '56.9', '81.0', '66.5', '88.6', '43.2', '75.7', '67.6', '80.9', '84.3', '81.2', '90.9', '49.0', '80.4', '61.2', '77.7', '69.0']
IoU: ['90.3', '72.6', '61.2', '61.6', '42.7', '46.9', '73.0', '53.7', '80.3', '33.6', '64.1', '33.9', '61.2', '72.0', '73.6', '80.7', '29.1', '63.5', '48.2', '71.2', '56.3']
mean IoU: 60.5

With these changes:

Test: Total time: 0:00:49
global correct: 91.4
average row correct: ['94.0', '80.9', '73.9', '72.3', '49.9', '56.9', '81.0', '66.5', '88.6', '43.2', '75.7', '67.6', '80.9', '84.3', '81.2', '90.9', '49.0', '80.4', '61.2', '77.7', '69.0']
IoU: ['90.3', '72.6', '61.2', '61.6', '42.7', '46.9', '73.0', '53.7', '80.3', '33.6', '64.1', '33.9', '61.2', '72.0', '73.6', '80.7', '29.1', '63.5', '48.2', '71.2', '56.3']
mean IoU: 60.5

cc @vfdev-5 @datumbox

Copy link
Contributor

@datumbox datumbox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @prabhat00155, the numbers look OK though it would be good to provide also runs on the latest main to confirm.

For example on our docs we report that the mAP of the Faster R-CNN ResNet-50 FPN model should e 37.0 not 36.9 that you got but I believe this is the result of a separate accuracy deterioration and not related to your changes. By having runs on the latest main you can prove that this PR is not to blame.

Finally below I left some comments for your consideration, let me know what you think:

Copy link
Contributor

@datumbox datumbox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@prabhat00155 prabhat00155 merged commit 6518372 into pytorch:main Sep 30, 2021
@prabhat00155 prabhat00155 deleted the prabhat00155/replace_ref_class branch September 30, 2021 12:53
facebook-github-bot pushed a commit that referenced this pull request Oct 1, 2021
… scripts (#4495)

Summary:
* Added PILToTensor and ConvertImageDtype classes in reference scripts

* Addressed review comments

* Fixed TypeError

* Addressed review comment

Reviewed By: prabhat00155, NicolasHug

Differential Revision: D31309550

fbshipit-source-id: 623e6c70d74348315118c269297d6d31bf4a68f6
pmeier pushed a commit to pmeier/vision that referenced this pull request Oct 1, 2021
…ytorch#4495)

* Added PILToTensor and ConvertImageDtype classes in reference scripts

* Addressed review comments

* Fixed TypeError

* Addressed review comment
mszhanyi pushed a commit to mszhanyi/vision that referenced this pull request Oct 19, 2021
…ytorch#4495)

* Added PILToTensor and ConvertImageDtype classes in reference scripts

* Addressed review comments

* Fixed TypeError

* Addressed review comment
cyyever pushed a commit to cyyever/vision that referenced this pull request Nov 16, 2021
…ytorch#4495)

* Added PILToTensor and ConvertImageDtype classes in reference scripts

* Addressed review comments

* Fixed TypeError

* Addressed review comment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants