Severe sensitivity to Randomize Pixels With Mask

0% found this helpful

Vulnerability Info

Vulnerability Typeperformance
CVE Number
Created Date2023/01/12
Reported By@robustintelligence
References

    Affected Versions

    @a4fd55aade86349731fef059b2632d5bf8b3011c10 December 2022

    Description

    Summary

    A Randomize Pixels With Mask test was performed on swinv2-large-patch4-window12to16-192to256-22kto1k-ft, in which a 16% failure rate was observed. In at least one case, the model's prediction changed 0.83. This caused the label to change from 706 to 580.

    Test Information

    This test measures the robustness of the model to Randomize Pixels With Mask transformations. It does this by taking a sample input, randomizing pixels with fixed probability, and measuring the behavior of the model on the transformed input.

    Why is this important?

    Production inputs can have unusual variations amongst many different dimensions, ranging from lighting changes to sensor errors to compression artifacts. It is important that the models are robust to the introduction of such variations.

    Model Information

    • Model name: swinv2-large-patch4-window12to16-192to256-22kto1k-ft
    • Model package URL: pkg:huggingface/microsoft/swinv2-large-patch4-window12to16-192to256-22kto1k-ft@a4fd55aade86349731fef059b2632d5bf8b3011c
    • Macro F1 score on reference / evaluation: 0.92 / 0.85
    • Multiclass accuracy on reference / evaluation: 0.93 / 0.85
    • Multiclass AUC on reference / evaluation: 1.00 / 1.00
    • Macro Precision on reference / evaluation: 0.94 / 0.87
    • Macro Recall on reference / evaluation: 0.93 / 0.85

    This report was automatically generated by the scanning engine rime-0.21.0rc4.post195+git.2a88076b.d on 2023-01-12 17:49.

    contributor image
    Robust Intelligence
    Helpfulness score: 5