Orthogonality Constraints for Improved Image Classification Performance in Deep Networks


Deep learning models, specifically convolutional neural networks (CNNs), have become the preferred choice for image classification tasks. When training CNNs, the weights of the model are learned using categorical cross-entropy loss, a loss function that can achieve high classification performance. Despite recent efforts using cross-entropy loss with different regularization functions, a few key challenges persist: (1) the final model may provide questionable reliability due to sensitivities to pruning techniques (used for making more compact models for edge applications) that can impact classification performance; (2) deep learning models do not offer clear insights into which aspects of the image contribute towards the final label prediction; and (3) use of cross-entropy loss can lead to overconfidence of the model, since ground-truth labels are represented as one-hot-coded vectors.

Invention Description
Researchers at Arizona State University have developed an Orthogonal Sphere (OS) regularizer that emerges from physics-based latent-representations under simplifying assumptions. Under further simplifying assumptions, the OS constraint can be written in closed-form as a simple orthonormality term and be used along with the cross-entropy loss function. Findings indicate that use of the orthonormality loss function results in (a) rich and diverse feature representations, (b) robustness to feature sub-selection, (c) improved semantic localization in the class activation maps, and (d) reduction in model calibration error. Effectiveness of the proposed OS regularization has been demonstrated on four benchmark datasets—CIFAR10, CIFAR100, SVHN, and Tiny ImageNet. 

Potential Applications
•  Deep learning
•  Image analysis, computer vision
•  Autonomous vehicles, health analytics

Benefits and Advantages
•  Increases robustness, accuracy, efficiency, and interpretability of deep learning methods
•  Simple approach can be integrated into any deep learning architecture

Related Publication: Role of Orthogonality Constraints in Improving Properties of Deep Networks for Image Classification

Research Homepage of Professor Pavan Turaga

Case ID:
Last Updated:

For More Information, Contact