Icons are a key building block of many modern day applications and websites. They make user interfaces more engaging and aesthetically pleasing while delivering meaning to the user in a compact and glanceable way. However, icons require special attention from designers and developers to make them accessible to people who rely on screen readers. In order for screen readers to recognize and read out icons to the user, they need special accessibility annotations that provide a textual description of an icon’s purpose and function. As various studies show, such annotations are missing in many cases. To solve this problem, we propose a semantic icon classifier that can be used to predict the semantic class of an icon based on its pixel values. The predicted label can then be further used to improve accessibility of icons. While there has been various work on classifying icons by their semantics on mobile platforms, less attention has been paid to the web, despite possible differences in the appearance and variety of icons. By extending a dataset of icons found in mobile applications with web icons of open-source icon sets, icon databases and websites, we improve the accuracy of a semantic icon classifier on a real-world task. Furthermore, we propose a, to the best of our knowledge, novel classification approach to automatically generate semantic labels for compound icons. This allows us to predict more detailed classes for each icon. We demonstrate the real-world practicality of the system by implementing a browser extension that automatically attaches missing accessibility annotations to icons.