Defect detection has achieved state-of-the-art results in both localisation and classification of various types of defects, manufacturing domains is no exception to this. Just like in many areas of computer vision there is an assume of very high-quality datasets that have been verif ied by domain experts, however labelling such data has become an increasing problem as we require greater quantities of it. Within defect detection the variability and composite nature of defect characteristics makes this a time-consuming and interaction-heavy task with great amount of expert effort. We propose a new acquisition function based on the similarity of defect properties for refining labels over time by showing the expert only the most required to be labelled. We also explore different ways in which the exp ert labels defects and how we should feed these new refinements back into the model for utilising new kno wledge in an effortful way. We achieve this with a graphical interface that provides additional inform ation as data gets refined into a dense segmentation, allowing for decision-making with uncertain areas of the image.