While fine-tuning a transfer learning model alleviates the need for a vast amount of training data, it still comes with a few challenges. One of them is the range of image dimensions that the input layer of a model accepts. This issue is of interest, especially in tasks that require the use of a transfer learning model. In scene classification, for instance, images could come in varying sizes that could be too large/small to be fed into the first layer of the architecture. While resizing could be used to trim images to a required shape, that is usually not possible for images with tiny dimension s, for example, in the case of the EuroSAT dataset. This paper proposes an Xception model-based framework that accepts images of arbitrary size and then resizes or interpolates them before extracting and enhancing the discriminative features using an adaptive dilation module. After applying the approach for scene classification problems and carrying out a number of experiments and simulations, we achieved 98.55% accuracy on the EuroSAT dataset, 99.22% on UCM, 96.15% on AID and 96.04% on the SIRI-WHU dataset, respectively. We also monitored the micro-average and macro-average ROC curve scores for all the datasets to further evaluate the proposed model’s effectiveness.