Determining the category of ripeness of oil palm fresh fruit bunches (FFB) based on maturity is essential in estimating suitable fruit for processing. Ripeness classification could prevent the oil palm from becoming over-ripe. When the oil palm becomes over-ripe, the quality of oil extracted is not optimal because of the increase in free-fatty acid level. The deep learning approach usually used in object detection could help in object classification based on the model trained with the oil palm dataset. Many kinds of research used shared convolutional neural network (CNN) architecture to detect the ripeness of oil palms fruits. This research contributes to finding the suitable CNN model specialised in oil palm FFB ripeness using an unequal scalable feature known as EfficientDet. We propose the most proper coefficient for object detection scaling with the following configuration. The compound coefficient D2’s resolution is used as EfficientDet-Lite2 input size. EfficientDet-Lite2’s backbone network is nearly identical to EfficientDet using D2. The bi-directional feature pyramid network (Bi-FPN) layer is five, and the box class per layer is three. The accuracy of the proposed EfficientDet-Lite2 using EfficientDet D2 input is 84% and has been tested in Indonesian plantations.