Autonomous navigation in oil palm plantations faces significant challenges due to dense vegetation, uneven terrain and the inability of traditional systems to distinguish between obstructive and non-obstructive flora. Current rover navigation systems often misclassify low-lying vegetation (e.g., grass, shrubs) as obstacles, leading to inefficient path planning and increased operational costs. To address this gap, a vegetation filtering (VEGFIL) system that leverages YOLOv8n object detection and point cloud data from ZED camera was developed to enhance navigation accuracy of autonomous rovers in the plantation environments. Validated using datasets, the system achieved an average precision of 95%, recall of 86%, accuracy of 85%, and F1-Score of 0.9 in the vegetation classification. This innovation addresses critical gaps in agricultural automation, minimising misclassification of low-lying vegetation, optimising route planning, addressing labour shortages and promoting sustainable resource management. This innovation advances the adoption of precision agriculture, offering scalable automation for high-value crops where navigation accuracy directly impacts yield and ecological sustainability.