Oil palm fresh fruit bunches (FFB) are the sources of crude palm oil (CPO). CPO is one of the largest export commodities for Indonesia and Malaysia. Automation or mechanisation in CPO processes is crucial to saving time, labour, and cost. This study used the computer vision method to count moving oil palm FFB using video frames. It is possible to implement this counting method in the oil palm FFB sorting and grading stage to estimate the CPO mill capacity. Video recordings used an RGB camera with a frame rate of 20 FPS for 117 oil palm FFB of Tenera varieties. The counting program consisted of two parts, a detector and a tracker. The detector algorithm was self-developed by adding a colour feature, converting the RGB images of oil palm FFB to HSV colour space. The counting results were validated using the confusion matrix method. The results showed that the counting accuracy could reach 100.00%. However, the counting accuracy depends on the colour of the FFB and room light intensity. The confusion matrix validation resulted in average counting accuracy of 79.35% for 10 videos. The results showed the potential use of computer vision in counting moving oil palm FFB.