While standard deep-learning models often lose fine-scale information, a specialized small-target detection branch allows robots to identify fruit from much greater distances. This breakthrough is particularly significant as the global agricultural sector in 2026 continues to struggle with an unprecedented labor shortage that jeopardizes the long-term viability of food supply chains. Harvesting remains the most labor-intensive component of farming, demanding a combination of endurance and precision that is increasingly difficult to source from a human workforce. While automation has long been touted as the ultimate remedy, a persistent gap has existed between a machine’s ability to recognize an object and its ability to physically interact with it. Researchers at the University of Minho’s Mechatronics Laboratory have addressed this critical “blind spot” by developing the E-YOLOv11-RGBD framework. This system does not merely detect fruit; it creates a logical bridge between high-level computer vision and complex robotic manipulation.
Hardware Integration: Building the Robotic Foundation
To validate their findings, the research team constructed a high-performance physical platform that integrates various industry-standard components into a cohesive unit. The perception system utilizes an Intel RealSense D435i camera, which provides both standard RGB imagery and critical depth data necessary for spatial navigation. This is paired with a UR3e collaborative robotic arm, a choice that reflects the current trend in 2026 toward safer, human-centric agricultural machinery. To handle the delicate task of grasping different fruit varieties, the arm is equipped with an OnRobot 2FG7 electric gripper. The entire assembly is mounted onto an Omron LD-60 mobile robot base. While the base remained stationary during this specific set of trials, its inclusion demonstrates a clear path toward fully autonomous mobile harvesters that can navigate greenhouse aisles without human intervention. This setup provides the necessary physical precision to test complex software.
Beyond the electronic components, the researchers emphasized the importance of mechanical stability and long-term durability in a farm environment. The robotic arm and its peripheral sensors were secured to a modular aluminum frame specifically designed to withstand the repetitive stresses of harvesting operations. Before physical testing began, the structure underwent a comprehensive finite-element analysis to identify potential points of failure or excessive vibration. This engineering rigor resulted in a confirmed safety factor of 15, which is exceptionally high for agricultural equipment. Such a robust foundation ensures that the delicate movements of the robotic arm are not undermined by structural wobbling or material fatigue over time. By combining cutting-edge robotics with a reinforced physical frame, the team established a reliable and repeatable baseline for testing their advanced software enhancements, ensuring that the results were a product of the vision system rather than mechanical luck.
Vision Enhancements: Optimizing the Detection Core
At the technological heart of this framework is a heavily modified version of the YOLOv11 object detector, which the team optimized to handle the specific visual chaos of a modern greenhouse. The addition of a specialized small-target detection branch proved essential for identifying crops like strawberries that might be partially obscured or located several meters away from the sensors. Furthermore, a bidirectional feature fusion mechanism was implemented to allow the model to share information across different scales of imagery. This ensures that the robot can identify a large, prominent bell pepper with the same level of confidence as a tiny, unripened berry. These architectural adjustments are vital because greenhouses are rarely uniform environments; they are filled with varying fruit sizes and complex visual layers. By refining how the model processes these different scales, the researchers significantly improved the robot’s ability to maintain focus on relevant targets despite the visual complexity.
To further improve reliability, the vision system incorporates an advanced attention module designed to filter out irrelevant background information. This is particularly important when harvesting green crops, such as unripened tomatoes or peppers, which can easily blend into the surrounding foliage. The attention module acts as a visual filter, helping the robot distinguish the texture and shape of the fruit from the similar colors of the leaves. Additionally, the team utilized a technique known as hard-negative learning, where the model was intentionally exposed to images of non-target objects such as support wires, shadows, and metallic greenhouse structures. By teaching the system what it should not pick, the researchers were able to drastically reduce the occurrence of false positives. This proactive approach to error reduction ensures that the robot does not waste energy attempting to grasp a shadow or a stem, which ultimately speeds up the harvesting cycle and protects the delicate plants from accidental damage.
Validation Gates: Bridging Perception and Physical Action
The most significant innovation of the E-YOLOv11-RGBD framework is the implementation of a multi-stage “validation gate” pipeline that bridges the gap between vision and motion. Once a piece of fruit is identified, the system does not immediately trigger a harvesting command. Instead, it aligns the depth data from the camera with the color imagery and performs a noise-suppression check. By sampling the depth readings around the center of the target, the system can determine if a detection is legitimate or if it is being fooled by a reflection or a visual anomaly. If the depth data appears inconsistent or physically impossible, the target is rejected before the robot even begins to move its arm. This layer of verification is crucial for maintaining operational efficiency, as it prevents the robotic arm from executing a time-consuming movement toward a phantom target, a common failure mode in earlier iterations of autonomous agricultural systems that relied purely on 2D data.
After passing the visual validation, every potential target must undergo a rigorous kinematic feasibility check to ensure the robot can actually reach it. The system’s controller performs a series of rapid simulations, testing 121 different potential orientations for the gripper to see if any are physically possible within the joint limits of the UR3e arm. This step accounts for the physical reality that just because a fruit is visible does not mean it is accessible. Furthermore, the system measures the estimated width of the fruit against the 73-millimeter maximum opening of the OnRobot 2FG7 gripper. Only if a target is confirmed as both reachable and graspable does the controller issue the final command for the arm to move. This failure-driven approach ensures that the robot’s “hands” are always capable of performing what its “eyes” see, effectively transforming a standard image detection into a highly reliable and actionable harvesting instruction that respects the physical limits of the machine.
Empirical Performance: Evaluating Harvesting Success Rates
The effectiveness of this integrated framework was demonstrated through extensive physical trials involving 240 harvesting attempts across three distinct crop types: tomatoes, strawberries, and peppers. The results revealed a dramatic improvement over standard systems that lack an integrated validation pipeline. While the baseline model achieved a success rate of only 51.7%, the enhanced E-YOLOv11-RGBD system reached an impressive 92.5% success rate. The most notable breakthrough occurred in the strawberry category, which has traditionally been difficult for robots due to the fruit’s small size and visual similarity to distant tomatoes. The baseline model failed to harvest a single strawberry in 40 attempts, whereas the new framework successfully picked 35 out of 40. This leap in performance proved that adding a layer of physical reasoning to the vision system is the key to unlocking the potential of robotic harvesting, particularly for smaller and more challenging crops that are prone to detection errors.
Beyond mere success rates, the researchers observed a significant improvement in operational speed and resource management. By implementing the validation gates, the average harvesting cycle time per fruit was reduced from 18.35 seconds to 17.20 seconds. This time savings occurred because the robot no longer spent several seconds attempting to calculate paths or move toward targets that were ultimately unreachable or incorrectly identified. In pepper harvesting, success rates jumped from 55% to 90%, highlighting how the system adapted to the varying shapes and sizes of larger fruit. Tomato harvesting remained highly effective in both versions, yet the refined system still showed a more fluid and direct approach to each target. These findings indicated that the integration of perception and action not only makes the robot more accurate but also more decisive. By reducing wasted motion, the system minimized mechanical wear and tear and lowered the total energy consumption required for a full day of greenhouse operations.
Strategic Directions: Solving Remaining Agricultural Hurdles
While the laboratory results were undeniably positive, the study highlighted several areas that required further refinement before the technology reached full commercial scale in the coming years. One primary limitation was the reliance on stable, artificial lighting and a fixed distance between the camera and the crops. Real-world greenhouses are far more dynamic, featuring swaying plants, shifting sunlight, and dense, overlapping foliage that can confuse even the most advanced sensors. Additionally, the specific gripper used in the trials had difficulty with unusually large or oddly shaped peppers, suggesting that a more adaptive or soft-robotic grasping solution would be necessary for diverse crop varieties. The researchers recognized that their current results provided a strong proof of concept, but they noted that the transition to more chaotic outdoor or commercial environments would demand even greater sensor fusion and more resilient software algorithms capable of handling high levels of environmental noise.
Looking forward, the focus moved toward developing more sophisticated, adaptive grasping techniques and integrating the vision system with mobile navigation for continuous operation. The team prioritized the optimization of the E-YOLOv11 software to run on lower-power, portable hardware, which would allow for easier deployment across a wide range of existing farming machinery. By addressing the physical constraints of the robot alongside the visual processing, the research provided a clear blueprint for closing the gap between seeing and picking. It was concluded that the success of future agricultural automation would depend on this holistic approach, treating the robot as a unified entity rather than a collection of separate parts. As these systems are refined through 2027 and 2028, the potential for a fully autonomous workforce to stabilize global food production became increasingly tangible, offering a sustainable path forward in the face of the ongoing agricultural labor crisis.
