Big Machine Autonomy: A Growing Industry
Big machine autonomy is a big business. At CES 2025, this was in full display by providers like Oshkosh, John Deere, Caterpillar, and Komatsu. Their solutions address many challenges, such as safety, productivity, 24×7 asset utilization, and operational optimization. Most importantly, they help tackle the shortage of skilled human labor required to work in remote and physically challenging environments (e.g., extreme weather, smoke, dust, noise, vibrations, pollution, and high altitudes). Applications include tillage and seeding, excavation and transport of construction materials and mining ore, garbage hauling, sorting of recyclables, cargo handling and transport in airports, and eventually remotely automated construction on the moon!
The Role of Perception and Sensors in Autonomy
Physical perception of the environment in which these autonomous machines operate is critical for safety (of people and equipment), navigation, path planning, speed control, and optimizing operational protocols. Different sensor modalities are critical, including cameras, LiDAR, radar, GPS, IMUs and gyros, along with software and AI based fusion. The environments are extremely harsh (compared to the on-road automotive use cases) including wide operating temperature ranges (- 40C to 110C), extreme shock and vibration, mud splatter, dust, and mining ore contamination (which causes confusion in perception). Outer space operation adds other layers of harsh environment challenges. Optical sensors (cameras and LiDAR) are especially impacted from a ruggedness and data quality perspective and require special hardware and software strategies to ensure operational integrity.
Caterpillar: Pioneering Autonomy in Mining and Construction
Caterpillar has been a leader in autonomy for mining, quarrying, oil exploration, and construction for the past 25 years. LiDAR is essential for obstacle detection since its ~1.3M lbs trucks move at 40 mph and long range detection of obstacles is critical for adequate braking and steering decisions. Caterpillar started off using Velodyne LiDAR and later other suppliers like Leica and Ouster (Ouster acquired Velodyne 2 years ago) who supply products in the mining industry. Given the demanding operating environment, Caterpillar initiated internal LiDAR efforts in 2018 to overcome challenges with performance over wide temperature ranges, managing dusty and smoky environments and ensuring high lifetime and durability. Customized software and signal processing to filter false positives due to dust particles and deep ground ruts in oilfields is critical. Caterpillar now works with commercial LiDAR companies under license to customize and adapt their commercial-off-the-shelf (COTS) products for different applications.
Cat® 794 AC Electric Drive Large Mining Truck Operating Autonomously (Caterpillar)
Other Industry Leaders in Heavy Machine Autonomy
Other heavy machine autonomy companies like John Deere, Oshkosh, and Komatsu also use LiDAR technologies, typically using COTS supplied LiDAR but with the hardware hardened internally to survive the harsh operating environments (more demanding than public road vehicles). Specialized software and calibration procedures are used to address the effects of shock, vibration, dust, and moisture. Vehicle specific temperature-controlled platforms are used to compensate for wide temperature swings.
John Deere
John Deere generally uses an array of cameras (along with other sensors like radar and GPS) for agricultural applications like tillage and seeding in open fields. For orchard applications such as insecticide spraying, LiDAR plays a crucial role in mapping vegetation canopies and tree branches, as well as providing accurate localization and path planning for optimal coverage and efficiency. The impact of shock and vibration on data quality and sensor durability are of concern. Solid state LiDAR with no moving parts is desirable, although FMCW performance is not required because of the low speeds (< 5 mph).
The Autonomous 5ML Orchard Tractor for air blast spraying (John Deere)
Oshkosh
Oshkosh uses IMUs and gyros to filter out the effects of vibration on the LIDAR signal. Perception confusion from dust particles is addressed using machine learning. Given the wide range of autonomy applications Oskhosh addresses (airport logistics to garbage hauling trucks and recycling), customization is very important. Solid state and FMCW LiDAR are desirable moving forward.
JetDock® Technology to Enhance Airport Operations (Oshkosh Corporation)
Komatsu
Komatsu is engaged in various sectors like mining, oil exploration and construction – all of which incorporate autonomy at various levels. Every location and application comes with a simulated autonomy stack which is fine tuned for the complexities of weather, road conditions and customer specific requirements. The mining autonomy stack incorporates high precision GPS, camera, radar, and LiDAR sensors. LiDARs with self-cleaning windows and IMUs integrated close to the optical axis are especially important (the latter for more precise vibration filtering effects on the point cloud). Mounting locations are optimized based on the need to keep the lens surfaces clean and achieve the required Field of View (some of the vehicles can be 50 feet in height and require full visibility of the ground surface). Solid state and FMCW LiDAR are good to have but only as long as range, point density and FoV are not compromised. Expected LiDAR sensor lifetimes are 5 years. OTA software upgrades are essential as well as self-diagnostic features that provide advance warnings of impeding malfunction or failures which are very expensive.
The Future of Big Machine Autonomy: The AoT™ Revolution
The AoT™ (Autonomy of Things) revolution is progressing in areas that solve critical problems in harsh environments where labor is scarce, capital is expensive, and 24×7 operation is imperative. Advances in sensing, perception, localization, computing, and physical AI are making this revolution possible.
Sabbir Rangwala provides consulting services in perception, sensing, LiDAR and AI for enabling the Autonomy of Things (AoT™) revolution. He is also a Senior Contributor on Transportation and Innovation at Forbes.com.
