Industrial robots are widely used in automobile assembly lines. These robots are called industrial robots since their goal is to follow some pre-taught paths or way points, which is essentially the same as automation machines. The tasks performed by these robots include welding, painting, pick-and-place in structured environment (e.g., glass installation, door installation, and nut/bolt fastening), etc.

The tasks performed by industrial robots are exciting. On the other hand, there are lots of tasks remained to be done by human workers. These tasks include (1) picking parts from cluster, (2) reorienting parts to certain poses, and (3) force-based assembly. These tasks cannot be done using industrial robots and teach pendants since: (1) The parts shapes and physical properties are varying. It is difficult to manually specify the grasping strategies for infinite number of parts. (2) The initial poses and goal poses of manipulated parts are changing. It is difficult to teach the robots all motions to reorient different parts. (3) The first two processes lead to variety in force control and assembly, which also make pre-teaching difficult.

One advisable way to automate the remaining tasks is to develop intelligent robots. The fundamental technique includes (1) computer vision for object recognition, (2) object analyzer for grasp planning, (3) motion planning, (4) force analysis and assembly planning, and (5) machine learning.

The manipulation group at National Institute of Advanced Industrial Science and Technology (AIST), Japan, is developing intelligent robots to assembly objects. Over the years, we have developed software for all fundamental technique.

(1) We developed object recognition technique by using point clouds collected from structure light- based depth sensors. The developed platform has high precision as it uses point clouds and geometric constraints in the robot execution phase to avoid unexpected noises [1]. We also developed multi-view vision systems by considering the results of previous manipulation [2].

(2) We developed object analyzer and gripping planner by segmenting and clustering objects surface meshes, as well as considering torque and object surface properties. The planner can find candidate grasps for parallel industrial robotic grippers with probabilistic completeness. Some technical details are available in [3,4].

(3) We developed middle and low-level manipulation planners to generate robot motions [5]. In the middle level, we use regrasp and handover technique to compute the intermediate states of robotic grippers and objects, and generate pre and post robot configurations using motion primitives to orchestrate the computed intermediate states. In the low level, we use probabilistic motion planning algorithms like Rapidly-exploring Random Tree (RRT) to plan more detailed collision-free and kinematic available motion sequences between the intermediate configurations.

(4) We developed assembly planners by analyzing the constraints between objects [6]. The assembly planner could automatically generate assembly sequences (order of assembly), assembly directions, and available grasps. The assembly planner is a high-level planning component and could work together with middle and low-level planning components discussed in item (3), and the vision system discussed in item (1), to perform integrated assembly and motion planning.

(5) We are exploring cutting-edge technique like machine learning to solve difficult problems like picking from clutter [7]. The technique in items (1) to (4) was conventional and the goal was to find collisionfree and kinematic available robot and hand configurations. Instead of the conventional approach, we developed new picking solutions by taking advantages of obstacles. Our assumption was that collision might not be negative and might even be helpful. We used machine learning to learn how to take advantages of the obstacles.

(6) We analyzed the states of force sensors and used state machines to perform the last step of assembly. The details were published in [8] where a dual-arm robot assemble two objects connected using snap joints. Some similar technique is available in [9,10] which used force sensors and state machines to fasten bolts. This technique for object manipulation can be directly used by intelligent robots in automobile industry. Some of them have been implemented for the specific tasks. One example is to assembly exhaustive parts [11]. Conventional solution is human workers pick out components from feeding trays, place them down on fixtures, and command industrial robots to weld them. The industrial robots cannot be taught to perform the pick-andplace tasks as the components are at various poses. Using the newly developed visual detection and grasp and manipulation planning technique, intelligent robots take all roles. They detect the components and pick them out directly from the tray using vision technique. They reorient and place down the components on fixtures, and weld them, using grasping and manipulation planning technique. Human workers are not directly involved in the process. The technique is superior in that it is adaptive to non-structured environment. It has high precision vision based on low-cost structure light-based sensors, and has high success rate in planning by using dual-arm humanoid robots.

Although the fundamental technique has been developed, there remains several challenges. For one thing, each of the fundamental technique has some bottlenecks: (1) Structure light-based 3D vision is not applicable to non-lambert surfaces, non-reflective surfaces, and transparent surfaces. Parts with these surfaces widely exist in automobile assembly. Recognizing them remains challenging. (2) Probabilistically complete gripping planner is only applicable to parallel robotic grippers. It is difficult to plan complete grasps for robot grippers with varying finger shapes [12,13]. It is also difficult to plan complete grasps for robot hands with more than two fingers (although non-complete planners are available [14,15]. (3) Planning optimal sequences by considering many more constraints like energy consumption and by taking advantages of environmental structures like supporting pins [16] are computational infeasible. There could be infinite combinatorics which require fast pruning and carefully designed heuristics or heuristic database. (4) Collecting training data for realworld robots is difficult. Although some inspiring examples have been done in [7,17], it is still difficult to allow failures and program robots to learn from failures online in real factories. (5) Force sensing and state analyzing are highly dependent on specific objects. Automatically performing state analyzing requires allowance of failures, which is still lab work (as discussed in item (4).

For the other, it is difficult to integrate the available technique. The integration requires a large software platform spanning from computational geometry to force analysis and state machines. It also requires the support of databases. We have started a project named PyHiro at Github [18]. The goal of this project is to develop an integrated assembly system using the available technique. The platform is under the support of MySQL database to save robots, grippers, intermediate states, etc. Integrating and coordinating the roles of the various technique in the platform is an open problem.

Using the available technique, we could develop a highly automatic system where the input is an assembled object model (geometric information and physical properties) and the relative relations between the parts in the model, the output is a sequence of robot motions that pick out parts from boxes, reorient parts to specific poses, and assemble them. The robot analyzes the environment and decides whether to take advantages of environmental structures during the assembly process. It could also provide suggestions to factory managers about how to set up the environment to facilitate the assembly line.

The system will be suitable for varying parts and objects, especially for the assembly of the coming connected and autonomous cars, which have more complicated components to assembly. Compared with existing industrial robots, the system will further reduce the necessary human interventions and replace the human workers in nonteachable tasks with intelligent robots. The input to the system could be from CAD software or human demonstration. The system could be reconfigured quickly for different parts. The automobile parts produced by manufacturers will be directly supplied to the system without unboxing. The system computes the assembly sequences without human intervention for underlying details. This paper presented the technique and discussed the difficulties to implement this kind of system. We expect it will provide a study motivation for researchers in these technique and systems.