|
0
|
1 title: Service robot system with an informationally structured environment
|
|
|
2 author: Tatsuki IHA, Nozomi TERUYA
|
|
|
3 profile: Kono lab
|
|
|
4 lang: English
|
|
|
5 code-engine: coderay
|
|
|
6
|
|
|
7 # 1. Introduction
|
|
|
8 - aging of the population is a common problem in modern societies, and rapidly aging populations and declining birth rates have become more serious in recent years
|
|
|
9 - for instance, the manpower shortage in hospitals and elderly care facilities has led to the deterioration of quality of life for elderly individuals
|
|
|
10 - robot technology is expected to play an important role in the development of a healthy and sustainable society
|
|
|
11 - in particular, daily life assistance for elderly individuals in hospitals and care facilities is one of the most urgent and promising applications for service robots
|
|
|
12
|
|
|
13 # 1. Introduction
|
|
|
14 - for a service robot, information about its surrounding, such as the positions of objects, furniture, humans, and other robots is indispensable for safely performing proper service tasks
|
|
|
15 - however, current sensing technology, especially for cases of robots equipped with external sensors, is not good enough to complete these tasks satisfactorily
|
|
2
|
16 - for example, a vision system is susceptible to changes in lighting conditions and the appearances of objects. moreover, the field of vision is rather narrow
|
|
0
|
17
|
|
|
18 # 1. Introduction
|
|
2
|
19 - although occlusions can be partly solved by sensors on a mobile robot, background changes and unfavorable vibrations of a robot body make processes more difficult
|
|
|
20 - in addition, the payload of a robot is not so high and computer resources are also limited
|
|
0
|
21
|
|
|
22 # 1. Introduction
|
|
2
|
23 - fixed sensors in an environment are more stable and can more easily gather information about the environment
|
|
|
24 - if a sufficient number of sensors can be embedded in the environment in advance, occlusion is no longer a crucial problem
|
|
|
25 - information required to perform tasks is acquired by distributed sensors and transmitted to a robot on demand
|
|
|
26 - the concept of making an environment smarter rather than the robot is referred to as an informationally structured environment
|
|
0
|
27
|
|
|
28 # 1. Introduction
|
|
|
29 - an informationally structured environment is a feasible solution for introducing service robots into our daily lives using current technology
|
|
|
30 - several systems that observe human behavior using distributed sensor systems and provide proper service tasks according to requests from human or emergency detection, which is triggered automatically, have been proposed
|
|
|
31 - several service robots that act as companions to elderly people or as assistants to humans who require special care have been developed
|
|
|
32
|
|
|
33 # 1. Introduction
|
|
|
34 - we also have been developing an informationally structured environment for assisting in the daily life of elderly people in our research project, i.e., the robot town project
|
|
2
|
35 - the goal of this project is to develop a distributed sensor network system covering a townsize environment consisting of several houses, buildings, and roads, and to manage robot services appropriately by monitoring events that occur in the environment
|
|
0
|
36
|
|
|
37 # 1. Introduction
|
|
|
38 - events sensed by an embedded sensor system are recorded in the town management system (TMS)
|
|
|
39 - and appropriate information about the surroundings and instructions for proper services are provided to each robot
|
|
|
40
|
|
|
41 # 1. Introduction
|
|
2
|
42 - we also have been developing an informationally structured platform in which distributed sensors and actuators are installed to support an indoor service robot
|
|
0
|
43 - objects embedded sensors and rfid tags, and all of the data are stored in the TMS database
|
|
2
|
44 - a service robot performs various service tasks according to the environmental data stored in the TMS database in collaboration with distributed sensors and actuators, for example, installed in a refrigerator to open a door
|
|
0
|
45
|
|
|
46 # 1. Introduction
|
|
2
|
47 - we herein introduce a new town management system called the ROS-TMS
|
|
0
|
48 - in this system, the robot operating system (ROS) is adopted as a communication framework between various modules, including distributed sensors, actuators, robots, and databases
|
|
|
49
|
|
|
50 # 1. Introduction
|
|
|
51 - thanks to the ROS, we were able to develop a highly flexible and scalable system
|
|
|
52 - adding or removing modules such as sensors, actuators, and robots, to or from the system is simple and straightforward
|
|
2
|
53 - parallelization is also easily achievable
|
|
0
|
54
|
|
|
55 # 1. Introduction
|
|
|
56 - we herein report the followings
|
|
|
57 - introduction of architecture and components of the ROS-TMS
|
|
|
58 - object detection using a sensing system of the ROS-TMS
|
|
2
|
59 - fetch-and-give task using the motion planning system of the ROS-TMS
|
|
0
|
60
|
|
|
61 # 1. Introduction
|
|
2
|
62 - the remainder of the present paper is organized as follows
|
|
0
|
63 - section 2 : presenting related research
|
|
|
64 - section 3: we introduce the architecture and components of the ROS-TMS
|
|
|
65 - section 4: we describe the sensing system of the ROS-TMS for processing the data acquired from various sensors
|
|
|
66 - section 5: describes the robot motion planning system of the ROS-TMS used to design the trajectories for moving, gasping, giving, and avoiding obstacles using the information on the environment acquired by the sensing system
|
|
|
67 - section 6: we present the experimental results for service tasks performed by a humanoid robot and the ROS-TMS
|
|
2
|
68 - section 7: concludes the paper
|
|
0
|
69
|
|
|
70 # 2. Related research
|
|
|
71 - a considerable number of studies have been performed in the area of informationally structured environments/spaces to provide human-centric intelligent services
|
|
|
72 - informationally structured environments are referred to variously as home automation systems, smart homes, ubiquitous robotics, kukanchi, and intelligent spaces, depending on the field of research and the professional experience of the researcher
|
|
|
73
|
|
|
74 # 2. Related research
|
|
|
75 - home automation systems or smart homes are popular systems that centralize the control of lighting, heating, air conditioning, appliances, and doors, for example, to provide convenience, comfort, and energy savings
|
|
|
76 - the informationally structured environment can be categorized in this system, but the system is designed to support not only human life but also robot activity for service tasks
|
|
|
77
|
|
|
78 # 2. Related research
|
|
|
79 - hashimoto and Lee proposed an intelligent space in 1996
|
|
|
80 - intelligent spaces (iSpace) are rooms or areas that are equipped with intelligent devices, which enable spaces to perceive and understand what is occurring within them
|
|
|
81 - these intelligent devices have sensing, processing, and networking functions and are referred to as distributed intelligent networked devices (DINDs)
|
|
|
82 - one DIND consists of a CCD camera to acquire spatial information and a processing computer, which performs data processing and network interfacing
|
|
|
83 - these devices observe the position and behavior of both human beings and robots coexisting in the iSpace
|
|
|
84
|
|
|
85 # 2. Related research
|
|
|
86 - the concept of a physically embedded intelligent system (PEIS) has been introduced in 2005
|
|
|
87 - PEIS involves the intersection and integration of three research areas: artificial intelligence, robotics, and ubiquitous computing
|
|
|
88 - anything that consists of software components with a physical embodiment and interacts with the environment through sensors or actuators/robots is considered to be a PEIS, and a set of interconnected physically embedded intelligent systems is defined as a PEIS ecology
|
|
|
89 - tasks can be achieved using either centralized or distributed approaches using the PEIS ecology
|
|
|
90
|
|
|
91 # 2. Related research
|
|
|
92 - Ubiquitous robotics involves the design and deployment of robots in smart network environments in which everything is interconnected
|
|
|
93 - define three types of Ubibots: software robots (Sobots), embedded robots (Embots), and mobile robots (Mobots), which can provide services using various devices through any network, at any place and at any time in a ubiquitous space (u-space)
|
|
|
94
|
|
|
95 # 2. Related research
|
|
|
96 - Embots can evaluate the current state of the environment using sensors, and convey that information to users
|
|
|
97 - Mobots are designed to provide services and explicitly have the ability to manipulate u-space using robotic arms
|
|
|
98 - Sobot is a virtual robot that has the ability to move to any location through a network and to communicate with humans
|
|
2
|
99 - The present authors have previously demonstrated the concept of a PIES using Ubibots in a simulated environment and u-space
|
|
0
|
100
|
|
|
101 # 2. Related research
|
|
|
102 - RoboEarth is essentially a World Wide Web for robots, namely, a giant network and database repository in which robots can share information and learn from each other about their behavior and their environment
|
|
|
103 - the goal of RoboEarth is to allow robotic systems to benefit from the experience of other robots
|
|
|
104
|
|
|
105 # 2. Related research
|
|
2
|
106 - the informationally structured environment/space (also referred to as Kukanchi, a Japanese word meaning interactive human-space design and intelligence) has received a great deal of attention in robotics research as an alternative approach to the realization of a system of intelligent robots operating in our daily environment
|
|
|
107 - human-centered systems require, in particular, sophisticated physical and information services, which are based on sensor networks, ubiquitous computing, and intelligent artifacts
|
|
|
108 - information resources and accessibility within an environment are essential for people and robots
|
|
|
109 - the environment surrounding people and robots should have a structured platform for gathering, storing, transforming, and providing information
|
|
0
|
110 - such an environment is referred to as an informationally structured space
|
|
|
111
|
|
|
112 # 2. Related research
|
|
|
113 - in section 5, we present a coordinate motion planning technique for a fetch-and-give including handing over an object to a person
|
|
|
114 - the problem of handing over an object between a human and a robot has been studied in HumanRobot Interaction (HRI)
|
|
|
115
|
|
|
116 # 2. Related research
|
|
|
117 - the work that is closest to ours is the one by Dehais et al
|
|
|
118 - in their study, physiological and subjective evaluation for a handing over task was presented
|
|
|
119 - the performance of hand-over tasks were evaluated according to three criteria: legibility, safety and physical comfort
|
|
|
120 - these criteria are represented as fields of cost functions mapped around the human to generate ergonomic hand-over motions
|
|
|
121 - although their approach is similar to our approach, we consider the additional criteria, that is, the manipulability of both a robot and a human for a comfortable and safety fetch-and-give task
|
|
|
122
|
|
|
123 # 2. Related research
|
|
|
124 - the problem of pushing carts using robots has been reported in many studies so far
|
|
|
125 - the earlier studies in pushing a cart were reported using a single manipulator mounted on a mobile base
|
|
|
126
|
|
|
127 # 2. Related research
|
|
|
128 - the problem of towing a trailer has also been discussed as an application of a mobile manipulator and a cart
|
|
2
|
129 - this work is close to the approach in this paper, however, a pivot point using a cart is placed in front of the robot in our technique
|
|
0
|
130
|
|
|
131 # 2. Related research
|
|
2
|
132 - the work that is closest to ours is the one by Scholz et al
|
|
0
|
133 - they provided a solution for real time navigation in a cluttered indoor environment using 3D sensing
|
|
|
134
|
|
|
135 # 2. Related research
|
|
2
|
136 - many previous works focus on the navigation and control problems for movable objects
|
|
0
|
137 - On the other hand, we consider the problem including handing over an object to a human using a wagon, and propose a total motion planning technique for a fetch-and-give task with a wagon
|
|
|
138
|
|
|
139 # 3. Overview of the ROS-TMS
|
|
|
140 - in the present paper, we extend the TMS and develop a new Town Management System called the ROS-TMS
|
|
|
141 - This system has three primary components
|
|
|
142 - real-world
|
|
|
143 - database
|
|
|
144 - cyber-world
|
|
|
145
|
|
|
146 <div style="text-align: center;">
|
|
|
147 <img src="./images/fig3.svg" alt="message" width="600">
|
|
|
148 </div>
|
|
|
149
|
|
|
150 # 3. Overview of the ROS-TMS
|
|
2
|
151 - events occurring in the real world, such as user behavior or user requests, and the current situation of the real world are sensed by a distributed sensing system
|
|
0
|
152 - the gathered information is then stored in the database
|
|
|
153
|
|
|
154 <div style="text-align: center;">
|
|
|
155 <img src="./images/fig3.svg" alt="message" width="600">
|
|
|
156 </div>
|
|
|
157
|
|
|
158 # 3. Overview of the ROS-TMS
|
|
|
159 - appropriate service commands are planned using the environmental information in the database and are simulated carefully in the cyber world using simulators, such as choreonoid
|
|
|
160 - service tasks are assigned to service robots in the real world
|
|
|
161
|
|
|
162 <div style="text-align: center;">
|
|
|
163 <img src="./images/fig3.svg" alt="message" width="600">
|
|
|
164 </div>
|
|
|
165
|
|
|
166 # 3. Overview of the ROS-TMS
|
|
|
167 - the following functions are implemented in the ROS-TMS
|
|
|
168 1. Communication with sensors, robots, and databases
|
|
|
169 2. Storage,revision,backup,and retrieval of real-time information in an environment
|
|
|
170 3. Maintenance and providing information according to individual IDs assigned to each object and robot
|
|
|
171 4. Notification of the occurrence of particular predefined events, such as accidents
|
|
|
172 5. Task schedule function for multiple robots and sensors
|
|
|
173 6. Human-system interaction for user requests
|
|
|
174 7. Real-time task planning for service robots
|
|
|
175
|
|
|
176 # 3. Overview of the ROS-TMS
|
|
|
177 - ROS-TMS has unique features, as described below
|
|
|
178 - Scalability
|
|
|
179 - ROS-TMS is designed to have high scalability so that it can handle not only a single room but also a building and a town
|
|
|
180 - Diversity
|
|
|
181 - diversity: The ROS–TMS supports a variety of sensors and robots
|
|
|
182 - for instance, Vicon MX (Vicon Motion Systems Ltd.), TopUrg (Hokuyo Automatic), Velodyne 32e (Velodyne Lidar), and Oculus Rift (Oculus VR) are installed in the developed informationally structured platform
|
|
|
183 - Safety
|
|
|
184 - data gathered from the real world is used to perform simulations in the cyber world in order to evaluate the safety and efficiency of designed tasks
|
|
|
185
|
|
|
186 # 3. Overview of the ROS-TMS
|
|
|
187 - ROS-TMS has unique features, as described below
|
|
|
188 - Privacy protection
|
|
|
189 - one important restriction in our intelligent environment is to install a small number of sensors to avoid interfering with the daily activity of people and to reduce the invasion of their privacy as far as possible
|
|
|
190 - we do not install conventional cameras in the environment
|
|
|
191 - Economy
|
|
|
192 - sensors installed in an environment can be shared with robots and tasks, and thus we do not need to equip individual robots with numerous sensors
|
|
|
193 - in addition, most sensors are processed by low-cost single-board computers in the proposed system
|
|
|
194 - this concept has an advantage especially for the system consisting of multiple robots since robots can share the resources in the environment
|
|
|
195
|
|
|
196 # 3. Overview of the ROS-TMS
|
|
|
197 - some features such as modularity, scalability, and diversity owe much to ROS’s outstanding features
|
|
|
198 - on the other hand, economical or processing efficiency strongly depends on the unique features of ROS-TMS, since various information gathered by distributed sensor networks is structured and stored to the database and repeatedly utilized for planning various service tasks by robots or other systems
|
|
|
199
|
|
|
200 # 3. Overview of the ROS-TMS
|
|
|
201 - ROS-TMS is composed of five components
|
|
|
202 - User
|
|
|
203 - Sensor
|
|
|
204 - Robot
|
|
|
205 - Task
|
|
|
206 - Data
|
|
|
207 - components are also composed of sub-modules
|
|
|
208 - such as the User Request sub-module for the user component
|
|
|
209
|
|
|
210 <div style="text-align: center;">
|
|
|
211 <img src="./images/fig4.svg" alt="message" width="450">
|
|
|
212 </div>
|
|
|
213
|
|
|
214 # 4. Sensing system
|
|
|
215 - sensing system (TMS_SS) is a component of the ROS-TMS that processes the data acquired from various environment sensors
|
|
|
216 - TMS_SS is composed of three sub-packages
|
|
|
217 - Floor sensing system (FSS)
|
|
|
218 - Intelligent cabinet system (ICS)
|
|
|
219 - Object detection system (ODS)
|
|
|
220
|
|
|
221 # 4.1 Floor sensing system(FSS)
|
|
|
222 - current platform is equipped with a floor sensing system to detect objects on the floor and people walking around
|
|
|
223 - this sensing systems is composed of a laser range finder located on one side of the room and a mirror installed along another side of the room
|
|
|
224 - this configuration allows a reduction of dead angles of the LRF and is more robust against occlusions
|
|
|
225
|
|
|
226 <div style="text-align: center;">
|
|
|
227 <img src="./images/fig6.svg" alt="message" width="600">
|
|
|
228 </div>
|
|
|
229
|
|
|
230 # 4.1 Floor sensing system(FSS)
|
|
|
231 - people tracking is performed by first applying static background subtraction and then extracting clusters in the remainder of the measurements
|
|
|
232 - this system can measure the poses of the robot and movable furniture such as a wagon using tags, which have encoded reflection patterns optically identified by the LRF
|
|
|
233
|
|
|
234 <div style="text-align: center;">
|
|
|
235 <img src="./images/fig6.svg" alt="message" width="600">
|
|
|
236 </div>
|
|
|
237
|
|
|
238 # 4.2. Intelligent cabinet system (ICS)
|
|
|
239 - the cabinets installed in the room are equipped with RFID readers and load cells to detect the types and positions of the objects in the cabinet
|
|
1
|
240 - every object in the environment has an RFID tag containing a unique ID that identifies it
|
|
|
241 - this ID is used to retrieve the attributes of the object, such as its name and location in the database
|
|
|
242
|
|
|
243 # 4.2. Intelligent cabinet system (ICS)
|
|
|
244 - using the RFID readers, we can detect the presence of a new object inside the cabinet
|
|
|
245 - the load cell information allows us to determine its exact position inside the cabinet
|
|
|
246
|
|
|
247 <div style="text-align: center;">
|
|
|
248 <img src="./images/fig7.svg" alt="message" width="1200">
|
|
|
249 </div>
|
|
|
250
|
|
|
251 # 4.3. Object detection system (ODS)
|
|
|
252 - available for detecting objects such as those placed on a desk, the object detection system using a RGB-D camera on a robot is provided in this platform
|
|
2
|
253 - in this system, a newly appeared object or movement of an object is detected as a change in the environment
|
|
1
|
254
|
|
|
255 <div style="text-align: center;">
|
|
|
256 <img src="./images/fig8.svg" alt="message" width="600">
|
|
|
257 </div>
|
|
|
258
|
|
|
259 # 4.3. Object detection system (ODS)
|
|
2
|
260 - the steps of the change detection process are as follows
|
|
1
|
261 1. Identification of furniture
|
|
|
262 2. Alignment of the furniture model
|
|
|
263 3. Object extraction by furniture removal
|
|
|
264 4. Segmentation of objects
|
|
|
265 5. Comparison with the stored information
|
|
|
266
|
|
|
267 # 4.3.1. Identification of furniture
|
|
|
268 - possible to identify furniture based on the position and posture of robots and furniture in the database
|
|
2
|
269 - using this information, robot cameras determine the range of the surrounding environment that is actually being measured
|
|
1
|
270 - the system superimposes these results and the position information for furniture to create an updated furniture location model
|
|
|
271
|
|
|
272 # 4.3.1. Identification of furniture
|
|
|
273 - the point cloud (Fig. 9a) acquired from the robot is superimposed with the furniture’s point cloud model (Fig. 9b)
|
|
|
274 - After merging the point cloud, the system deletes all other points except for the point cloud model for the furniture and limits the processing range from the upcoming steps
|
|
|
275
|
|
|
276 <div style="text-align: center;">
|
|
|
277 <img src="./images/fig9.svg" alt="message" width="800">
|
|
|
278 </div>
|
|
|
279
|
|
|
280 # 4.3.2. Alignment of the furniture model
|
|
2
|
281 - We scan twice for gathering point cloud datasets of previous and current scene
|
|
1
|
282 - in order to detect the change in the newly acquired information and stored information, it is necessary to align two point cloud datasets obtained at different times because these data are measured from different camera viewpoints
|
|
|
283
|
|
|
284 # 4.3.2. Alignment of the furniture model
|
|
|
285 - in this method, we do not try to directly align the point cloud data, but rather to align the data using the point cloud model for the furniture
|
|
|
286 - the reason for this is that we could not determine a sufficient number of common areas by simply combining the camera viewpoints from the two point cloud datasets and can also reduce the amount of information that must be stored in memory
|
|
|
287 - using the aligned point cloud model, it is possible to use the point cloud data for objects located on the furniture, without having to use the point cloud data for furniture from the stored data
|
|
|
288 - alignment of the furniture model is performed using the ICP algorithm
|
|
|
289
|
|
|
290 # 4.3.3. Object extraction by furniture removal
|
|
|
291 - after alignment, all points corresponding to furniture are removed to extract an object
|
|
|
292 - the system removes furniture according to segmentation using color information and three-dimensional positions
|
|
|
293 - more precisely, the point cloud is converted to a RGB color space and then segmented using a region-growing method
|
|
|
294
|
|
|
295 # 4.3.3. Object extraction by furniture removal
|
|
2
|
296 - each of the resulting segments is segmented based on the XYZ space
|
|
|
297 - system then selects only those segments that overlap with the model and then removes these segments
|
|
1
|
298
|
|
|
299 <div style="text-align: center;">
|
|
|
300 <img src="./images/fig10.svg" alt="message" width="800">
|
|
|
301 </div>
|
|
|
302
|
|
|
303 # 4.3.4. Segmentation of objects
|
|
|
304 - after performing the until now processing, only the points associated with objects placed on furniture remain
|
|
|
305 - these points are further segmented based on XYZ space
|
|
|
306 - the resulting segments are stored in the database
|
|
|
307
|
|
|
308 # 4.3.5. Comparison with the stored infomation
|
|
|
309 - finally, the system associates each segment from the previously stored information with the newly acquired information
|
|
|
310 - system finds the unmatched segments and captures the movement of objects that has occurred since the latest data acquisition
|
|
|
311 - segments that did not match between the previous dataset and the newly acquired dataset, reflect objects that were moved, assuming that the objects were included in the previously stored dataset
|
|
2
|
312 - segments that appear in the most recent dataset, but not in the previously stored dataset, reflect objects that were recently placed on the furniture
|
|
1
|
313
|
|
|
314 # 4.3.5. Comparison with the stored infomation
|
|
2
|
315 - set of segments that are included in the association process are determined according to the center position of segments
|
|
|
316 - for the segments sets from the previous dataset and the newly acquired dataset, the association is performed based on a threshold distance between their center positions, considering the shape and color of the segments as the arguments for the association
|
|
1
|
317
|
|
|
318 # 4.3.5. Comparison with the stored infomation
|
|
|
319 - use an elevation map that describes the height of furniture above the reference surface level to represent the shape of the object
|
|
|
320 - reference surface level of furniture is, more concretely, the top surface of a table or shelf, the seat of a chair
|
|
|
321 - elevation map is a grid version of the reference surface level and is a representation of the vertical height of each point with respect to the reference surface level on each grid
|
|
|
322
|
|
|
323 <div style="text-align: center;">
|
|
|
324 <img src="./images/fig11.svg" alt="message" width="800">
|
|
|
325 </div>
|
|
|
326
|
|
|
327 # 4.3.5. Comparison with the stored infomation
|
|
2
|
328 - comparison is performed on the elevation map for each segment, taking into consideration the variations in size, the different values obtained from each grid, and the average value for the entire map
|
|
1
|
329 - the color information used to analyze the correlation between segments is the hue (H) and saturation (S)
|
|
|
330 - Using these H-S histograms, the previous data and the newly acquired data are compared, allowing the system to determine whether it is dealing with the same objects
|
|
|
331
|
|
|
332 <div style="text-align: center;">
|
|
|
333 <img src="./images/fig11.svg" alt="message" width="800">
|
|
|
334 </div>
|
|
|
335
|
|
|
336 # 4.3.5. Comparison with the stored infomation
|
|
|
337 - bhattacharyya distance BC(p, q) within H-S histograms(p,q) is used for determining the similarity between histograms and is calculated according to Eq. (1)
|
|
|
338 - once distance values are calculated, the object can be assumed to be the same as for the case in which the degree of similarity is equal to or greater than the threshold value
|
|
|
339
|
|
|
340 <div style="text-align: center;">
|
|
|
341 <img src="./images/eq1.svg" alt="message" width="800">
|
|
|
342 </div>
|
|
|
343
|
|
|
344
|
|
|
345 # 4.3.5. Comparison with the stored infomation
|
|
|
346
|
|
|
347 <div style="text-align: center;">
|
|
|
348 <img src="./images/fig12.svg" alt="message" width="800">
|
|
|
349 </div>
|
