- 1 Team Members
- 2 Project Overview
- 3 Mechanical Components
- 4 Hardware
- 5 Software
- 6 Results
- 7 Challenges
- 8 References
- 9 Future Improvements
- 10 Project Links
- Aaron Hanna - ECE Department
- Stephen Lien - MAE Department
- Brian Rush - ECE Department
Our goal is to have the RC car localize and map a designated area and avoid the obstacles within the space through SLAM algorithm and a 3D camera. SLAM, or simultaneous localization and mapping, aims to construct a map of an unknown environment while also keeping track of the vehicle position within the environment. The 3D camera consists of two lens and a infrared sensor. The two lens constructs the 3D environment the camera sees, while the infrared sensor detects the depth of the things within view. With the two combined, we were able to avoid obstacle while mapping the environment in our vehicle path
- Detect objects in path of vehicle using RealSense camera
- Avoid all obstacles
- Generate map and odometry using a SLAM implementation (RTABMAP)
Nice to haves
- Determine if detected obstacle is small enough to drive over
- Intelligently route around obstacles to minimize blind spot
- Create a skinned map using RGB data
The base plate is fabricated using laser cut. The first iteration of the base plate was designed to hold the components and have some cable management features. The plate broke after an impact during our outdoor training session. The second iteration of the base plate features an array of screw mounts for the 3D camera in case we need to change the position of it. The cable management slots are designed to withstand the weight of the components. We also used a 1/4 inch thick acrylic to enhance its strength.
Intel RealSense Camera mount
The camera mount is 3D printed. The 3D camera mount is designed to with stand flat impact in the front to protect that camera. An issue with the design is the long flange from the mount slightly interfered with the camera vision. Nonetheless we eliminated the problem by cropping out the sides of the frame.
Nvidia Jetson Nano is our on board computer.
Intel RealSense 3D Camera
Our 3D camera is the Intel RealSense D435. It has 2 cameras and 1 infrared sensor.
The algorithm used for this project is RTABMAP. It is easy to set up and use with Ubuntu/ROS on the Jetson. Full instructions are available on the ROS wiki page. Once ROS Melodic is installed, we can easily add both the nodes we need on the Jetson: realsense_ros and rtabmap_ros - they can be install directly via apt
$ sudo apt install ros-melodic-realsense2-camera ros-melodic-rtabmap-ros
Running these nodes for the RealSense D435 is really easy. A tutorial is given on the ROS wiki. However, there is a lot of information there so I will copy the commands we used to get SLAM running below:
1. Start the camera
$ roslaunch realsense2_camera rs_camera.launch align_depth:=true
2. Start ROS
$ roslaunch rtabmap_ros rtabmap.launch rtabmap_args:="--delete_db_on_start" depth_topic:=/camera/aligned_depth_to_color/image_raw rgb_topic:=/camera/color/image_raw camera_info_topic:=/camera/color/camera_info approx_sync:=false
And that's it. We added additional arguments as needed to each node - e.g limiting the frame rate, enabling or disabling visualization on SLAM, etc. Additional arguments for these nodes are on their respective wikis. For anyone following this in the future, I recommend putting these into a launch file for ease of use.
This node is responsible for polling the camera and sending the images out to the topics
We had to limit the camera to 15Hz to avoid crashing - the Jetson was unable to keep up with 30Hz when SLAM was also running.
This node is where the mapping happens. It subscribes to the two camera topics, and will output a map and odometry data. This odometry data can be used in path planning and object avoidance.
Connecting to Donkey
There are a few classes that were added in order to connect ROS output to Donkey in a relatively clean way. At the top level, a new SLAM part was added. This part does no calculation or take any inputs, it will simply talk to ROS (in a background thread) and expose the camera and odometry data to other Donkey parts. This allows us to separate ROS and Donkey and allow each to run independently of each other.
Our Object Detection primarily uses OpenCV library in python to operator on images obtained from the Realsense camera. All of the examples below can be found in the Github link below.
Realsense Python Wrapper
1. Align RGB/Depth images
align = rs.align(rs.stream.color) frameset = align.process(frameset)
2. Post Processing Filters
aligned_depth_frame = rs.hole_filling_filter().process(aligned_depth_frame) aligned_depth_frame = rs.spatial_filter().process(aligned_depth_frame) aligned_depth_frame = rs.temporal_filter().process(aligned_depth_frame)
OpenCV Methods 1. Convert depth color image to grayscale (type of colormap is important)
imgray = cv2.cvtColor(self.depth_color, cv2.COLOR_BGR2GRAY)
2. Apply blurring and threshold
blurred = cv2.GaussianBlur(imgray, (5, 5), 0) thresh = cv2.threshold(blurred, 25, 255, cv2.THRESH_BINARY_INV) kernel = np.ones((5, 5), np.uint8)
3. Apply erosion/dilation morphology to eliminate noise
closing = cv2.morphologyEx(thresh, cv2.MORPH_CLOSE, kernel) # Erosion/Dilation and Closing Morphology
4. Find external contours
contours, hierarchy = cv2.findContours(closing.copy(), cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)
The object avoidance algorithm overlays a box over the image frames that outlines the width of the car. The object sent from the detection algorithm is then determined if it is on the left or right side of the vehicle path. The car steers the opposite direction of the obstacle until it is not in the vehicle path. The object avoidance class takes the box borders, vehicle throttle, and object dimensions as input and outputs the steering for the vehicle.
We were able to make our vehicle drive and avoid obstacles for a brief period. Due to the amount of processing that happens for every frame, we found out that it was unable to keep up with the speed it was traveling. Objects would appear and approach rapidly between frames and we found our car would often run into walls before too long. Ways to avoid this would be to find a way to do our processing for each frame faster so we could run a higher frame rate. Our object detection part would take ~300ms on average allowing us to avoid obstacles at a whopping 3Hz. This is unsatisfactory to avoid obstacles in real time.
Our results also showed that RTABMAP is simply unable to be run practically on the Jetson. I do not have accurate timing measurements on how quickly it was running, but visually it appeared to run at 2-3Hz. It will run, but the camera must be moved so slowly as to make it impractical to use on a robot. A different algorithm might be more efficient, but I don't believe any will run well enough to make is feasible. As with Donkey, this could be mitigated by running on a more powerful computer, but I think you would need a full-size system and not simply an onboard computer. It could be possible to send data to an external system (such as the GPU cluster) to run SLAM, and return the result back to the Jetson, but network delays would dictate if this is possible or not.
Making the object detection robust for any unknown object was one of the biggest challenges. In the end we used an algorithm that consisted of an threshold and erosion/dilation (morphologyEX) manipulation on the image. This ensured we isolated the data from the Realsense camera to exactly what we wanted to see. However, the processing power of the Jetson Nano could not keep up with these heavy OpenCV functions. One way to circumvent this was to reduce the ROI of the image as well as the resolution of the images from the camera. In addition, there was a lot of depth interference from the part of the case protruding as well as from the ground when the RC car was placed on the ground.
Computing power was a huge issue for us. The SLAM algorithm (limited to 15Hz) used up ~70% of all four cores on the Jetson. This doesn't leave a lot of overhead for Donkey, but fortunately Donkey doesn't take too much power and was still able to run. The larger issue was that since SLAM was forced to run so slowly, it was unable to run on the vehicle under autonomous driving. The car simply moved too fast and we could not slow the motor down any further without the car coming to a complete stop. We were able to generate a simple map driving the vehicle by hand, but it involved a lot of backing up and going very slowly in order to keep SLAM tracking at all times.
- We have ran into the issue of insufficient computing power from a single Jetson Nano, causing the algorithm to have lower frame rate than desired. In the future, teams could attempt to use two Jetson Nanos in serial connection or a stronger onboard computer with this algorithm