2019SpringTeam7
Project Team Members
- David Zhenan Yin - Electrical Engineering
- Anne Kimberly Edquilang - Mechanical Engineering
- Mayuki Adam Sasagawa - Mechanical Engineering
- Matias Ricardo Lee Choi - Mechanical Engineering
Project Overview
This project develops an autonomous car that reacts to objects in its surroundings. Real-time object detection is implemented to recognize common road objects and react as a human driver would. In the real world, autonomous cars need to be at least as safe as human drivers to be viable. They must emulate a human driver's response time (0.7-3 seconds to react, 2.3 seconds to hit brakes) to react quickly enough to avoid danger. Like human drivers, they also must react differently to every object. The system is implemented on our autonomously-driven car using one Raspberry Pi and one camera in order to minimize footprint and cost.
Main Components
Schematic
Hardware
- Lazer cut parts
Acrylic top plate
- 3D printed parts
Dual Pi Camera Mount
- Mascot
Drive Train
Challenges
| Challenge | Solution |
|---|---|
| Car not driving straight | Shaft realignment |
| Steering and throttle not working, burned ESC | Replaced ESC |
| Steering stopped working, throttle works, overheating servo motor | Replaced servo motor |
| Raspberry pi does not boot | Replace pi |
What we learned
When we force the servo motor to steer too much then turn off the pi/car, it could cause the motor to be stuck constantly running. The DC motor inside the servo could have been burned that way.
It is crucial to make sure all of the car's hardware is perfect before we start programming or training the car. For example, all the images and training might be for nothing if the camera mount is not secure and consistent.
Software
Using the classroom wifi and SSH to access the raspberry pi, we installed Donkey, Tensorflow, and paired the PS3 controller with the raspberry pi
Calibration Values
- Throttle:
- forward: 440
- neutral: 410
- reverse: 380
- forward: 440
- Steering:
- left: 200
- neutral: 280
- right: 360
- left: 200
- Autonomous Driving Training:
Roughly 27,000 images were recorded from driving 40 laps around the track. Those images were used to develop the autonomous driving model.
After training all the data collected on the GPU cluster, we were able to obtain a relatively high accuracy model.
Challenges
| Challenge | Solution |
|---|---|
| Could not access directory | Directory name has space, changed it to underscore |
| Protobuf installation was time-consuming | Patience |
| RaspbianLite not compatible for screen share on Mac |
|
Milestones
Indoor Autonomous Driving
6 Autonomous Indoor Laps
Outdoor Autonomous Driving
3 Autonomous Outdoor Laps
Object Recognition
Software and Hardware List
The following are needed to complete this project.
- Raspberry Pi 3B or 3B+
- 32GB MicroSD card
- Any high-resolution Pi Camera
- Raspbian Stretch with desktop and recommended software(Link to download)
- TensorFlow
- OpenCV
Steps Overview
- Raspberry Pi setup
- Install TensorFlow, OpenCV, and Protobuf
- Setup Tensorflow directory structure and path variable
- Download model from model zoo
- Start object detection
For this project, we were able to follow the guide provided by EdjeElectronics. We used TensorFlow's object detection API on raspberry pi 3B+, which uses Protobuf. Protobuf is a package that implements Google's Protocol Buffer data format. Currently there's no easy way to install Protobuf on Raspberry Pi because it needs to be compiled from the source and then installed, but fortunately, we found an installation guide by OSDevLab.(Link to Installation Guide)
First, we start by downloading TensorFlow and installing necessary packages.
The packages used are: libatlas-base-dev,pillow, lxml, jupyter, matplotlib, cython, and python-tk.
Then we install some packages and OpenCV.
The packages used are: libjpeg-dev, libtiff5-dev, libjasper-dev, libpng12-dev, libavcodec-dev, libavformat-dev, libswscale-dev, libv4l-dev, libxvidcore-dev, libx264-dev, and qt4-dev-tools
IF ERROR OCCURED, RUN sudo apt-get update to solve
Then install OpenCV
- pip3 install opencv-python
Then follow the installation instruction written by OSDevLab to compile and install Protobuf. (Link to Installation Guide)
We downloaded the Protobuf release from GitHub, double check if there's a newer version available then download the newest version. Configure the build process by following the installation guide and start building the package. Generally, the process takes about 1 hour.
After finishing the building process, start the checking process. This takes even longer (around an hour and a half).
We now have to start setting up the TensorFlow structure and modifying the PYTHONPATH environment variable.
- git clone --recurse-submodules https://github.com/tensorflow/models.git
Then, modify the PYTHONPATH environment variable to point at some directories inside the TensorFlow repository we just downloaded by issuing:
- sudo nano ~/.bashrc
On the last line (at the end of the file), add:
- export PYTHONPATH=$PYTHONPATH:/home/pi/tensorflow1/models/research:/home/pi/tensorflow1/models/research/slim
Next, we used the TensorFlow detection model zoo (Link to model zoo). This is Google's collection of pre-trained object detection models with different levels of speed and accuracy. Because we are using Raspberry Pi, which doesn't have very good computing power, we need to use a model with less processing power so that our model will be less laggy. However, this will make it less accurate. The model we choose to use is called SSD_Lite. Once you have the model and the pi is configured, you can start object detection.
Atainter has made a very detailed instruction on how to do screen sharing between Mac and Raspberry Pi.
The instruction for completing the task is the following: (Link to the Guide)
Results
Realtime Object Detection
With a second camera and raspberry pi with TensorFlow and OpenCV installed, the robot can detect everyday home objects at a rate of 0.54 - 0.70 FPS while also being able to drive our autonomous car. The program did an especially good job recognizing humans and could recognize our team member up to 10.67 meters away from the camera.
Every object, including person, was more easily recognized when in slight motion, as the program could differentiate it from the background. This was less of a problem when the car was in slow motion.
Sometimes, objects would not be recognized at all, and this may be because the object was not registered in the list of recognizable everyday objects or because the objects we used did not fit the standard description of said object. The green Hydroflask bottle seemed to be recognized very easily, however.
Potential Improvement
Due to screen sharing, the video feed and object recognition from the Pi camera is a little bit lagging on the Mac screen. A better way to solve this issue is that instead of using screen sharing, we could have used a monitor directly connected to the Raspberry using HDMI cable.
Due to the Raspberry Pi's relatively low computing power, we could have chosen a different model; however, the tradeoff would be lower accuracy in recognizing objects.
Future Developments
In our current car, we have 2 pis:
1 - the first pi is responsible for autonomous driving
2 - the second pi is responsible for real time object detection
In the future, we would like the implement communication between the two pis which are currently separately controlled. By having the two raspberry pis and cameras communicate, we can apply machine learning on a greater data set. Rather than only applying machine learning on driving images sent from only the bottom pi camera, we can have machine learning be applied to the images recorded by the 2nd camera from the object detection portion of the project. Using this information, we can train the model to accomplish a greater variety of things such as:
- Roadside sign recognition
- Train our own model for object detection
- Pedestrian recognition
- Incoming obstacle based automated maneuvers
Reference
We would like to give special thanks to:
- Professor Jack Silberman and professor Mauricio de Oliveira, (Link to web)
- Donkey Car, https://github.com/tawnkramer/donkey/blob/master/docs/guide/install_software.md
- OSDevLab, How to Install Google Protocol Buffers on Raspberry Pi and Test with Python. (Link to OSDevLab web)
- Tensorflow object detection API, https://github.com/EdjeElectronics/TensorFlow-Object-Detection-on-the-Raspberry-Pi
- Tensorflow detection model zoo, COCO-trained models. https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md
- Screen sharing instruction, Raspberry Pi VNC Mac. https://github.com/HackerShackOfficial/Raspberry-Pi-VNC-Mac