2020WinterTeam4

From MAE/ECE 148 - Introduction to Autonomous Vehicles
Jump to: navigation, search

Weeyoo - Team Members

From left to right:

  • Adam Lin - Mechanical Engineering
  • Jiayi Luo - Electrical Engineering
  • Leeor Nehadea - Computer Engineering
Error creating thumbnail: Unable to save thumbnail to destination

Project Description

Our team decided to work on project involving the tracking of emergency vehicles, namely firetrucks. We want our car to be able to pull over to the side of the road when it detects the sound of a siren and recognizes an emergency vehicle behind through use of a rearview camera. Our design will have the addition of a webcam with built in audio sensors added to the rear of the car. This camera will be trained to detect the sound of an emergency siren as well as react to an image of a firetruck. When these things are detect the car will pull over to the side of the track and stop its autonomous motion. This is important in the creation of an autonomous vehicle in order to allow it drive as safe as possible in real world situations.

  • Must Have:

Detect the ambulance car using sounds

Detect the ambulance car using vision

Move to the side and stop vehicle

  • Nice to Have:

Detect if the emergency vehicle passed

Resume to the lane and driving

Detect all types of emergency car


Error creating thumbnail: Unable to save thumbnail to destination

Hardware

Components

  • Camera Case
Error creating thumbnail: Unable to save thumbnail to destination
Error creating thumbnail: Unable to save thumbnail to destination
  • Rearview Camera Case
Error creating thumbnail: Unable to save thumbnail to destination
Error creating thumbnail: Unable to save thumbnail to destination
  • Camera Mount
Error creating thumbnail: Unable to save thumbnail to destination
  • Jetson Nano Case
Error creating thumbnail: Unable to save thumbnail to destination
  • Car Chassis
Error creating thumbnail: Unable to save thumbnail to destination

Software

Siren Detection

The main algorithm we use is Mel-frequency cepstrum, which is given by librosa as a function. The model was trained on hundreds of audio data of emergency cars and non-emergency cars. Later, to predict if an audio input is siren or not, we used the weights from the trained model and the given array (from the microphone); the model then return 1 (it's a siren) or 0 (siren is not present). All the audio input can be read as an numpy array and we use librosa.mfcc to extract features of every input and use them to train a model by using Keras.

Error creating thumbnail: Unable to save thumbnail to destination

After having a model, we can integrate with donkey framework. We build a part called RecordSound to record an audio input and predict label in donkey framework.

Error creating thumbnail: Unable to save thumbnail to destination

We set default recording time to three seconds to improve the accuracy of the siren detector. However, the recording time will slow down everything else in the donkey framework, including webcam, so that the webcam cannot capture 20 images every second, which makes the car not run autonomously. To make everything work at the same time, we change our recording time to one second, which surprisingly can work simultaneously with the webcam, although one second recording time will make the siren detector less robust.

Error creating thumbnail: Unable to save thumbnail to destination

Object Detection

For object detection, we tried two ways.

The first one is using image recognition classification using Tensorflow and COCO dataset. This way gave us impressive results regarding detecting only emergency vehicles

Error creating thumbnail: Unable to save thumbnail to destination

It is possible to train the net to detect almost any desired object. So this is what we did, trained it to detect emergency cars (small modifications to the already built classifier of cars/ trucks). Yet, the problem with this approach is that the classification is relatively slow. Furthermore, the output is an image (as can be seen above.).

Thus, we chose a simpler approach. We used OpenCV to detect shapes and contours. Then, once the siren part returns 1 (siren was detected), our new part to detect cars was triggered. Then it got an image from the rear camera, and if a square within a specific range (size of the square) was detected, this function returns 1.

If both functions return 1, the car detected an emergency car, and it should stop.


For that we call built a third part, to stop the car:

Error creating thumbnail: Unable to save thumbnail to destination

Once SirenStop, which takes the predicted labels and throttle as inputs, is called it changes the value of throttle to 0 which makes the car stop.


The events pipeline works as follows: the siren detector runs continuously, once it detects a siren, the image detector is called. If both classifiers agree (both return 1- detected the emergency car), a third part is called to stop the car.

Circuitry

Components

  • Jetson Nano w/ Micro SD and Fan (Car's Computer)
  • PWM Board (Servo and ESC Control)
  • USB Camera (Front Image Detection)
  • Webcam w/ Audio Sensor (Rearview Camera)
  • Red and Blue LED (Activation Indication)
  • Voltage Checker (Alarm for Low Battery Life)
  • Emergency Relay Switch (Allow Emergency Stop)
  • Battery
  • 5V Regulator
  • Electronic Speed Controller (ESC)

Electronics Diagram

Error creating thumbnail: Unable to save thumbnail to destination

Important Milestones

Indoor Laps Testing

We were able to train our car to drive multiple laps on the indoor track given to us in the design studio

Outdoor Laps Testing

We were able to train our car to drive multiple laps on the outdoor track given to us at EBU II

Competition!

We were able to train our car to do laps on the track at DIYrobocar in Oakland with UCSD's Triton AI team

Final Product

We did it! We were able to get our car to stop when detecting the sound of the siren and recognizing an image of an emergency vehicle. The video below shows both a static and dynamic test of our project.

For the Future

  • If we had one more week we would:

Make the sound and object detector much more robust. Work on making the car detect if the emergency car passed. Make our car look prettier - lower chassis, small plate, wiring.

  • Advice for Future Teams

Libraries can be very tricky to install on Jetson Nano, especially ilbrosa since it requires many dependencies. Sound detection will probably slow down the frequency of the camera to capture images.

Acknowledgement

Our team would like to give special thanks to:

  • Prof. Mauricio de Oliveira (Instructor)
  • Prof. Jack Silberman (Instructor)
  • Nick Martin (TA)
  • Saikiran (Sai) Komatineni (TA)


Reference

- Github tutorial to use for object detection on Jetson Nano [[1]]

- Tutorial to use librosa for audio analysis[2]