Weeyoo - Team Members
From left to right:
- Adam Lin - Mechanical Engineering
- Jiayi Luo - Electrical Engineering
- Leeor Nehadea - Computer Engineering
Our team decided to work on project involving the tracking of emergency vehicles, namely firetrucks. We want our car to be able to pull over to the side of the road when it detects the sound of a siren and recognizes an emergency vehicle behind through use of a rearview camera. Our design will have the addition of a webcam with built in audio sensors added to the rear of the car. This camera will be trained to detect the sound of an emergency siren as well as react to an image of a firetruck. When these things are detect the car will pull over to the side of the track and stop its autonomous motion. This is important in the creation of an autonomous vehicle in order to allow it drive as safe as possible in real world situations.
- Must Have:
Detect the ambulance car using sounds
Detect the ambulance car using vision
Move to the side and stop vehicle
- Nice to Have:
Detect if the emergency vehicle passed
Resume to the lane and driving
Detect all types of emergency car
- Camera Case
- Rearview Camera Case
- Camera Mount
- Jetson Nano Case
- Car Chassis
The main algorithm we use is Mel-frequency cepstrum, which is given by librosa as a function. The model was trained on hundreds of audio data of emergency cars and non-emergency cars. Later, to predict if an audio input is siren or not, we used the weights from the trained model and the given array (from the microphone); the model then return 1 (it's a siren) or 0 (siren is not present). All the audio input can be read as an numpy array and we use librosa.mfcc to extract features of every input and use them to train a model by using Keras.
After having a model, we can integrate with donkey framework. We build a part called RecordSound to record an audio input and predict label in donkey framework.
We set default recording time to three seconds to improve the accuracy of the siren detector. However, the recording time will slow down everything else in the donkey framework, including webcam, so that the webcam cannot capture 20 images every second, which makes the car not run autonomously. To make everything work at the same time, we change our recording time to one second, which surprisingly can work simultaneously with the webcam, although one second recording time will make the siren detector less robust.
For object detection, we tried two ways.
It is possible to train the net to detect almost any desired object. So this is what we did, trained it to detect emergency cars (small modifications to the already built classifier of cars/ trucks). Yet, the problem with this approach is that the classification is relatively slow. Furthermore, the output is an image (as can be seen above.).
Thus, we chose a simpler approach. We used OpenCV to detect shapes and contours. Then, once the siren part returns 1 (siren was detected), our new part to detect cars was triggered. Then it got an image from the rear camera, and if a square within a specific range (size of the square) was detected, this function returns 1.
If both functions return 1, the car detected an emergency car, and it should stop.
For that we call built a third part, to stop the car:
Once SirenStop, which takes the predicted labels and throttle as inputs, is called it changes the value of throttle to 0 which makes the car stop.
The events pipeline works as follows: the siren detector runs continuously, once it detects a siren, the image detector is called. If both classifiers agree (both return 1- detected the emergency car), a third part is called to stop the car.
- Jetson Nano w/ Micro SD and Fan (Car's Computer)
- PWM Board (Servo and ESC Control)
- USB Camera (Front Image Detection)
- Webcam w/ Audio Sensor (Rearview Camera)
- Red and Blue LED (Activation Indication)
- Voltage Checker (Alarm for Low Battery Life)
- Emergency Relay Switch (Allow Emergency Stop)
- 5V Regulator
- Electronic Speed Controller (ESC)
Indoor Laps Testing
We were able to train our car to drive multiple laps on the indoor track given to us in the design studio
Outdoor Laps Testing
We were able to train our car to drive multiple laps on the outdoor track given to us at EBU II
We were able to train our car to do laps on the track at DIYrobocar in Oakland with UCSD's Triton AI team
We did it! We were able to get our car to stop when detecting the sound of the siren and recognizing an image of an emergency vehicle. The video below shows both a static and dynamic test of our project.
For the Future
- If we had one more week we would:
Make the sound and object detector much more robust. Work on making the car detect if the emergency car passed. Make our car look prettier - lower chassis, small plate, wiring.
- Advice for Future Teams
Libraries can be very tricky to install on Jetson Nano, especially ilbrosa since it requires many dependencies. Sound detection will probably slow down the frequency of the camera to capture images.
Our team would like to give special thanks to:
- Prof. Mauricio de Oliveira (Instructor)
- Prof. Jack Silberman (Instructor)
- Nick Martin (TA)
- Saikiran (Sai) Komatineni (TA)
- Github tutorial to use for object detection on Jetson Nano []
- Tutorial to use librosa for audio analysis