Difference between revisions of "2018SpringTeam6"

From MAE/ECE 148 - Introduction to Autonomous Vehicles
Jump to navigation Jump to search
Line 1: Line 1:
The objective of this project is to add an voice control over the original car model. This voice feature allows user to emergency stop and restart the car by talking to the car as well as switching back and forth from autonomous and joy stick controls. This project was split into several parts as follows:
The objective of this project is to add an voice control over the original car model. This voice feature will allow users to talk to the car to perform things like emergency stop as well as switching back and forth from autonomous-mode and joystick-mode. There are a few key parts to this project:
# Voice recognition through Sopare running on PC
# Use Socket to communicate Pi and PC
# Interference the DonkeyCar with voice signals


# Voice recognition through Sopare running on a PC
# Using a socket to communicate commands to the Pi from the PC
# Editing the default DonkeyCar code so it responds to the voice commands


----
----
Line 10: Line 10:
== Team Members ==
== Team Members ==


Will Xu
Will Xu  


Jose Pablo Valle
Jose Pablo Valle jpvalle@ucsd.edu


Siming Deng
Siming Deng (graduated)


Yihan Zhang
Yihan Zhang (graduated)


== Voice Recognition Using Sopare ==
== Voice Recognition Using Sopare ==


Sopare stands for '''SO'''und '''PA'''ttern '''RE'''cognition. Sopare is a python project written by bishoph. This software is able to listen offline in real time to microphone input and detect trained labels.  
Sopare stands for '''SO'''und '''PA'''ttern '''RE'''cognition. Sopare is a python project written by github user "bishoph." This software is able to listen offline in real time to microphone input and detect trained labels.  


=== Installation and usage tutorials ===
=== Installation and usage tutorials ===
Line 30: Line 30:
=== Usage ===
=== Usage ===


1. Assuming you've created a car folder using the donkey library, begin by changing directory to the d2 folder, and running $ python manage.py drive, along with the model you wish to load. A host IP and a port number should be printed. This is the script that you will run using the onboard Raspberry Pi, as this is the script that drives the car.
*We assume you've successfully constructed the car, installed the donkeycar software onto the pi, and configured the joystick controller by following the instructions provided in class*


2. On the computer connected to the same Wifi network that you want to use to send voice signals, make sure your sopare is trained with the following commands: "drive", "steer", "user", "stop", and "go". Instructions for training are in the tutorial linked above.  
1. Change into the d2 folder (/home/pi/d2), and run the command "python manage.py drive," along with the model you wish to load. A host IP and a port number should be printed. This is the script that you will run using the onboard Raspberry Pi, as this is the script that drives the car.
 
2. On the computer connected to the same Wifi network that you want to use to send voice signals, make sure your sopare is trained with the following commands: "drive", "steer", "user", "stop", and "go". Instructions for training are in the tutorial linked above.
 
In our experience, training in different settings with varying amounts of background noise will improve recognition. Just keep in mind the issues with your microphone: if a jet is flying overhead louder than your voice, or if the wind is blowing hard across the microphone, recognition performance will be affected. More training does not help -- better microphones and filtering techniques are the only way to improve it.  


3. Run the client.py script in the sopare folder, ensuring that the host and port variables in the script align with those printed by the manage.py script on the other pi. Then, you can drive the car as you normally would, with the addition of voice commands sent from your computer to the pi.
3. Run the client.py script in the sopare folder, ensuring that the host and port variables in the script align with those printed by the manage.py script on the other pi. Then, you can drive the car as you normally would, with the addition of voice commands sent from your computer to the pi.


Commands and what they do:
=== Commands and what they do ===


"stop": immediately stops the car until a "go" command is received
"stop": immediately stops the car until a "go" command is received
"go": allows the car to move again after a "stop" command
"go": allows the car to move again after a "stop" command
"drive": has the car drive autonomously using the model specified in the options of the drive script
"drive": has the car drive autonomously using the model specified in the options of the drive script
"steer": has the car steer autonomously using the model specified in the options of the drive script. Throttle is still controlled by the driver
"steer": has the car steer autonomously using the model specified in the options of the drive script. Throttle is still controlled by the driver
"user": returns total control of the car to the driver
"user": returns total control of the car to the driver



Revision as of 01:07, 15 June 2018

The objective of this project is to add an voice control over the original car model. This voice feature will allow users to talk to the car to perform things like emergency stop as well as switching back and forth from autonomous-mode and joystick-mode. There are a few key parts to this project:

  1. Voice recognition through Sopare running on a PC
  2. Using a socket to communicate commands to the Pi from the PC
  3. Editing the default DonkeyCar code so it responds to the voice commands


Team Members

Will Xu

Jose Pablo Valle jpvalle@ucsd.edu

Siming Deng (graduated)

Yihan Zhang (graduated)

Voice Recognition Using Sopare

Sopare stands for SOund PAttern REcognition. Sopare is a python project written by github user "bishoph." This software is able to listen offline in real time to microphone input and detect trained labels.

Installation and usage tutorials

git clone [1]

step-by-step tutorial [2]

Usage

  • We assume you've successfully constructed the car, installed the donkeycar software onto the pi, and configured the joystick controller by following the instructions provided in class*

1. Change into the d2 folder (/home/pi/d2), and run the command "python manage.py drive," along with the model you wish to load. A host IP and a port number should be printed. This is the script that you will run using the onboard Raspberry Pi, as this is the script that drives the car.

2. On the computer connected to the same Wifi network that you want to use to send voice signals, make sure your sopare is trained with the following commands: "drive", "steer", "user", "stop", and "go". Instructions for training are in the tutorial linked above.

In our experience, training in different settings with varying amounts of background noise will improve recognition. Just keep in mind the issues with your microphone: if a jet is flying overhead louder than your voice, or if the wind is blowing hard across the microphone, recognition performance will be affected. More training does not help -- better microphones and filtering techniques are the only way to improve it.

3. Run the client.py script in the sopare folder, ensuring that the host and port variables in the script align with those printed by the manage.py script on the other pi. Then, you can drive the car as you normally would, with the addition of voice commands sent from your computer to the pi.

Commands and what they do

"stop": immediately stops the car until a "go" command is received

"go": allows the car to move again after a "stop" command

"drive": has the car drive autonomously using the model specified in the options of the drive script

"steer": has the car steer autonomously using the model specified in the options of the drive script. Throttle is still controlled by the driver

"user": returns total control of the car to the driver

Modifying Sopare

We made an Sopare plugins to allow communication between the driving Pi and voice recognition.

Communication between Pi and PC

Modification to the car's driving was implemented through minor adjustments to the controller.py script in ~/donkeycar/donkeycay/parts. The controller.py has a function called update() which loops infinitely while checking for any button presses from the controller that is being used to control the car. We modified this script by creating a server socket in a separate method that is called when update() is called. This server receives byte strings from the client socket, which is implemented through a code injection of sopare. These byte strings match the commands they represent (e.g. "stop" = b'stop'). As strings are received, they are pushed onto a queue. These values are popped from the queue in the update() loop, and the values are checked in a series of if-statements. The 'stop' command stops the loop until a 'go' command is received. The remaining three commands all adjust the self.mode variable to match the desired input.

Creating the server socket:

Codesnippet1.png

Checking for signals in update():

Codesnippet2.png

Socket

DonkeyCar Framework Modification