2018SpringTeam6

The objective of this project is to add an voice control over the original car model. This voice feature will allow users to talk to the car to perform things like emergency stop as well as switching back and forth from autonomous-mode and joystick-mode. There are a few key parts to this project:

1. Voice recognition through Sopare running on a PC
2. Using a socket to communicate commands to the Pi from the PC
3. Editing the default DonkeyCar code so it responds to the voice commands

---

Team Members

Will Xu

Jose Pablo Valle jpvalle@ucsd.edu

Siming Deng (graduated)

Yihan Zhang (graduated)

Voice Recognition Using Sopare

Sopare stands for SOund PAttern REcognition. Sopare is a python project written by github user "bishoph." This software is able to listen offline in real time to microphone input and detect trained labels.

Installation and usage tutorials

git clone [1]

step-by-step tutorial [2]

Usage

• We assume you've successfully constructed the car, installed the donkeycar software onto the pi, and configured the joystick controller by following the instructions provided in class*

1. Change into the d2 folder (/home/pi/d2), and run the command "python manage.py drive," along with the model you wish to load. A host IP and a port number should be printed. This is the script that you will run using the onboard Raspberry Pi, as this is the script that drives the car.

2. On the computer connected to the same Wifi network that you want to use to send voice signals, make sure your sopare is trained with the following commands: "drive", "steer", "user", "stop", and "go". Instructions for training are in the tutorial linked above.

In our experience, training in different settings with varying amounts of background noise will improve recognition. Just keep in mind the issues with your microphone: if a jet is flying overhead louder than your voice, or if the wind is blowing hard across the microphone, recognition performance will be affected. More training does not help -- better microphones and filtering techniques are the only way to improve it.

3. Run the client.py script in the sopare folder, ensuring that the host and port variables in the script align with those printed by the manage.py script on the other pi. Then, you can drive the car as you normally would, with the addition of voice commands sent from your computer to the pi.

Commands and what they do

"stop": immediately stops the car until a "go" command is received

"go": allows the car to move again after a "stop" command

"drive": has the car drive autonomously using the model specified in the options of the drive script

"steer": has the car steer autonomously using the model specified in the options of the drive script. Throttle is still controlled by the driver

"user": returns total control of the car to the driver

Modifying Sopare

We made an Sopare plugins to allow communication between the driving Pi and voice recognition.

Communication between Pi and PC

Modification to the car's driving was implemented through minor adjustments to the controller.py script in ~/donkeycar/donkeycay/parts. The controller.py has a function called update() which loops infinitely while checking for any button presses from the controller that is being used to control the car. We modified this script by creating a server socket in a separate method that is called when update() is called. This server receives byte strings from the client socket, which is implemented through a code injection of sopare. These byte strings match the commands they represent (e.g. "stop" = b'stop'). As strings are received, they are pushed onto a queue. These values are popped from the queue in the update() loop, and the values are checked in a series of if-statements. The 'stop' command stops the loop until a 'go' command is received. The remaining three commands all adjust the self.mode variable to match the desired input.

Creating the server socket:

Checking for signals in update():