From MAE/ECE 148 - Introduction to Autonomous Vehicles
Jump to navigation Jump to search

Winter 2019 Team 9


After Team 9 accomplished the autonomous outdoor and indoor tracks for the Donkey Car we decided to build a delivery robot. The ultimate goal of this project is to pick up objects at a desired location and deliver them to a different location.

Team Members

Naziha Kibria
Dave Kier
Jason Li
Pedro Sousa Meireles
Cole O'Connell

Team Objectives

Primary Goals
1. Following a Track
We wanted our car to follow a track autonomously. As part of the first part of the course, we collected data for both indoor and outdoor tracks and the car was successfully able to perform more than 5 laps on the indoor track and 3 on the outdoor track.
2. Pattern Recognition
We wanted to be able to use the camera and the Raspberry Pi to be able to recognize specific patterns or shapes. For our project, we chose to identify brightly-colored rectangles and triangles shapes.
3. Perform a Specified Action Upon Pattern Recognition
Our minimum goal here was to perform some action to indicate that our car recognized a specified pattern. This could be achieved in a number of ways, such as blinking an LED, make a sound, etc.

Secondary Goals
1. Pick Up and Drop Off Objects
By using the output of the raspberry pi’s pattern recognition, we wanted to turn an electromagnet on and off to pick up and drop off small objects. We wanted a pink square/rectangle to prompt the electromagnet to turn on and pick up a screw and once a triangle is detected to then turn off the magnet and drop the screw onto the triangle.
2. Detect Objects
Instead of recognizing patterns and shapes, we wanted the car to detect objects (such as screws, nuts, paperclips) to trigger the appropriate action (that is turn OFF or ON on the electromagnet.
3. Store Picked Up Objects on the Car
As a final objective, we wanted to have a tray on our car to store the objects already collected. For that, we would need a mechanic arm to swing between the place to pick up the object and the tray, to drop it off.
4. Autonomous Search
Have the car autonomously search a taped off boundary in search of the objects.

Challenges with Our Project Goals
We had difficulty finding pre-trained openCV images of traditional metallic objects (such as screws and nuts) so we ran out of time to train for object detection to trigger actions. We decided to focus on our primary objective of pattern recognition instead.
The electromagnet proved to be too heavy for the servo motor we had available. We also worried about having the arm as a moving object within the FoV of both of our cameras, so we opted to mount the electromagnet on the arm in a fixed position.
While we were able to successfully communicate between the two Raspberry Pis outside the Donkey framework, we were unable to switch modes within the Donkey framework.
We could not train the car to go backwards when it approached a boundary line; we could only get it to stop.

Project Goal Evaluation
We were successful in all our of primary goals, but only achieved our first secondary goal. As previously discussed, issues with availability of screw/nut recognition on openCV and the weight of the electromagnet paired with the lack of power of our servo prevented us from accomplishing these goals.

Our Car


Base Vehicle

Mechanical Components
Started with 1/10 size model race car Chassis frame, motor, servos, wheels, drive train, and 11.4V Lithium Ion Polymer Battery, Alarm circuit for LV detection, and PS3 Game Controller.

Electrical Components
Raspberry Pi3 B+ Microcontroller boards
EMO Wireless Relay for emergency shut-off
PWM board for Throttle and Steering servo control

Added Components
Second Raspberry Pi3 B+

Custom Components


Laser-Cut Acrylic Baseplate

Our baseplate is laser cut from an acrylic sheet. The design was based around the locations of the pre-existing standoffs on the car and included a center cut-out for easy accesibility to underlying components. This cut-out was especially useful and made all of the wire connections much easier and cleaner.

Camera Mount and Stand


Our camera mount was composed of two 3-D printed parts to create a simple hinge mechanism to allow us to adjust the angle of the camera if needed. The mount that attaches to the camera was made to dimensioned drawings we found of the camera. We did not test for optimal height or positioning, but did try two revisions of the base at different heights. We used the taller one, which was 12cm and worked well enough. We ended up using the smaller one later as a mount for our second camera. We settled on a camera angle of ~55 degrees with respect the the horizontal, however it may have been easier to train outdoors if we had a smaller angle to exclude some of the glare from windows and other objects from the frame. Also, we learned that pipe cleaners, zipties, and velcro work well in lieu of screws.

Magnet Arm

Laser-Cut Acrylic Baseplate

Originally, the arm was designed to allow for the magnet to be rotated by a servo so that the magnet would be out of the frame when trying to locate the markers for picking up screws. However, the magnet was too heavy for the servo we had. Also, the car was still able to navigate with the magnet in the frame, so we decided to just use the arm to mount the magnet in a fixed position below the baseplate.

Raspberry Pi Case
We found a pre-made case on Thingiverse [1] and downloaded the .stl file to 3D print in EnVision.


Here is the schematic for the base vehicle (Steering and Throttle Control):
Here is the schematic of added parts (2nd Pi, USB Camera, and electromagnet circuitry):

Some Challenges with the Vehicle

Broken steering components and missing screws resulting in steering issues and a narrower turning radius than what would have been without those issues
Advice: Verify the servo is securely mounted by placing the car on the stand and turning back and forth. Double check the tightness and existence of all expected hardware.

Missing screws/hardware on steering servo and related components Advice: Use pipe cleaners instead - they work great!

Difficulty mounting robotic arm and magnet (too heavy and interfered with FoV of both cameras).
Advice: Maybe use a crane-like system instead to raise and lower heavy objects rather than having a swinging arm motion and the servo as the center of rotation.

Residual magnetic charge on the screw after being picked up by the electromagnet sometimes prevented the screw from being released from the electromagnet.
Advice: Use components that are only as strong as you need them to be and/or try different objects that less ferromagnetic.

Car crashed and camera broke
Advice: Be careful with the analog sticks on the PS3 controller ...they tend to stick.

Field of View (FoV) of camera grabbed too much background noise when driving and collecting “bad” data that presented problems with training model
Advice: Angle the camera so that it sees just above the horizon. Too much of the picture above the horizon causes too much background noise. This may have saved us at least a week.

Weather presented many challenges with respect to collecting data and testing on the outdoor track
Advice: Finish your outdoor laps as soon in the quarter as possible and when testing your trained models, try to do it in exactly the same settings (time of day, lighting conditions, etc).

Indoor track availability presented limitations because of its location and other classroom activity
Advice: Plan your project such that the indoor track is not necessary.

Our Project

Pattern Recognition

For pattern recognition we decided to use only two shapes: a rectangle and a triangle. After some research on computer vision libraries we decided to use OpenCV to detect the shapes in our images. We found a really nice tutorial on real time shape detection with openCV here [2].

How it Works
We have a mask for our camera. It means it only “sees” pixel in a range of colors. Knowing we would have many different colors when running the car, we decided to use post-its with unusual colors, as a bright pink, as our shapes.

Once we have our mask configured, we have a binary image, and openCV has some useful functions to detect contours of shapes. But it’s hard to have a perfect square in real life, so we need to have some tolerance on what can be considered a straight line, otherwise the contours will be defined as polygons with too many sizes.

Once we add the tolerance, we check for all contour forms in the image. If the contour has 4 sizes, it is a rectangle. If it has 3 sizes, it is a triangle. That’s how we detect our shapes.

Below we have two screenshots of the recognition:
Pattern1.png Pattern2.png

The black and white image represents the image after the mask is applied. The colored image is the original one, but with the contours highlighted after the object is detected. On the right side there are some trackbars to configure the range of colors accepted by the mask. By changing those colors we can configure our car to “see” the color we want it to. In the middle we can see a terminal sending messages to another computer with the detected shapes.

Picking Up Objects

Now we have the shape recognition working we want to use this information to pick up objects. Our car should pick up objects placed over a pink square and drop them off over a pink triangle.
It’s important to understand that all microcontrollers are limited in the amount of current they can deliver through their GPIO pins. The electromagnet draws quite a bit of current which could damage the GPIO pin of the Raspberry if connected directly to the the electromagnet, therefore we used a solid state relay between the Raspberry Pi GPIO pin and the electromagnet to control when the electromagnet is turn off vs on.
Detecting the appropriate shape -- using the openCV recognition algorithm -- turns on or off the magnet. The detection of a rectangle turns the electromagnet on thus picking up a metallic object that the car runs over until the camera detects a triangle, which will make the pi turn off the magnet.
To make sure the magnet would only be turned on when it was close to the object, we place the camera over the arm that holds the magnet, so if the camera detects a rectangle, it is guaranteed that the magnet will pick up the object.

Swinging Arm
As previously discussed, we wanted to have a swinging arm so that we could collect objects in different positions in front of the car, and maybe store them in a tray in the car. To do this, we would connect the servo for controlling the car in the PWM connected to the pi that runs the donkey framework. Thus, we would need to communicate between the raspberry pi’s so that when one of them detected a rectangle, it could tell the other to control the arm to pick the object. We developed a socket system to communicate between the pi’s. Once the pi that handles the recognition detects a rectangle, it sends a message to the other, that handles the message as necessary. However, after having communications working, we noticed that our army and magnet were too heavy to be able to swing around. The servo could spin them around, but the weight made the movement lack precision, so we decided to give up on this and attach the magnet at a fixed position.

Software Code

pi_server.py [3]
shape_detection.py [4]

Videos of Our Final Project

Here we have two videos of the car picking up and dropping off screws:

Potential Directions for Our Project

Have Pi2/Cam2 provide Steering, Throttle and Mode commands to Pi1 in order to navigate the car autonomously. One example could be to navigate the car such that the object is centered about the FoV.

Train openCV to actually detect objects rather then shapes.

Interface with Donkey framework to get the car to go backwards autonomously to create a sort of 'search mode'.

Figure out the mechatronics portion of the robotic arm so it’s not in the FoV of the camera and is mechanically stable.