From MAE/ECE 148 - Introduction to Autonomous Vehicles
Jump to: navigation, search

Winter 2019 Team 9

Team Members

Naziha Kibria
Dave Kier
Jason Li
Pedro Sousa Meireles
Cole O'Connell

Team Objectives

Primary Goals
1. Following a Track
We wanted our car should be able to follow a track autonomously. As part of the first part of the course, we collected data for both indoor and outdoor tracks and the car was able to perform more than 5 laps on the indoor track and 3 on the outdoor.
2. Pattern Recognition
We wanted to be able to use the camera and the Raspberry Pi to be able to recognize specific patterns. for our project, we chose to identify brightly-colored rectangles and triangles.
3. Perform a Specified Action Upon Pattern Recognition
Our minimum goal here was to perform some action to indicate that our car recognized a specified pattern. This could be achieved in a number of ways, such as blinking an LED.

Secondary Goals
1. Pick Up and Drop Off Objects
By using the output of the raspberry pi’s pattern recognition, we wanted to turn an electromagnet on and off to pick up and drop off small objects. In this case we wanted a pink rectangle to prompt the electromagnet to turn on and pick up a screw and then turn off and drop the screw off only once it had detected a triangle.
2. Detect Objects
Instead of recognizing patterns, we wanted the car to detect objects (such as a screw) and trigger the actions of the car.
3. Store Picked Up Objects on the Car
As a final objective, we wanted to have a tray on our car to store the objects already collected. For that, we would need a mechanic arm to swing between the place to pick up the object and the tray, to drop it off.

Challenges with Our Project Goals
We had difficulty finding pre-trained openCV images of traditional metallic objects (such as screws and nuts) so we ran out of time to train for object detection to trigger actions. We decided to focus on our primary objective of pattern recognition instead.
The electromagnet proved to be too heavy for the servo motor we had available. We also worried about having the arm as a moving object within the FoV of both of our cameras, so we opted to mount the electromagnet on the arm in a fixed position.
While we were able to successfully communicate between the two Raspberry Pis outside the Donkey framework, we were unable to switch modes within the Donkey framework.

Project Goal Evaluation
We were successful in all our of primary goals, but only achieved our first secondary goal. As previously discussed, issues with availability of screw/nut recognition on openCV and the weight of the electromagnet paired with the lack of power of our servo prevented us from accomplishing these goals.

Our Car


Base Vehicle

Mechanical Components
Started with 1/10 size model race car Chassis frame, motor, servos, wheels, drive train, and 11.4V Lithium Ion Polymer Battery, Alarm circuit for LV detection, and PS3 Game Controller.

Electrical Components
Raspberry Pi3 B+ Microcontroller boards
EMO Wireless Relay for emergency shut-off
PWM board for Throttle and Steering servo control

Added Components
Second Raspberry Pi3 B+
Magnet (need specs)

Custom Components

Laser-Cut Acrylic Baseplate

Our baseplate is laser cut from an acrylic sheet. The design was based around the locations of the pre-existing standoffs on the car and included a center cut-out for easy accesibility to underlying components. This cut-out was especially useful and made all of the wire connections much easier and cleaner.

Camera Mount and Stand

Our camera mount was composed of two 3-D printed parts to create a simple hinge mechanism to allow us to adjust the angle of the camera if needed. The mount that attaches to the camera was made to dimensioned drawings we found of the camera. We did not test for optimal height or positioning, but did try two revisions of the base at different heights. We used the taller one, which was 12cm and worked well enough. We ended up using the smaller one later as a mount for our second camera. We settled on a camera angle of ~55 degrees with respect the the horizontal, however it may have been easier to train outdoors if we had a smaller angle to exclude some of the glare from windows and other objects from the frame. Also, we learned that pipe cleaners, zipties, and velcro work well in lieu of screws.

Magnet Arm
Laser-Cut Acrylic Baseplate

Originally, the arm was designed to allow for the magnet to be rotated by a servo so that the magnet would be out of the frame when trying to locate the markers for picking up screws. However, the magnet was too heavy for the servo we had. Also, the car was still able to navigate with the magnet in the frame, so we decided to just use the arm to mount the magnet in a fixed position below the baseplate.

Raspberry Pi Case
We found a pre-made case on Thingiverse [1] and downloaded the .stl file to 3D print in EnVision.


Here is the schematic for the base vehicle:
Here is the schematic including the added parts for our project: Projectschematic.png

Some Challenges with the Vehicle

Broken steering components and missing screws resulting in steering issues and a narrower turning radius than what would have been without those issues
Advice: Verify the servo is securely mounted by placing the car on the stand and turning back and forth.

Missing screws/hardware on steering servo and related components Advice: Use pipe cleaners instead

Difficulty mounting robotic arm and magnet (too heavy and interfered with FoV of 2nd camera.
Advice: Maybe use a crane-like system to raise and lower heavy objects rather than having a swinging motion with the servo as the center of rotation.

Residual magnetic charge on the screw after being picked up by the electromagnet sometimes prevented the screw from being released from the electromagnet.
Advice: Use components that are only as strong as you need them to be.

Car crashed and camera broke
Advice: Be careful with the analog sticks on the PS3 controller getting stuck.

Field of View (FoV) of camera grabbed too much background noise when driving and collecting “bad” data that presented problems with training model
Advice: Angle the camera so that it sees just above the horizon. Too much of the picture above the horizon causes too much background noise. This may have saved us a week.

Weather presented many challenges with respect to collecting data and testing on the outdoor track
Advice: Finish your outdoor laps as soon in the quarter as possible.

Indoor track availability presented limitations because of its location and other classroom activity
Advice: Plan your project such that the indoor track is not necessary.

Training for Autonomous Navigation

Framework and Code

Data Collection


Our Project

Pattern Recognition

For pattern recognition we decided to use only two shapes: a rectangle and a triangle. After some research on computer vision libraries we decided to use OpenCV to detect the shapes in our images. We found a really nice tutorial on real time shape detection with openCV here [2].

How it Works We have a mask for our camera. It means it only “sees” pixel in a range of colors. Knowing we would have many different colors when running the car, we decided to use post-its with unusual colors, as a bright pink, as our shapes.

Once we have our mask configured, we have a binary image, and openCV has some useful functions to detect contours of shapes. But it’s hard to have a perfect square in real life, so we need to have some tolerance on what can be considered a straight line, otherwise the contours will be defined as polygons with too many sizes.

Once we add the tolerance, we check for all contour forms in the image. If the contour has 4 sizes, it is a rectangle. If it has 3 sizes, it is a triangle. That’s how we detect our shapes.

Below we have two screenshots of the recognition:
Pattern1.png Pattern2.png

The black and white image represents the image after the mask is applied. The colored image is the original one, but with the contours highlighted after the object is detected. On the right side there are some trackbars to configure the range of colors accepted by the mask. By changing those colors we can configure our car to “see” the color we want it to. In the middle we can see a terminal sending messages to another computer with the detected shapes.

Picking Up Objects

Now we have the shape recognition working we want to use this information to pick up objects. Our car should pick up objects placed over a pink square and drop them off over a pink triangle.
It’s important to understand that all microcontrollers are limited in the amount of current they can deliver through their GPIO pins. The electromagnet draws quite a bit of current which could damage the GPIO pin of the Raspberry if connected directly to the the electromagnet, therefore we used a solid state relay between the Raspberry Pi GPIO pin and the electromagnet to control when the electromagnet is turn off vs on.
Detecting the appropriate shape -- using the openCV recognition algorithm -- turns on or off the magnet. The detection of a rectangle turns the electromagnet on thus picking up a metallic object that the car runs over until the camera detects a triangle, which will make the pi turn off the magnet.
To make sure the magnet would only be turned on when it was close to the object, we place the camera over the arm that holds the magnet, so if the camera detects a rectangle, it is guaranteed that the magnet will pick up the object.

Swinging Arm
As previously discussed, we wanted to have a swinging arm so that we could collect objects in different positions in front of the car, and maybe store them in a tray in the car. To do this, we would connect the servo for controlling the car in the PWM connected to the pi that runs the donkey framework. Thus, we would need to communicate between the raspberry pi’s so that when one of them detected a rectangle, it could tell the other to control the arm to pick the object. We developed a socket system to communicate between the pi’s. Once the pi that handles the recognition detects a rectangle, it sends a message to the other, that handles the message as necessary. However, after having communications working, we noticed that our army and magnet were too heavy to be able to swing around. The servo could spin them around, but the weight made the movement lack precision, so we decided to give up on this and attach the magnet at a fixed position.