jack.minardi.org

SyncNet: A Decentralized Web Browser

Jack Minardi — Wed, 15 Jan 2014 00:00:00 -0000

Imagine if we lived in a world where more demand meant less load for a webserver. Imagine if we lived in a world where no organization could take down a website, and everyone could publish a site with no hassle or upfront cost. SyncNet is a first step into that world.

SyncNet is an experimental new browser built on top of BitTorrent Sync and (soon) Colored Coins. Every time you access a site, you store all of its contents on your machine. The next user to request the site can get the contents from both your machine and the original server. As more people access a page, it becomes available from more machines, reducing the load on the original server.

Technology

SyncNet uses BitTorrent Sync to handle distributing the content of a site and Colored Coins to handle domain resolution. (Colored Coins not implemented yet.)

BitTorrent Sync

BitTorrent Sync uses the BitTorrent protocol to keep directories in sync across the internet, much like DropBox. Say you have two folders that you want to keep in sync, one at home and one at work. On one of the computers you add the directory to BTSync, and it will give you a "secret". (A secret is a string of 33 characters - 160 bits encoded in base32) Now on the other computer you add a new folder using the same secret and they will stay in sync as long as they are both connected to the internet. BTSync works behind the scenes breaking up your files into little chunks and pushes the changes between both computers, and it does this all without requiring a central server like DropBox does.

Along with the secrets mentions above, BTSync also lets you generate "read only secrets". If you share your read only secret with someone they will be able to download your files, but they will not be able to make changes that will be synced back to your computer. Every secret has a corresponding read only secret.

Colored Coins

Colored Coins are a new idea that enable "smart property" and are implemented on top of the BitCoin protocol. Colored Coins essentially allow you to color a certain coin and mark that it represents ownership of something else. In SyncNet a colored coin will represent ownership of a domain name. Anyone with access to the Bitcoin blockchain (which is public data) will be able to see who owns a domain name and what secret it resolves to.

The current implementation of SyncNet does not use Colored Coins, but that should be coming soon.

Tying it together

You can probably see where this is going. All SyncNet does is use BTSync to fetch a directory of HTML files and then renders them for the user. If those HTML files include a hyperlink to a URL starting with sync://<secret> then that secret will be added and the contents of the new directory will be displayed.

Because all the heavy lifting is done by BTSync and a QWebView, SyncNet itself is quite small. You can browse all the source code on github.

User Interface

Here is a screen shot of the current SyncNet user interface:

I am serving the website you are currently reading under the read only secret B4KWMK3VBJSH35YZMS7ZEMSQ6XNVBHALY

Entering that secret in the URL bar of SyncNet will cause it to fetch my whole site and display the index page. Now any time I update my website the changes will be synced to anyone who has downloaded it. SyncNet monitors the directory and refreshes the page if any changes are detected. This is useful because if anyone with a read/write secret makes changes to the pages, all connected clients will quickly reflect that change.

To add your own content to SyncNet you just need to add a directory of HTML files to BitTorrent Sync. This can be accomplished in SyncNet easily by clicking the plus icon in the top right. That will open up the following dialog:

Using this dialog you can create a new SyncNet site. You must generate your secret from a seed, which is just a string of characters. Anyone who knows this seed will be able to recover your secret. And remember, anyone with your secret is able to modify the contents of your site. So keep this seed secret! (And keep your secret secret!)

Once you are satisfied with your seed, click "Ok" and your new SyncNet site will be added. A directory should pop up, and if you drop HTML files into this folder, they will be served to anyone who requests your page.

Caveats

SyncNet only works with static content. This means no social networks or other dynamic content. However many sites today do not need to be dynamic and would benefit from converting to only static resources. Most blogs or news sites could be served with SyncNet with little to no modifications.

Another caveat is long load times. This is because SyncNet needs to pull down all files for a requested site, not only those needed to render the current page. However BTSync has selective sync capabilities so this could be improved in the future. At the very least ensuring that the index page is synced first would go a long way in speeding things up.

Future Work

As mentioned above, a major missing piece is domain resolution. This will be implemented with Colored Coins (or something similar such as namecoins.)

It would also be nice to be able to create a sync site from any WWW site that currently exists. Browse to a page you like in SyncNet, click on a single button and convert the site to a new SyncNet site.

Another future goal is to implement SyncNet as a chrome or firefox plugin, where a whole separate browser is not needed. This may be possible in the future.

Conclusion

I was pretty happy with how little work it ended up being to tie these technologies together into SyncNet. While SyncNet may not be the implementation that exists into the future, I believe it is a step down the path that the internet seems to be going. Decentralizing more and more aspects of the core technologies of the internet will make it more robust in the face of targeted attacks and censorships attempts.

If you like these ideas feel free to submit a pull request on github or reach out to me on twitter.

Make an Internet Controlled Lamp with a Raspberry Pi and Flask

Jack Minardi — Fri, 15 Nov 2013 00:00:00 -0000

The Internet of Things.

For far too long humans having been hogging all the internet to themselves. It is time to change that. Now that the future is here we can start connecting everything we own, allowing the world of buggy software to permeate deeper into our lives.

Overview

Here I will be showing you how to turn on and off a lamp from anywhere in the world. However, you can control any device that works by toggling its power source, such as a fountain, TV, Christmas tree lights, projector, etc.

Required Hardware

Raspberry Pi
Internet Connection (Ethernet or WiFi)
Device Under Control (Lamp in my case)
Remote Controlled Outlets (I used these.*)
6 2N2222A Transisitors (Here's 100 of them*)
Jumper wires and breadboard
Soldering Iron

* affiliate links

Our goal it to hack into the remote that controls the outlet, allowing us to toggle the buttons with the Raspberry Pi's GPIO pins. Check out my page on hardware control for more information about GPIO pins. We will then set up a simple webserver with Flask to allow remote control of the outlets from anywhere with a web connection.

Hacking into the Remote

To open up the remote I first had to remove the battery cover and undo one screw. Once this was done I was able to pry the two covers apart using a flat head screwdriver. With the plastic gone, you can see the simple through-hole circuit board. See below for a picture of the front and back. (Ignore the wires for now, I forgot to take a picture before I soldered them on.)

As you can see in the first picture, there are two rows of buttons - an on and off button for each of the three outlets. Our goal is to control those buttons with the Raspberry Pi. We can accomplish this with a circuit element known as a transistor. Basically, a transistor is just a switch that can be controlled with a computer, which is exactly what we need.

2N222A Transistor

The specific transistor I will be using here is the infamous 2N222A. All 2N222A transistors have three pins. Two of the pins are inserted in series into your circuit, and the third pin controls if the transistor conducts current (closed switch) or does not conduct current (open switch). To close the switch and allow current to flow, this third control pin needs to be raised high. To open the circuit and stop current, the control pin should be set low. The circuit diagram symbol for transistors is shown below:

The collector is where current should flow in and the emitter is where current should flow out. The base pin is the control pin I mentioned above, which is used to control the state of the transistor (conducting or not conducting). The arrow drawn on the symbol indicates the direction of current flow. You can read more about the specifics of the 2N222A in the datasheet (PDF).

To identify which pin is which, hold the transistor with the flat side facing you. The left pin is the emitter, the middle pin is the base, and the right pin is the collector.

Our goal is to use the transistor to bridge the same gap as the buttons on the remote, so that applying a voltage to the base pin on the transistor will close the circuit just as pressing the button would. In this way we are able to simulate a button press without knowing anything at all about the circuit the buttons are a part of.

Time to Solder Things

The first step is to solder wires to each side of the buttons. Here I am only going to connect up two of the buttons (on and off for outlet 1) but you can follow the pattern and connect up all six. In the picture below I've labeled the three places you will need to solder cabled to.

Solder wires to the positions indicated above so it is easier to connect this remote to a breadboard. Once this is done connect the wires across a transistor using your breadboard. See below for a circuit diagram

While not captured in this photo, you will also need to solder a wire to the ground terminal and connect that to the ground of the Raspberry Pi. You can see this ground connection in the finished photo below.

Now all we have to do is connect a GPIO pin from the Pi to the base of each transistor. Here I am using GPIO pins 18 and 23. You can see a photo of this whole setup below.

Software

The next step is to use Flask to expose our new circuit to the internet. I will be using my hardware control library RobotBrain to simplify the code. Here I'll show the code in its entirety and then point out the important bits.

There are only two files. lamp_control.py and templates/main.html (note that main.html needs to be in a directory called templates)

If you'd rather git pull than copy/paste you can find all the code on github.

`lamp_control.py`

import time                                                                     
from itertools import cycle                                                     
from flask import Flask, render_template                                        
from robot_brain.gpio_pin import GPIOPin

app = Flask(__name__)                                                           
on_pin = GPIOPin(18)                                                            
off_pin = GPIOPin(23)                                                           
state_cycle = cycle(['on', 'off'])

@app.route("/")                                                                 
@app.route("/<state>")                                                          
def update_lamp(state=None):                                                    
    if state == 'on':                                                           
        on_pin.set(1)                                                           
        time.sleep(.2)                                                          
        on_pin.set(0)                                                           
    if state == 'off':                                                          
        off_pin.set(1)                                                          
        time.sleep(.2)                                                          
        off_pin.set(0)                                                          
    if state == 'toggle':                                                       
        state = next(state_cycle)                                               
        update_lamp(state)                                                      
    template_data = {                                                           
        'title' : state,                                                        
    }                                                                           
    return render_template('main.html', **template_data)

if __name__ == "__main__":                                                      
    app.run(host='0.0.0.0', port=80)

Here we are using Flask to respond to HTTP requests. Whenever the root URL or an immediate subdirectory of root is requested, the update_lamp() function is called. If any subdirectory is requested, the name of that subdirectory is fed as an argument to the function. For example, if <my-ip>/foo is requested, Flask will call update_lamp('foo') and respond with whatever the function returns. The update_lamp() function checks for one of three states (on, off, or toggle) and performs the desired behavior.

`templates/main.html`

<!DOCTYPE html>                                                                 
   <head>                                                                       
    <title>{{ title }}</title>                                                  
    <style type="text/css">                                                     
        body {                                                                  
            padding: 0;                                                         
            margin: 0;                                                          
        }                                                                       
        .large_button {                                                         
            position: absolute;                                                 
            width: 100%;                                                        
            height: 50%;                                                        
            text-align: center;                                                 
            text-decoration: none;                                              
            font-size: 1000%;

        }                                                                       
        #on {                                                                   
            background-color: #fbf09a;                                          
            color: rgb(223, 204, 103);                                          
            text-shadow: 1px 1px 10px #5C4E17;                                  
        }                                                                       
        #off {                                                                  
            background-color: #1e170b;                                          
            top: 50%;                                                           
            color: rgb(83, 71, 48);                                             
            text-shadow: 1px 1px 10px #000000;                                  
        }                                                                       
    </style>                                                                    
    </head>                                                                     
    <body>                                                                      
        <h1>                                                                    
            <a href="/on" id="on" class="large_button">ON</a>                   
        </h1>                                                                   
        <h1>                                                                    
            <a href="/off" id="off" class="large_button">OFF</a>                
        </h1>                                                                   
    </body>                                                                     
</html>

This is the template that is rendered by the above Flask function. When rendered it looks like this:

To start the server simply run the script with root privileges (GPIO access needs root):

sudo python lamp_control.py

Now just navigate to your Pi's IP address and you should be able to control the lamp!

NOTE: I am not a Flask or web design expert, so I might not be doing things the correct way. But it wouldn't be hacking if we knew what we was doing all the time, would it?

Conclusion

I usually run the script with screen so I can reconnect later. I've also added an Android homescreen shortcut that links to the toggle URL so I can easily control the light from my homescreen.

In the end I was pretty happy with how simple this turned out to be. If you follow these instructions and connect one of your devices to the internet, I'd love to hear about it! You can reach me on twitter @jackminardi.

Displaying Realtime Github Activity on a Full Color LED Matrix

Jack Minardi — Sun, 13 Oct 2013 00:00:00 -0000

Overview

The goal of this project was to display real time activity from my companies Github feed. I used an RGB LED matrix to display information about the last 30 events, including the username of whoever is responsible for the most recent event.

Here is a video of the display operating.

This project uses the following hardware:

Arduino
Electric Imp and Breakout Board
TiM Board (8x16 RGB LED Display)
4.5 V power supply.
A few jumper cables

You can find all the software on Github or at the end of this article.

Free Hardware!

I met Justin Shaw at SciPy 2013, where we talked about software development and open hardware hacking. He told me about his company Wyolum that focuses on open source hardware. He explained that they have a Google+ Page where they routinely give away hardware to people who promise to do cool things with it.

I joined immediately and the first hardware giveaway was a full color 8x16 smart LED matrix known as The intelligent Matrix (TiM board). I applied with the idea of building a real time Github activity display for the breakroom at my office. My idea was selected and Wyolum sent me a TiM board. (In fact, everyone who entered with an idea was sent one -- You should join now!)

The intelligent Matrix

The TiM board is made made up of 128 "Smart Pixels", which are full color RGB LEDs. Each LED has its own control circuit, making it easy to control many LEDs with a single GPIO pin. The specific type of smart pixels used in the TiM board are 5050-WS2811. You can find more details about the board in the User Guide

I decided to operate my board in serial mode, so I soldered the pads on the back. I also soldered a couple of header pins to make it easier for me to connect power and signal to the board. See below for photos of the soldered pads and header pins:

Connecting these pads lets me operate the TiM board in "serial mode".

I soldered on some header pins to make connections easier.

The Internet of Things with an Electric Imp

I needed some method to collect data about github events. Luckily for me, Github makes it incredibly easy to get activity information using their API. All I had to do was generate an access token for my account and I was able to easily request the latest organization events using this URL:

https://api.github.com/orgs/enthought/events?access_token=<mytoken>

Since I built this project with an Arduino, I needed some way to access the Internet easily. There are a couple of methods for connecting an Arduino to the Internet, and in the end I choose to use an Electric Imp and breakout board. (I choose this because the company was giving them away at the YCombinator Hardware Hackathon, so I just had one lying around.)

The Electric Imp breakout board I have has to be programmed in a language called Squirrel, which is a C like language designed for embedded platforms. Programming with the Imp involves writing code for two platforms: The "Agent" which is some machine in the cloud, and the "Imp" which is the actual hardware on my desk. The Imp sends a message to the agent, which hits the Github API and returns the desired data.

Simplifying Display Logic with Adafruit's NeoMatrix Library

On the Arduino side of things I decided to use Adafruit's NeoMatrix library to drive the LED display. While the TiM board I am using is not an Adafruit product, I was still able to make use of their open source software libraries. Yay for open source!

Data Flow

I wanted to centralize all my processing to the Arduino, and treat the Imp like a dumb component. However due to memory constraints on the Arduino I had to offload some of the computation to the Imp. The general dataflow I implemented is:

Arduino sends a load request ("l") to the Imp over serial.
The Imp receives this request and sends a message to the "Agent" to request the Github data
The agent receives this request, loads the data, and sends it as a response.
The Imp receives the Github data and stores it.
Arduino sends a next chunk request ("c") over serial. Chunk size is 60 bytes.
Imp receives this chunk request and sends the next chunk of Github data over serial.
Arduino keeps requesting the next chunk of data until the end of message character ("$") arrives.
Once the whole message has been transmitted, the Arduino processes it and outputs to the LED display.
Rinse and repeat.

Parting Thoughts

I am by no means an expert programmer on the Arduino or Electric Imp. The code I ended up with works, but I make no promises that it is the best way to do things.

While I was able to build what I set out to build, I can't help but feel that I'm not using this brilliant color LED matrix to its full potential. At any given time I am only displaying ~10 different colors, while the board is capable of displaying 2^24 different colors per pixel! I think my next step is to display the users downsampled avatar on the display. It will be cool to see just how much detail a viewer can discern on a 8x16 display. Stay tuned for updates in that area!

Question? Reach out to me on twitter

Appendix

Connections

I should probably do a proper diagram, but for know I will simply list the connections needed for the software to work:

Arduino 5V -> Imp VIN
Arduino Gnd -> Imp GND
Arduino Pin 9 -> Imp Pin5
Arduino Pin 10 -> Imp Pin7
Arduino Pin 6 -> TiM Serial Data In
Power Supply 5V and Ground -> TiM Power Pins

Code

If you'd rather git pull than copy/paste go here.

Arduino

#include <SoftwareSerial.h>
#include <Adafruit_GFX.h>
#include <Adafruit_NeoMatrix.h>
#include <Adafruit_NeoPixel.h>

#define PIN 6
#define WIDTH 16
#define HEIGHT 8

// Parameter 1 = number of pixels in strip
// Parameter 2 = pin number (most are valid)
// Parameter 3 = pixel type flags, add together as needed:
//   NEO_KHZ800  800 KHz bitstream (most NeoPixel products w/WS2812 LEDs)
//   NEO_KHZ400  400 KHz (classic 'v1' (not v2) FLORA pixels, WS2811 drivers)
//   NEO_GRB     Pixels are wired for GRB bitstream (most NeoPixel products)
//   NEO_RGB     Pixels are wired for RGB bitstream (v1 FLORA pixels, not v2)

Adafruit_NeoMatrix matrix = Adafruit_NeoMatrix(WIDTH, HEIGHT, PIN,
  NEO_MATRIX_TOP     + NEO_MATRIX_RIGHT +
  NEO_MATRIX_ROWS    + NEO_MATRIX_PROGRESSIVE,
  NEO_GRB            + NEO_KHZ800);

//defining the Pins for TX and RX serial communication
SoftwareSerial electricimpSerial = SoftwareSerial(9,10);

char character;
String content;

char eventSeparator = ',';
char itemSeparator = '.';
String lastActor;
String first = "^";
String last = "$"; 

byte ghEvents[30];

void setup() {
    Serial.begin(9600);
    electricimpSerial.begin(9600);

    electricimpSerial.print("l");  //load giithub data
    delay(500);
    electricimpSerial.print("c");  //prepare first chunk

    matrix.begin();
    matrix.setTextWrap(false);
    matrix.setBrightness(255);
 }

void loop() {
  while(electricimpSerial.available() > 0) {
     character = electricimpSerial.read();
     content += character;
  }
  if (character == '$') {
    electricimpSerial.print("l");
    delay(500);
    Serial.println(content);
    processContent();
    content = "";
  }
  electricimpSerial.print("c");
  delay(500);
 }

void processContent() {
  if (!content.startsWith(first) || !content.endsWith(last))
    return;  // bail if content string is invalid

  int start_idx = 0;
  int end_idx = 0;
  int loop_idx = 0;
  while (end_idx != -1) {

    if (loop_idx > 0) {
      String sub = content.substring(start_idx+1, end_idx);      
      int item_break = sub.indexOf(itemSeparator);
      String actor = sub.substring(0, item_break);
      String event = sub.substring(item_break+1);
      Serial.println(actor + "***" + event);

      byte x = (loop_idx-1) % (WIDTH/2);
      byte y = (loop_idx-1) / (WIDTH/2);

      if (loop_idx-1 < sizeof(ghEvents))
        ghEvents[loop_idx-1] = (byte)event[0];
      if (loop_idx == 1) lastActor = actor;
    }

    start_idx = end_idx;
    end_idx = content.indexOf(eventSeparator, end_idx+1);
    loop_idx++;

  }

  drawText();
}

void drawText() {

  for (int i; i < 100; i++) {
    drawData();  
    matrix.setCursor(WIDTH-i, 0);
    matrix.print(lastActor);
    matrix.setTextColor(matrix.Color(255, 255, 255));
    matrix.show();
    delay(70);
  }
}

void drawData() {
  int len = sizeof(ghEvents);
  matrix.fillScreen((ghEvents[0]*10)<<8);
  for (int i = 0; i < len; i++) {
    byte val = ghEvents[i];
    byte x = (i+2) % (WIDTH/2);
    byte y = (i+2) / (WIDTH/2);
    set2x2(x, y, val);
  }
  matrix.drawPixel(15, 0, matrix.Color(0, 255, 0));
}

void set2x2(byte x, byte y, byte e) {
  x = 2*x;
  y = 2*y;
  matrix.drawRect(x, y, 2, 2, (e*10)<<8); 
}

Agent

function HttpGetWrapper (url, headers) {
  local request = http.get(url, headers);
  local response = request.sendsync();
  return response;
}

function pollGithub(param) {
    local url = "https://api.github.com/orgs/enthought/events?access_token=nope";
    local result = HttpGetWrapper(url, {});
    local body = result["body"];
    local obj = http.jsondecode(body);
    //server.log(param);
    local to_send = "";
    foreach(event in obj) {
        local actor = event["actor"]["login"];
        local type = event["type"];
        type = ::eventMap[type];
        to_send += actor + "." + type + ",";
    }
    device.send("res", to_send)
}

device.on("req", pollGithub)

::eventMap <- {
    "CommitCommentEvent": "a"
    "CreateEvent": "b"
    "DeleteEvent": "c"
    "DownloadEvent": "d"
    "FollowEvent": "e"
    "ForkEvent": "f"
    "ForkApplyEvent": "g"
    "GistEvent": "h"
    "GollumEvent": "i"
    "IssueCommentEvent": "j"
    "IssuesEvent": "k"
    "MemberEvent": "l"
    "PublicEvent": "m"
    "PullRequestEvent": "n"
    "PullRequestReviewCommentEvent": "o"
    "PushEvent": "p"
    "ReleaseEvent": "q"
    "StatusEvent": "r"
    "TeamAddEvent": "s"
    "WatchEvent": "t"
}

Imp

local old_payload = "";

function readSerial() {

    local result = hardware.uart57.read();
    if (result == 99) {  //"c" in ASCII
        server.log("chunk - " + result);
        server.log("calling sendNextChunk()")
        sendNextChunk();
    } else if (result == 108) { //"l" in ASCII
        server.log("load - " + result);
        server.log("sending req to agent")
        agent.send("req", result);
    }
    imp.wakeup(0.1, readSerial);
}

function sendNextChunk() {
    server.log("sendNextChunk() called");
    local chunk_size = 60;
    local to_send = "";
    if (old_payload.len() < chunk_size) {
        to_send = old_payload;
        old_payload = "";
    } else {
        to_send = old_payload.slice(0, chunk_size);
        old_payload = old_payload.slice(chunk_size);
    }
    server.log("to_send: " + to_send)
    hardware.uart57.write(to_send);
}

function saveGithub(payload) {
  server.log("payload: " + payload);
  old_payload = "^" + payload + "$";
}

agent.on("res", saveGithub);

hardware.uart57.configure(9600, 8, PARITY_NONE, 1, NO_CTSRTS);
server.log("starting...");
readSerial();

Computational Synesthesia

Jack Minardi — Tue, 01 Oct 2013 00:00:00 -0000

Front Matter

If you just want to look at python code, checkout the repo on github

The blog post was written in an IPython Notebook. You can download the notebook here.

Audio Analysis with Image Processing Algorithms

GPUs were originally developed to simulate a specific system - the physics of light - yet today we use them to solve problems across many different domains. Could the same thing be done with image processing algorithms? Why not use them to process signals from other domains?

To explore this idea I decided to use image template matching to identify audio. Basic template matching algorithms accept an image and a template, and return the likelihood of the template being at any place on the image. My goal was to match a given audio sample to a database of stored audio tracks using image template matching. (Basically do what Shazaam or Soundhound does.)

Preprocessing

Usually audio data is represented as a 1D array of samples evenly spaced in time. Single channel images are often stored and processed on as 2D arrays, so I needed to get some representation of audio in two dimensions. The naive approach is to simply reshape the 1D array into a 2D array.

However this would be a very fragile in practice. Say we started sampling an audio source half a second later than the one in the database, the reshaped image would be quite different. Two adjacent peaks in the database may be on opposite sides of the image in the sample.

In [48]:

audio_data = (np.sin(200*np.arange(-30, 30, .01)) + 
              np.sin(500*np.arange(-30, 30, .01)))
audio_sample = audio_data[60:1440]  # take a subsample

full_image = audio_data.reshape((60, -1))
sample_image = audio_sample.reshape((60, -1))

imshow(full_image)
figure()
imshow(sample_image)

Out[48]:

<matplotlib.image.AxesImage at 0x1155bdcd0>

The second image does not immediately appear to be a subsample of the first.

A simple image template matching algorithm slides the template over the image comparing each overlap. (Here I will be using the match_template function in skimage.feature which does essentially this.) Image template matching algorithms that work in this manner would not recognize the above smaller image inside the larger one.

In [49]:

from skimage.feature import match_template
print match_template.__doc__[:340]

Match a template to an image using normalized correlation.

    The output is an array with values between -1.0 and 1.0, which correspond
    to the probability that the template is found at that position.

    Parameters
    ----------
    image : array_like
        Image to process.
    template : array_like
        Template to locate.

In [50]:

result = match_template(full_image, sample_image)
result.max()

Out[50]:

0.019117253

As you can see match_template gives a very low probability of the template being anywhere in the image.

Enter the Fourier Transform

This large sensitivity to time shifts can be mitigated by moving out of the time domain and into the frequency domain. This can be accomplished by using the Discrete Fourier Transform, and it can be accomplished quickly by using the Fast Fourier Transform. (For a good discussion of the DFT and FFT see this blog post by Jake VanderPlas)

If we take the fourier transform of each column in our reshaped signal, we can get a feel for how the frequency components change over time. It turns out this is a common technique and is known as a spectrogram.

In a nutshell the fourier transform tells us which frequencies have the highest energies in a time domain signal. Since the frequency components in an audio signal usually change over time, the spectrogram combines the nice properties of both the time and frequency domains.

Lets see a spectrogram of the dummy audio data:

In [51]:

def spectrogram(data, segment_size=60):
    end = len(data) - len(data) % segment_size
    stacked = data[:end].reshape((-1, segment_size))
    freq_space = np.fft.fft(stacked)
    real = np.abs(freq_space)
    # fft results are mirrored, this trims the excess
    trimmed = real.T[:segment_size/2, :]
    return trimmed

spec = spectrogram(audio_data)
imshow(spec)

Out[51]:

<matplotlib.image.AxesImage at 0x11174bdd0>

As you can see there are two prominent horizontal bands. This indicates the audio signal consists of the combination of two frequencies that do not change over time. This makes sense of course because we generated the dummy audio data simply by summing two sine waves of difference frequencies.

Now we have something that looks like an image and in theory corresponds well with time shifted audio sub samples. Let's test that theory:

In [52]:

sample_spec = spectrogram(audio_sample)
result = match_template(spec, sample_spec)
result.max()

Out[52]:

1.0000006

Success! match_template is telling us that these two "images" are highly correlated. However, does it work with audio samples that are more than just the sum of two sine waves?

Real Data

I ultimately want to use my computer to detect which episode of a given TV series is currently playing. Here I will be using Adventure Time Season 1. I have 11 WAV files (one for each episode) in a subdirectory adv_time.

In [56]:

from scipy.io import wavfile
sampling_rate, audio = wavfile.read('adv_time/ep1.wav')
audio = np.sum(audio, 1)  #sum the channels
sample = audio[10000000:10200000]  # ~4.5 second subsample
spec = spectrogram(audio, segment_size=512)
sample_spec = spectrogram(sample, segment_size=512)
imshow(sample_spec)

Out[56]:

<matplotlib.image.AxesImage at 0x114eb8090>

In [64]:

%timeit result = match_template(spec, sample_spec)

1 loops, best of 3: 18 s per loop

In [62]:

plot(result[0,:])  # plot 1 dim as a line

Out[62]:

[<matplotlib.lines.Line2D at 0x115184050>]

A peak in the result! match_template is fairly confident that the subsample is part of the original audio data. However the template matching algorithm took about 20 seconds to run, which is entirely too long. We will need to lose some data to speed this up.

Ask Nyquist

A man named Nyquist taught us all that the highest frequency we can reliably detect in a signal is equal to one half the sampling rate. This is known as the Nyquist Frequency. The sampling rate of our data was returned by wavefile.read():

In [63]:

sampling_rate

Out[63]:

A sampling rate of 44.1 kHz means we can detect audio frequencies up to 22 kHz. This is convenient since the upper limit on human hearing is around 20 kHz

However, looking at the spectrum above it seems like almost all of the interesting bits are in the top half, and a large number of them are in the top eighth. It would seem that use useful information embedded in the audio track of Adventure Time is not close to the upper limit of human perception. This implies we can resample the audio by a factor of 8 and not lose too much information.

In [15]:

downsampled = audio.reshape((-1, 8)).mean(1)
downsampled_sample = sample.reshape((-1, 8)).mean(1)
spec = spectrogram(downsampled, segment_size=512)
sample_spec = spectrogram(downsampled_sample, segment_size=512)
result = match_template(spec, sample_spec)
print result.max()

0.714121

In [16]:

%timeit match_template(spec, sample_spec)

1 loops, best of 3: 1.23 s per loop

In [17]:

plot(result[0,:])

Out[17]:

[<matplotlib.lines.Line2D at 0x126972c10>]

Much better. The downsampling awarded us a factor of ~15 speedup while still giving plenty of signal above the noise. Even though the highest likelihood of a match is only 71%, that is still much higher than the surrounding noise floor. However I still want to drop some data to speed up the matching. If we downsample any more in the time domain we will really begin to lose important higher frequency information, but we can still try to downsample in the frequency domain.

In [18]:

def downsample2d(a, factor):
    e0 = a.shape[0] - a.shape[0] % factor
    e1 = a.shape[1] - a.shape[1] % factor
    shape = a.shape[0] / factor, a.shape[1] / factor
    sh = shape[0], a.shape[0]//shape[0], shape[1], a.shape[1]//shape[1]
    return a[:e0, :e1].reshape(sh).mean(-1).mean(1)

down_spec = downsample2d(spec, 2)
down_sample_spec = downsample2d(sample_spec, 2)

result = match_template(down_spec, down_sample_spec)
plot(result[0,:])

Out[18]:

[<matplotlib.lines.Line2D at 0x11110bf50>]

In [19]:

%timeit match_template(down_spec, down_sample_spec)

1 loops, best of 3: 278 ms per loop

Another factor of 4 speed up while still maintaining a good signal to noise ratio! At this point I am happy enough with the speed. With the downsampling I can compare a ~5 second sub sample to about two hours of audio in one second. But there is still one final test. So far we have only been running the tests using exact sub samples. I need to find out if this can work with samples taken using my computer's microphone.

Below is the code I used to run this experiment. First I preprocessed the audio for each episode and stored them in a dictionary called store. I then used pyaudio on my laptop to record the audio playing from an episode of Adventure Time on my TV.

In [33]:

import os
import pyaudio

def process_wavfile(filename, store):
    """ Open the given wavfile, downsample it, compute the
    spectrogram, and downsample again. Store the result in
    the given `store` keyed under the filename.
    """
    name = filename.split('/')[-1].split('.')[0]
    sampling_rate, audio = wavfile.read(filename)
    downsampled = audio.reshape((-1, 16)).mean(1)
    spec = spectrogram(downsampled, segment_size=512)
    down_spec = downsample2d(spec, 2)
    store[name] = down_spec

def acquire_audio(seconds=5):
    """ Acquire audio for the given duration.
    """
    rate = 11025
    chunk = 1024
    p = pyaudio.PyAudio()
    stream = p.open(format=pyaudio.paInt16,
                    channels=1,
                    rate=rate,
                    input=True,
                    frames_per_buffer=chunk)
    frames = []
    for _ in range(int(rate / chunk * seconds)):
        frames.append(stream.read(chunk))
    stream.stop_stream()
    stream.close()
    p.terminate()
    
    ary = np.fromstring(b''.join(frames), dtype=np.short)
    ary = ary.reshape((-1, 2)).mean(1)
    return ary

def process_acquired(ary):
    """ Calculate the spectrogram and downsample the
    given audio array.
    """
    spec = spectrogram(ary, segment_size=512)
    down_spec = downsample2d(spec, 2)
    return down_spec

store = {}
for filename in os.listdir('adv_time'):
    process_wavfile('adv_time/' + filename, store)

In [42]:

acquired = acquire_audio(5)
processed = process_acquired(acquired)

results = {}
for name, signature in store.iteritems():
    result = match_template(signature, processed)
    results[name] = result
    
top = sorted(results.items(), key=lambda i: i[1].max(), reverse=True)
for name, result in top[:3]:  # print the top three matches
    print name, result.max()

ep1 0.750591
ep8 0.458766
ep10 0.440036

In [39]:

for name, result in results.iteritems():
    plot(result[0, :])

It works! As you can see in the printed scores and the plot there is a strong peak for episode one, which is the episode I was playing at the time. This code was able to successfully identify which episode of Adventure Time (11 episodes total) I was watching with a high likelihood in under 3 seconds.

Conclusion

I set out to use image processing algorithms in domains other than image processing. I ended up with a successful application in audio fingerprinting. This may not be the best way to tackle the audio fingerprinting problem (I am sure Shazaam has a much more performant algorithm) but it was fun to explore. I tried using other template matching approaches such as Harris Corner Detection and SURF (See github), but this simple approach was the most robust and easy to implement. Further work might explore those routes more.

I didn't write memory effecient numpy code, and I imagine some more performance could be eeked out there. Along with memory optimizations these algorithms have many knobs to twist, and I did not spend much time trying to optimize them. Future work might explore the sweet spot of downsampling in both the time and frequency domains. I would also like to play with the tradeoff between sample length in time and downsampling.

I would like to explore the robustness of this algorithm in the face of varying audio sources. My downsampling might not have been detrimental to the specific type of audio I was using, but may have ruined detection in another source.

If you have questions, suggestions, ideas or complaints please reach out to me on twitter: jackminardi

You can find all the code on github

Follow the discussion on Hacker News

This blog post was written entirely in an IPython Notebook. You can download the full notebook here

What's in a Shadow?

Jack Minardi — Mon, 22 Jul 2013 00:00:00 -0000

If you'd rather read python than english you can go straight to the code on github: https://github.com/jminardi/volume-from-shadow

Motivation

I was having fun casting shadows with my laser pointer the other day. I began to wonder about the 3D information that was embedded in the 2D shadow. Given enough shadows of an object, could you recreate it? What information is lost, what is preserved?

I needed to apply a few constraints to simplify the system:

Let the light rays be parallel to each other
Let the light fall perpendicular to the shadowed surface.

Parallel light rays can be thought of as coming from an infinitely far source. Having the rays come in orthogonal to the surface allows me to ignore any perspective distortions like the keystone effect. I've drawn up a simple diagram to explain what I am talking about.

Now imaging rotating that object and watching the shadow change on the wall.

These musings led to some python code that implements a sort of shadow back projection - recreating a 3D object from a series of shadows taken at different rotations. The general algorithm goes something like this:

points = generate_random_cloud()
for each shadow:
    points.remove_impossible_points_for_shadow(shadow)

The basic idea is to iterate over all shadows removing the points that would fall outside the shadow. The points that remain after all iterations fall inside the 3D volume consistent with all the given shadows. In my implementation I assumed shadows taken at equal intervals of rotation about the z-axis, but you could easily specify the object's angle of rotation for each shadow.

Implementation

import numpy as np
from matplotlib import mlab

def whittle(cloud, polygons):
    """
    Parameters
    ----------
    cloud : ndarray
        Mx3 array containing the point cloud. Columns are x, y, and z
        Should be centered about the origin
    polygons : list of ndarray
        Each array is Mx2 and represents a series of x, y points defining a
        polygon.

    Returns
    -------
    cloud : ndarray
        Mx3 array representing the points from the original cloud that are
        contained inside the 3d volume defined by the series of polygons
    """
    # Divide half the circle evenly among the polygons
    angle = np.pi / len(polygons)
    rotation_matrix = get_rotation_matrix(angle)
    for polygon in polygons:
        # Find and remove the points that fall outside the current projection
        mask = mlab.inside_poly(cloud[:, 1:], polygon)
        cloud = cloud[mask]
        # Rotate the cloud and go to the next projection
        cloud = cloud.dot(rotation_matrix)
    return cloud

def get_rotation_matrix(angle):
    # Rotate about the z-axis
    cos, sin = np.cos(angle), np.sin(angle)
    rotation_matrix = np.array([[cos,  sin, 0],
                                [-sin, cos, 0],
                                [0,    0,   1]])
    return rotation_matrix

As you can see there is really not that much going on here. This code simply masks out the points that fall outside the shadow using inside_poly(), rotates the cloud, then masks again. You may notice you don't need the whole circle, two profiles taken 180 degrees apart should be mirror images of each other - giving no new information.

For example, I could define two diamonds and run them through the wittle function. The result should be two pyramids touching at their bases. One diamond is defines by its vertices below.

polygon = np.array([[1, 0],
                    [0,-1],
                    [-1, 0],
                    [0, 1]])

Results

Two of the diamonds above produce the following 3D volume:

Stereoscopic image, cross your eyes to view. (tutorial)

So it works for simple shadows and objects! I wanted to try something a little more difficult, so I decided to test the algorithms on my (dead) bonsai tree.

I took four photographs at (hopefully) 45 degree increments. I then wrote a quick tool that allows me to click on an image to define a polygon. I used it to define the profile of each of the four images. You can see the images and profiles below.

Running the 4 polygons above through the function produced the following point cloud:

Click for high res GIF. Stereoscopic image, cross your eyes to view.

Conclusion

I could imaging automating the polygon extraction with some image processing. Using a lazy susan and a cheap web cam you could easily import your objects into a video game like Minecraft.

I was pleasantly surprised with these results, however there are a few limitations. This method can never recreate concave shapes such as a bowl. Any concavity will simply be filled in, as that information is lost in the 3d -> 2d projection.

While this algorithm's output is great for human consumption, a point cloud is not the optimal data store for a 3d volume. I played around a bit with being a little more clever than random when creating the cloud, which you can see here , but I think this is barking up the wrong tree. I have a few ideas involving more complex polygon intersection algorithms that would result in a mesh grid rather than a point cloud. Maybe someday I'll try to get those spelled out in python.

Questions? Hit me up on twitter

You can find all the code on github: https://github.com/jminardi/volume-from-shadow

Check out the discussion on Hacker News where you can find links to similar research.

Replace Dropbox with BitTorrent Sync and a Raspberry Pi

Jack Minardi — Sat, 20 Jul 2013 00:00:00 -0000

Notes Everywhere

source

I like to take long running notes, jotting down an idea here and there when one pops up. I never know which devices I will have with me, so I like to have these notes synced across all of them.

Previously I had been using the Chrome plugin QuickNote which is a light weight interface on top of the diigo cloud service. I had been using it for about two years and never had a problem until a few days ago. I somehow managed to lose about three months of edits. I still have not quite shaken that feeling of despair knowing I will never get my data back, but it did finally force me to beef up my note-syncing system. While I will primarily be using this setup to sync notes, it is by no means limited to only text files.

Enter BitTorrent Sync

BitTorrent Sync is a free utility that uses the bittorrent protocol to keep folders in sync across devices. It can be used with OS X, Windows, Android and Linux. It is not however open source, which might be a deal breaker for some. But if this isn't too big a pill for you to swallow, with a little bit of work you can use btsync as a free syncing solution.

While Dropbox requires you to keep a copy of your data on their servers, btsync never requires your data to inhabit a device you do not own. This is nice for security, but it means to sync between two devices they both need to be online at the same time. That is where the Raspberry Pi comes in. You can use the Pi as a node in your sync network, so a change to a file on any device will sync with the Pi and then other devices will get the change when they come online.

First we will set up the Pi as the central server and then we will connect our devices to it. (It is important to note however that the Pi is not a special "server" node, it is just a regular node like any other device.)

Installing BitTorrent Sync on the Raspberry Pi

NOTE: I am using Raspbian Wheezy, but I suspect these ideas will translate to other operating systems.

You need to grab the ARM build of BitTorrent Sync:

mkdir ~/.btsync && cd ~/.btsync
wget http://btsync.s3-website-us-east-1.amazonaws.com/btsync_arm.tar.gz
tar -xvf btsync_arm.tar.gz

Unless you have a large SD in your Pi you will probably want to use an external drive for your sync location. I will be using a USB thumb drive. You may need to format your thumb drive as EXT4 if you are having issues. (WARNING! This will erase all data on your drive)

df -h                               # find your drive here, e.g. `sda1`
sudo umount /dev/sda1               # replace sda1 with your drive name !
sudo mkfs.ext4 /dev/sda1 -L BTSync  # replace sda1 with your drive name !

Now all you have to do is launch the btsync application and you will be up and running!

cd ~/.btsync
sudo ./btsync  # can be killed with `sudo killall btsync`

If you see the following output btsync is properly running.

BitTorrent Sync forked to background. pid = 3003

Navigate your browser to Your-Pi-IP-Address:8888/gui to manage the btsync process. To add the thumb drive select "Add Folder" and navigate to /media/BTSync. You will need to generate a secret as this is the first time you are adding the folder.

Other Devices

Now go download the Android app and/or the desktop app and connect them using the secret you just generated It's that simple! Any change on any device will be synced across all online devices. If you keep your Pi online it will store and push the most up to date content as your other devices go on and off line.

Extra Credit

Start at Boot

You may want to set btsync to start when you boot your Raspberry Pi. To do that we will place a script in /etc/init.d/ and then register it with update-rc.d.

sudo nano /etc/init.d/btsync

Paste the following code in the script

#! /bin/sh
# /etc/init.d/btsync
#

# Carry out specific functions when asked to by the system
case "$1" in
start)
    /home/pi/.btsync/btsync
    ;;
stop)
    killall btsync
    ;;
*)
    echo "Usage: /etc/init.d/btsync {start|stop}"
    exit 1
    ;;
esac

exit 0

Then change the permissions, test, and register it to run at boot:

sudo chmod 755 /etc/init.d/btsync
sudo /etc/init.d/btsync start       # test that the script starts
sudo /etc/init.d/btsync stop        # test that the script stops
sudo update-rc.d btsync defaults

Password Protect Web Interface

If you expose your web interface to the outside world (or if you don't trust your friends) you are going to want to password protect it. This can be done with a btsync config file passed to the executable at runtime with the --config flag. First, use btsync to generate a sample config file, modify it to fit your needs, and restart the process.

cd ~/.btsync
./btsync --dump-sample-config > btsync.conf
# browse the sample config file and change what you want
sudo killall btsync
sudo ./btsync --config btsync.conf

HINT: Use jsonlint to validate your config file if btsync complains. Also make sure to modify the /etc/init.d/btsync script to use the config file as well.

Questions? Hit me up on twitter @jackminardi

You can also follow the discussion on HN

open('/dev/real_world')

Jack Minardi — Mon, 08 Jul 2013 00:00:00 -0000

I gave a talk at SciPy 2013 titled open('dev/real_world') Raspberry Pi Sensor and Actuator Control. You can find the video on youtube, the slides on google drive and I will summarize the content here.

Typically as a programmer you will work with data on disk, and if you are lucky you will draw pictures on the screen. This is in contrast to physical computing which allows you as a programmer to work with data sensed in from the real world and to control devices that move in the real world.

physical computing at work. (source)

Goal

Use a Raspberry Pi to read in accelerometer value and to control a servo motor.

Definitions

Raspberry Pi
- Small $35 Linux computer with 2 USB ports, HDMI out, Ethernet, and most importantly...
GPIO Pins
- General Purpose Input/Output Pins
- This is the component that truly enables "physical computing". You as a programmer can set the voltage high or low on each pin, which is how you will talk to actuators. You can also read what the voltage is currently on each pin. This is how sensors will talk back to you. It is important to note that each pin represents a binary state, you can only output a 0 or a 1, nothing in between.

In this article I will go over four basic Python projects to demonstrate the hardware capabilities of the Raspberry Pi. Those projects are:

Blink an LED.
Read a pot.
Stream data.
Control a servo.

Blink an LED.

An LED is a Light Emitting Diode. A diode is a circuit element that allows current to flow in one direction but not the other. Light emitting means ... it emits light. Your typical LED needs current in the range of 10-30 mA and will drop about 2-3 volts. If you connect an LED directly to your Pi's GPIO it will source much more than 30 mA and will probably fry your LED (and possibly your Pi). To prevent this we have to put a resistor. If you want to do math you can calculate the appropriate resistance using the following equation:

R = (Vs - Vd) / I

But if you don't want to do math then pick a resistor between 500-1500 ohms. Once you've gathered up all your circuit elements (LED and resistor), build this circuit on a bread board:

thats not so bad, is it?

The code is also pretty simple. But first you will need to install RPi.GPIO. (It might come preinstalled on your OS.)

import time
from itertools import cycle
import RPi.GPIO as io

io.setmode(io.BCM)
io.setup(12, io.OUT)

o = cycle([1, 0])
while True:
    io.output(12, o.next())
    time.sleep(0.5)

The important lines basically are:

io.setup(12, io.OUT)
io.output(12, 1)

These lines of code setup pin 12 as an output, and then output a 1 (3.3 volts). Run the above code connected to the circuit and you should see your LED blinking on and off every half second.

Read a pot.

A pot is short for potentiometer, which is a variable resistor. This is just a fancy word for knob. Basically by turning the knob you affect the resistance, which affects the voltage across the pot. (V = IR, remember?). Changing voltage relative to some physical value is how many sensors work, and this class of sensor is known as an analog sensor. Remember when I said the GPIO pins can only represent a binary state? We will have to call in the aide of some more silicon to convert that analog voltage value into a binary stream of bits our Pi can handle.

That chunk of silicon is refered to as an Analog-to-Digital Converter (ADC). The one I like is called MCP3008, it has 8 10-bit channels, meaning we can read 8 sensors values with a resolution of 1024 each (2^10). This will map our input voltage of 0 - 3.3 volts to an integer between 0 and 1023.

I've turned the Pi into ephemeral yellow labels to simplify the diagram

To talk to the chip we will need a python package called spidev. For more information about the package and how it works with the MCP3008 check out this great blog post

With spidev installed and the circuit built run the following program to read live sensor values and print them to stdout.

import spidev       
import time

spi = spidev.SpiDev()
spi.open(0,0)

def readadc(adcnum):
    if not 0 <= adcnum <= 7:
        return -1
    r = spi.xfer2([1, (8+adcnum)<<4, 0])
    adcout = ((r[1] & 3) << 8) + r[2]
    return adcout

while True:
    val = readadc(0)
    print val
    time.sleep(0.5)

The most important parts are these two lines:

r = spi.xfer2([1, (8+adcnum)<<4, 0])
adcout = ((r[1] & 3) << 8) + r[2]

They send the read command and extract the relevant returned bits. See the blog post I linked above for more information on what is going on here.

Stream data.

To stream data over the wire we will be using the ØMQ networking library and implementing the REQUEST/REPLY pattern. ØMQ makes it super simple to set up a client and server in Python. The following is a complete working example.

Server

import zmq

context = zmq.Context()
socket = context.socket(
    zmq.REP)
socket.bind('tcp://*:1980')

while True:
    message = socket.recv()
    print message
    socket.send("I'm here")

Client

import zmq

context = zmq.Context()
socket = context.socket(
    zmq.REQ)
a = 'tcp://192.168.1.6:1980'
socket.connect(a)

for request in range(10):
    socket.send('You home?')
    message = socket.recv()
    print message

Now we can use traits and enaml to make a pretty UI on the client side. Check out the acc_plot demo in the github repo to see an example of the Pi streaming data over the wire to be plotted by a client.

Control a servo

Servos are (often small) motors which you can drive to certain positions. For example, for a given servo you may be able to set the drive shaft from 0 to 18o degrees, or anywhere in between. As you can imagine, this could be useful for a lot of tasks, not least of which is robotics.

Shaft rotation is controlled by Pulse Width Modulation (PWM) in which you encode information in the duration of a high voltage pulse on the GPIO pins. Most hobby servos follow a standard pulse width meaning. A 0.5 ms pulse means go to your min position and a 2.5 ms pulse means go to your max position. Now repeat this pulse every 20 ms and you're controlling a servo.

The pulse width is much more critical than the frequency

These kind of timings are not possible with Python. In fact, they aren't really possible with a modern operating system. An interrupt could come in at any time in your control code, causing a longer than desired pulse and a jitter in your servo. To meet the timing requirements we have to enter the fun world of kernel modules. ServoBlaster is a kernel module that makes use of the DMA control blocks to bypass the CPU entirely. When loaded, the kernel module opens a device file at /dev/servoblaster that you can write position commands to.

I've written a small object oriented layer around this that makes servo control simpler. You can find my library here:

https://github.com/jminardi/RobotBrain

Simple connect the servo to 5v and ground on your Pi and then connect the control wire to pin 4.

The python code is quite simple:

import time
import numpy as np
from robot_brain.servo import Servo

servo = Servo(0, min=60, max=200)

for val in np.arange(0, 1, 0.05):
    servo.set(val)
    time.sleep(0.1)

All you have to do is instantiate a servo and call its set() method with a floating point value between 0 and 1. Check out the servo_slider demo on github to see servo control implemented over the network.

YCombinator Hardware Hackathon

Jack Minardi — Tue, 05 Mar 2013 00:00:00 -0000

I recently participated in the Upverter + YCombinator Hardware Hackathon, where my team placed first overall.

You can find my TechCrunch interview about the event here: http://techcrunch.com/2013/02/26/y-combinator-hardware-hackathon-winner/

I went it with only an idea, and spent the first half hour trying to convince other people to spend their day hacking with me.

The hackathon lasted about 10 hours, and the goal was to design and build a prototype hardware device. For our entry, my team built a wearable force feedback glove. When worn, the glove is able to simulate the feeling of holding a physical object.

Here is a picture of the completed glove:

A device like this could be used for gaming, surgical assistance, or a number of other augmented reality applications. I have been interested in expanding the human-computer interface for a while, and this project allowed me to open up another channel for computers to feed back information to humans.

Mechanical Design

A length of twine is connected from the glove's fingertips, through two guiding braces, and back to a hobby servo. This is repeated for each finger. The servo is connected to a platform which is connected to the back of the glove. When the servo is actuated it pulls back on the twine holding the fingers open. Through this process we are able to simulate the resistive force of an object holding the wearers hand open.

-Drawing by teammember Tom Sherlock

For the demo at the competition, we used a distance sensor to set the hand position. The closer your hand was to the distance sensor, the more your fingers were pulled open. This simulated the feeling of squeezing a virtual object in your hand.

The distance sensor used was a HC-SR04 Ping Sensor:

Hardware I/O

To control servos and read from sensors usually you use GPIO pins. In our case we used the GPIO pins on a Raspberry Pi. If you would like to know more about hardware control with a Raspberry Pi check out my article on that topic.

The basic idea is that you can set the voltage on a GPIO pin high (5 volts) or low (0 volts). Sending the correct sequence of high and low pulses to a servo will cause it to go to a certain position. You can also read whether a certain pin is high or low. If you connect a sensor to GPIO pin, it is able to communicate information by sending specific sequences of high and low pulses.

Control Software

To control the GPIO pins I wrote a Python script that you can find here: https://gist.github.com/jminardi/5022297

This script uses a library I wrote called RobotBrain. It sits on top of RPi.GPIO and provides a higher level interface for controlling individual pins and motors. The only module used in this project was the Servo, which makes it easy to set a servo to a given position. The Servo module uses the ServoBlaster kernal module under the covers, which exposes the servo as device in the filesystem.

Conclusion

In the end I had a great time at the hackathon. I think we were able to put together a winning demo in part because of the power of Python. With just a few libraries you are able to reason at a high level about what you wanted your hardware to do and what your sensors are seeing. If you have ever tried doing hardware control in a lower level language, you know just how hard that can be, and as you can see, how easy Python makes it.

The team after our victory

Android Controlled Toy Car

Jack Minardi — Sun, 09 Dec 2012 00:00:00 -0000

I wanted to build something fun that would introduce me to hardware control with the Raspberry Pi. I decided to build a small car that could be controlled with my phone's accelerometer.

Here is a picture of the fully assembled car:

And here is a quick video of the operation:

I wrote an android app that streams the accelerometer data from the phone to the pi over a simple socket. The pi then uses this data to drive the DC motor and the servo motor. Tilting the phone to control the car feels very natural.

In this pic you can see the wifi dongle I've used:

I am using Adafruit Occidental v0.2 as my OS because it has support for my wifi dongle. It also makes some hardware interaction easier and comes pre-installed with a few good python libraries.

Here is a picture of the breadboard:

I am using the L293DNE hbridge chip for DC motor control. The two black wires you see coming off the board connect to the motor.

In this pic is the battery pack I am using to power the pi:

I purchased it on amazon here.

Here is a pic of the battery pack I am using for the DC motor:

Here is a closeup of the steering servo:

It is an HS-55 and I power it directly from the pi's 5v rail. To control it I use the servoblaster kernal module and my own servo control library RobotBrain

My next plans are to add some sensors and make it autonomous.