Thursday, June 5, 2014

"Shintanki" project notes

If you played Shinteki Decathlon 9, you may recall the little mini tanks:


I'm excited to drive a tank! (Photo credit Linda Holman.)

Teams got one of those little soldier-pegs every time they solved a puzzle. (Each puzzle was themed after a Stratego rank, so the peg was labeled with that rank: Marshal, Sergeant, General, etc.) At two points during the game, teams were loaned one of the little tanks, and asked to use their pegs to make the tank drive through a maze.

The "tutorial" maze. Note this team's particularly artistic arrangement of pegs!

The final challenge mazes.

The pegs could be installed in the tank (snapping into sockets on top). Each peg had its own "personality" which translated into driving commands. For example, the Scout would go forward, then turn left, then go forward, then turn right. The tank played voice messages as actions took place: "[tank voice] Hello, commander. Let's have a roll call. [Scout voice] Scout on board! [Colonel voice]  Colonel on board! [tank voice] Okay, let's get started. [Scout voice] This is the Scout driving. A scout has to look both ways! You never know what you might find. [Colonel voice] This is the Colonel driving. Let's move forward! [tank voice] Let's take it again from the top. ..."

Teams had to figure out how the tank worked in general and learn what each peg did. Then they had to figure out how to sequence the pegs to solve the problem at hand (drive through a maze, employing particular peg actions at the right time, such as the Miner disarming a bomb).

This blog post is all about the process of building the tanks, and a description of how it all worked.

The idea

When I joined the Shinteki team as "guest GC" for this year, I felt that I didn't have a lot to add to the core game or puzzles. Brent and Linda have probably run more Games than I've played in, Ian has got to be one of the world's most awesome puzzle constructors, the Decathlon format is tried and tested. What I thought maybe I could add was a little bit of "flash" or "pizzazz" -- something a little unusual in the presentation, something that sticks in the memory.

Once I learned about the Stratego theme, I started thinking about what to do. I wrote an e-mail in late December summarizing the tank concept. I said "Here's a truly nutty thought. It's super fancy and blue sky but maybe I can trim it down."

Originally, every team was going to get their own tank, which they would carry along. As they received "drivers", they could figure out what they do. There would be "tank challenges" at several points during the race where you would use what you'd collected so far to accomplish some task. This would require building at least 30 tanks (if we collected them at end of game) or 60 (if we didn't).

In retrospect, this seems totally mad. Even at the time, it seemed pretty mad. But Brent was encouraging and enthusiastic: "So my take is that this whole tank thing could be just awesome if we do it right. And I'm willing to throw a good chunk of the budget at it to make it happen. It's mainly a question of figuring out exactly what it is we're talking about and determining if we can pull it all off in time. But in general, the idea of a motorized toy that learns as the game progresses (whether it's by players earning people that affect the programming, by players learning new ways to use the toy, or by some other means) sounds like a ton of fun to me. I'm excited about the potential!"

So with that, I started to experiment.

Early prototypes

The first thing I did was try to find a tank chassis that would work. I ordered some purpose-built tank robot bases, like the Rover 5. They were fancy, with position encoders and plenty of room for electronics, but tended to be a little bigger than I was imagining, and were quite expensive ($60+ each, and that's not counting any of the electronics or pegs or batteries or anything).

The other approach was to hack a mass market toy. After some searching, I ordered a pair of these "RC Fighting Battle Tanks" from some anonymous Chinese brand on Amazon:

"With Music" is kind of overstating it.



These were $20 (Prime-eligible) for a package of 2 -- a pretty good price. It's actually a cool little toy. They make neat sounds, and you can "fire" a gun which makes a nice boom and kicks the tank backward in a recoil, and there's a sensor on each gun that can tell if it was "hit" (infrared LED/photodiode), and if it was the other tank jerks a little bit, and if it's hit three times it plays a sad little song and turns off.

Of course that was all the stuff I was planning to throw away (though I hoped to reuse the speakers). I was more interested in the basic driving mechanism:



Happily, it was easy to take apart. Once the main electronics board is removed, what remains is very simple -- batteries wired to a power switch, and two DC motors (one for each tread).





As a proof of concept, I used an Arduino Uno (leftover from BANG 28) and a Motor Shield to make the tank drive in a programmed way and annoy our cats. Here's a video I took to share with the Shinteki folks:



As you can see, the Arduino is just sort of dangling on top. But this much success led me to think this could work. I still wasn't sold on the whole enchilada, but figured "some sort of motorized toy" could be fun for puzzle purposes somehow, and a tank fit the theme nicely.

Blind driving

The core assumption was the concept that a tank could be programmed to drive a maze by blindly driving forward, turning left, and so on. I didn't want to have to add bump sensors, position sensing, line following, or any of that fancy robotics stuff, which would have greatly increased the complexity and cost of the tank, and also made it harder to match to a puzzle. I really wanted something where you built a sequence of moves, like in RoboRally or Ricochet Robots.

I knew the robot couldn't be counted on to drive perfectly straight or turn perfectly left or right, but I was hoping the maze design could be forgiving, with walls that would nudge the robot back on track if it wasn't going perfectly straight. But I wasn't at all sure this could be made to work. (In fact, until the day of the game, I was terrified that this wouldn't work.) So that was the next thing to prototype. Here's another status-report video:



As you can see, now there was a surface on top (a chunk of wood I had lying around, taped on) so it could bump into walls without catching the tread. I also got a few bits of lumber and plywood and bricks at Home Depot to prototype a maze with.

At that point I ran into a few issues. If the surface had sharp corners, it tended to get caught on things, turning a sideswipe nudge into a spin or a complete stall. Sanding down the corners of the wood helped that a lot. Also, once it was pressed against a wall, the tank would be unable to turn very well, so I added a little bit of logic to back up after driving straight and before turning. Even then, it had a tendency to get stuck in corners; even after backing up it would slide against the side wall and be unable to turn. So instead of backing up straight, I made it back up toward the direction of the turn, and thus away from the wall, giving the tank room to maneuver.

This seemed pretty promising! But this was a hardcoded program; there were no pegs, no audio, and it was still a pile of Arduinos. Also the fact that I'd made it work one time, in my apartment, with one tank was no guarantee I could make it work reliably with a bunch of different tanks getting (ab)used by teams in the field.

This was the scariest part of the whole project. I kept tweaking and tuning the motor timings, and every time I thought I had it in the bag, tanks would start behaving differently and failing to make any of their turns. During the full playtest, the final tank activities took place later on a chilly evening, and the tanks definitely drove more slowly and completely missed all their turns, creating much despair and anguish in my heart. Fortunately, for the full event, we were able to move the tanks inside the restaurant before it got chilly (which was advantageous in several ways).

There was tremendously variable performance from tank to tank. Many tanks would curve strongly to one side or the other. I tried compensating for the curve, but it was tricky since the response was nonlinear. Eventually we held "tank races" where the Shinteki folks tested all the tanks by hand (before disassembly and conversion). Of the 40 tanks (20 pairs) I'd purchased, the best 15 (straightest, most consistent forward driving) ended up being selected for use in the game. Even then, as some of you noticed, a few of them had a tendency to curve.

To avoid changes in behavior as batteries wore down, the CPU was given a sense line for battery voltage. The motor control is based on PWM, so by controlling the duty cycle to compensate for the battery voltage, the tank is able to maintain constant voltage at the motor (this is a primitive switching power supply). The motor controller was selected for very low dropout to enable the motors to be driven very close to input battery voltage, so we can achieve a reliable 5V output even if weak batteries sag as low as 5.25V under load. If the battery voltage ever fell too low, the tank would shut down and speak "Tank batteries are low. Please contact Game Control". This never happened, even though the same batteries were used for both runs of the event. (Hooray for Energizer Ultimate Lithium AAA batteries.)

Printed circuit board

After the initial prototypes next step was to start designing a PCB to make this more "real". I had been prototyping with the Arduino platform. So far I'd used the Motor Shield to drive the tank and Adafruit's Wave Shield to make sounds, but not both at the same time (in fact they both used the same pins, making that impossible without more hackery). And I hadn't done anything about peg sensing.

But the Arduino itself, motor shield, and wave shield are open hardware, and the schematics are published. So I took those designs as a starting point and built my own circuit board that combined everything I needed, and also added a stab at peg sensing. I used compatible parts so I could still use all the Arduino libraries and bootloader and so on.

This was the first time I'd designed a PCB! So it was a learning experience to figure out how to use Eagle, locate part libraries and footprints, and generally figure out just what the heck I was doing. After several weekends of struggle I had something to send out. I shipped off my design to Advanced Circuits and got a handful of boards back for testing.



This is actually the final tank board design (you can see "Shintanki v4" in the upper left). The initial revision was auto-routed, so the traces were a lot messier, and it didn't have the big ground planes, and there were different choices for some parts, and various other differences.

The board was designed to mount on the tank using the mounting holes, which were placed to correspond with the screw-holes on the tank base (the ones that originally connected the base to the top shell of the R/C toy). The electronic components face down (that's the side you see above). The other side has the contacts to sense pegs (more on that later).

Once I had boards in hand, I had to add parts! This whole "surface mount soldering" thing was new to me. (I was never even very good at regular soldering.) Learning those processes -- I primarily used hand-applied solder paste and a hot air gun -- was fun, but also maddening, because basically nothing ever worked the first time. Every board I made, up through the very end, had many defects that had to be painstakingly discovered and fixed.

Over the course of this project, I accumulated a number of tools for my workbench. During the later stages, desperation started to sink in and I would do anything that seemed like it would help.

This is after cleanup. It was much, much messier during the project,
as was every semi horizontal surface anywhere nearby.

The microscope on the left turned out to be an absolutely vital purchase. I would make a board and then look at it under the scope and recoil in horror at the Mordor-like terrain, and then head in with a soldering iron to try to make it look less like someone sneezed in a steel mill.

Here's what a board looks like after assembly (with the speaker attached, but without the power and motor connections wired up):



Not so bad, right? Most of the design is visible here. 3.3V (for the SD card) and 5.0V (for everything else) LDO regulators in the lower right. Serial I/O header for programming on the lower left. Motor driver above that, CPU (square) above that. Between the CPU and the SD card is the 5.0V<>3.3V level shifter. Above the CPU is a resistor network with 10K pullups for the pegs (see below) and other purposes. To the right is an analog mux for the pegs, the small chip below that is the DAC for audio. Hiding behind the white speaker wire is a tiny but quite powerful audio amp.

The metal discs are the magnets, which are used to hold pegs (which come in on the opposite side) in place. The magnets are held down with discs of VHB tape. Speaking of which...

Peg sensing

Motor driving and audio output are both normal things people do with Arduino type hardware, so there were plenty of designs I could crib from. But I was on my own for reliably distinguishing the type and position of 10 little pegs in 9 different holes.



I always intended to use wooden peg pieces with round bases. They're widely available, cheap, and once I found the helmeted dudes I thought they were perfect. I also planned from the start to use a PCB base with electrical contacts. I used "tab routing", a common technique for making many small boards where the PCB manufacturer creates a larger board with many "panelized" small boards on it that are nearly but not quite cut out, leaving "tabs" between them which are broken apart by the customer (me).

The parts of a peg: Tab routed PCB fragment (upper left), wooden soldier
(upper right), magnet (lower right), assembled base with spacer (bottom).

Close up.

In my case, the boards came in sheets of 40 pegs arranged in an 8-by-5 pattern, which had to be broken apart (this can be done by hand). That leaves rough tab stubs (as you can see in the fragment above), which we then sanded off with a Dremel (thank Linda for diligently sanding the tabs off hundreds and hundreds of little round boards).

Originally I had not planned to include any components on the boards, but instead to have several contacts, and use different board designs that connects the contacts together in different ways to indicate which peg is present. Because the round pegs can rotate freely in the socket, the contacts all need to be rings. I couldn't think of any way to reliably distinguish 10 different pegs with fewer than 5 rings (maybe you can?), and that seemed to be pretty excessive.

Even three rings is pushing it. (This is an obsolete design.)
Fully assembled final peg design.

Eventually I settled on using two contacts (a central disc and a ring around it). These are connected (on the top side) by a resistor. The resistance determines which type of peg it is. On the tank side, there are three contacts to provide tripod stability for the peg, but two of the contacts are wired together and both touch the same pad on the peg. Each of the nine sockets is wired to a 10K pull-up resistor and ground, forming a resistor divider with the 10K and the peg resistor. By sensing the output voltage from each socket (via analog multiplexer and on-chip ADC), the tank can tell which pegs (if any) are installed.

The contact side of the board (the top side on the tank). 
Closer view of the contacts.

But that means each and every peg base needed a resistor (of the appropriate type!) soldered to it. I did that before the panelized boards were torn apart. (I forgot to hot-air one of the boards, oops. All the resistors came off in handling with unmelted solder paste, and I had to redo them later.)

Also, the resistor makes a bump on the top side of the peg board. I made laser-cut plastic spacers including a cutout for the resistor as well as a slot for the magnet. I felt the magnet would be needed to make good contact, and I was right; when we forgot to install the magnet in a couple pegs, they just fell immediately out of the tank at the slightest touch.

The whole stack of board, spacer, magnet, and peg is assembled with glue (cyanoacrylate). Pegs were manufactured as a big joint effort by many people, including intrepid volunteers (thanks!!).

Problems

So far this maybe sounds like everything just worked. Ha ha! Not so.

The first few revisions of PCB were absolutely plagued by electrical glitches. Running the motors would cause sound playback (from the micro-SD card) to fail. Playing full volume sound would cause the CPU to reset. Doing both at once would cause errors all over the place. This was really, really distressing to a software-head like me who is terrified of analog considerations.

On the advice of a EE friend, I made a variety of changes over the iterations. I cleaned up the trace layout (doing it by hand instead of using the autorouter -- this took many hours, even for this simple board, due to my inexperience). I used thick, heavy traces for all power lines. I added ground planes everywhere that wasn't a trace, and made sure they were all connected together. I rechecked my decoupling capacitors, and added ferrite beads to quell high frequency noise on the motor and speaker wires. I twisted the motor and speaker wires to reduce electromagnetic interference (otherwise clearly visible by waving a grounded oscilloscope probe anywhere in the vicinity). I don't know which of these measures was the most important, but collectively they made the glitches go away, and I didn't feel inclined to try removing any of them.

Peg contact unreliability was another head-bashingly frustrating problem in the first iterations. Peg sensing would work great... around 70% of the time. 30% of the time, when a peg was inserted, it just wouldn't read (empty socket). Twisting the peg a little bit in the socket, or just pushing down on it a bit, would cause it to read properly.

Eventually I did three things: I had the peg PCBs done with gold plating, I increased the strength of the magnets, and most importantly I replaced the spring contacts I had been using.  Pogo pins worked great but cost 50c each, which is kind of a bunch since there were 27 (9 x 3) on each board. Fortunately, I found a product that worked well and only cost 15c or so in the quantity I was using.

Contact under a microscope (low zoom).

There were a couple failures to sense pegs during the second run, but none on the first run -- I'm guessing some of the peg bases had gotten a bit dusty by the second run. Wiping them off and re-seating them seemed to help.

Final assembly

For a while, I was really worried about making some kind of nice enclosure. Laser cut plastic was my fallback plan. But when I made an initial stab at a "rough" laser cut plastic surface (mostly to test peg reading and general operation), I thought it actually looked quite nice and decided to run with it.

There were two layers of plastic. The top layer had the Shintanki name and speaker grille. The bottom layer had cutouts for protrusions from the PCB (pins poking through from connectors on the other side) and a mounting hole for the speaker. Both layers had holes for the pegs and screws, and collectively were thick enough to provide a good "socket" for pegs to fit into.

The top and bottom layers, and speaker.

Layers assembled with the speaker in place.

Once assembled, the layers and PCB made a stack which screwed into the tank base (with stand-offs). The end result seemed quite solid! The PCB electronics were exposed on the underside, but nobody seemed too disturbed by that.

The shape changed several times to improve the tank's ability to nudge around corners and straighten itself against walls. In this demo video you can see an earlier version of the design:


Software

For me, the software was the relatively easy part, since embedded software is what I do for a living. Also, there was already an open source library to do the tricky work of accessing the SD card, reading the FAT filesystem, decoding WAV files, and sending samples to a DAC on a timed basis to play sounds in the background.

This is pretty impressive because the chip (an ATmega328P, selected for Arduino compatibility) is really underpowered! This is a chip that runs at 16MHz and has 32KB of program memory (flash) and 2KB of RAM. Yes, that's kilobytes. My first computer (an Atari 400) in the early '80s had 8 times as much RAM! 2KB doesn't leave a lot of room when you're opening FAT filesystems and reading WAV files. But the library works well and leaves a few hundred bytes free, which is plenty for the relatively simple game logic, peg sensing, motor output and so on.

The curious can browse the project here: http://svn.ofb.net/svn/egnor/dec9/
This is the main program and core game logic: game.cpp

People often ask about the tank voices. They are all synthesized voices, from AT&T's "Natural Voices" demo page or from www.fromtexttospeech.com. Originally these were placeholders, and I intended to record live human voices, but the synthetic voices actually sounded pretty good, and distinct, and meant I didn't need to bother with getting several distinct male and female voices and cleaning up recordings and normalizing volume levels and so on.

Puzzle design

Master design sketch. Note that some behaviors got changed (Captain and Sergeant swapped).
The "MAJOR BUG" is a note to myself to fix a software bug found in playtesting.

As I mentioned before, originally every team was going to get a tank, and there would be several "tank challenges". Eventually it became clear that making that many tanks would be impractical, and staffing several stations would also be impractical.

I also wanted to avoid integrating the tanks too deeply into the game. Until the very last minute I wasn't sure this was going to work! Literally two days before the game, I still had only four working tanks, which was nowhere near enough. (The day before the game, I finished up nine more.) So I wanted to make sure that even if the tanks were a flop, I didn't ruin Decathlon.

So ultimately we settled on having a "tutorial level" at clue site #4 (the Bay Model), and then a final mission at the end. Originally the final mission was going to be one huge maze, something like 6' x 8' in size (the size of the two final mazes put together). I was struggling to make a final maze design that would incorporate most of the characters in some complicated way, but Ana talked me out of it. So I settled on the two final missions that ran in the game: One to extract the Spy, and the final one to use the Spy to capture the flag.

I would have liked to use more Stratego-thematic elements. When I was planning a final mega-maze, I had notions that it would resemble a Stratego board. But I was unable to make that work in any kind of reasonable way.

In the end I had an intended solution for each of the mazes, but I was willing to accept alternate solutions if they worked. Indeed, a popular alternate solution (first employed by the Burninators) for the final maze used a repeating sequence that incorporated both the Miner and Spy in each "hazard" section. A different approach (employed with particular hilarity by CRANEA) was to use the Marshal mid-maze and hope it came out in a decent direction. The Marshal was added mostly because I couldn't figure out what to make the last peg do, and decided to do something that would be amusing, fit with the Marshal's personality from the puzzle, and theoretically serve some purpose (ensuring contact with the flag).

The other goals were to make sure each peg did something unique, and that there were some interesting "modifiers" (the Sergeant and Major). Many teams noticed that combining those two directly (they both repeat the previous action, in the Major's case with a left/right swap) was expressly forbidden. If this was Dominion there would be some careful rule about how that works, but it wasn't needed for the solution, and I didn't want to have teams waste time thinking about it, or make it more confusing to work out the effect of those pegs.

Finally, we wanted to give each peg a distinct "personality" based on the voice and messages, and where possible match it up to the puzzle. We made sure the genders aligned (Colonel Bogey is male, Lieutenant Uhura is female, etc).

Emotional impact

Most people who help run a puzzle hunt knows it can be a weird emotional roller coaster. This one was especially so for me. I ran into really bad timing with a major crunch of job related activity; I spent most of the last two weeks before the hunt flying to Michigan for work. I knew the crunch was coming and tried to get everything lined up in advance but things never quite work out that way. I wasn't sure until the end whether it would even work; the initial yield rate on tank manufacturing was abysmally low (something like 2 of the 13 tanks worked the first time in testing, the rest required some degree of repair); playtest teams ran into slow-cold-tank behavior that I thought I understood (based on refrigerating tanks) but wasn't sure of; all kinds of total meltdown scenarios played through my head. And in the end, it was just a huge amount of work.

Final production verification.


On the positive side, the rockstar team of Brent, Ian and Linda had everything else about the game completely in hand (and you don't hear them whining about the workload, despite pouring in more than me!), so I was able to focus on nothing but tanks.

Once the game is over, there's always a bit of a letdown for me. Suddenly all the stress is over, which is great! (I can return to being anxious about the other stresses of my life.) Running the hunt is a rush, but after it ends there's no real payoff commensurate to the work put in, and there's some whiplash from the sudden release. The Shinteki team run Decathlon every year (and plenty of other private hunts to boot) so they're tough as nails, but I'm still a sensitive greenhorn.

So if I knew then what I know now, I would actually have scaled this project back a bunch. There was just a little too much stress for my fragile constitution.

That said, this was a fantastic learning exercise, it was an amazing privilege to work with the best in the biz, I'm honored to have been part of an absolute core tradition of the puzzle scene, and hopefully I made the hunt a little zingier for everyone who participated. I'd like to give a big thank you to all the players, especially to all the volunteers and helpers, and I the kind comments warm my heart.

I'll close with a few more photos I managed to grab:





Tuesday, May 12, 2009

The "dead fish" flight control algorithm may not be FAA approved

No photos this time, sorry, I was too busy keeping my finger perched on the "/" key (which shuts down the flight control system and all the motors) to take any photos or videos...

So I set the flight control parameters to some reasonable seeming values, held my breath, and slowly turned up the collective gain (overall thrust). I think the longest flight I had was a second long, maybe two, before I would panic and shut everything off. I definitely made a number of propeller gouges in the foam pads...

One-second hop-and-flop flights aren't the best way to tune the feedback parameters. I did get it into a fascinating mode where it really was flopping about like a dead fish -- technically this is oscillation from having the proportional gain on the pitch and roll too high -- but actually failed to crash entirely for quite a while, though it was bouncing on its floor pads continuously. Also, there's a lot of friction between the bits of foam strapped to its underbody and the foam pads on the ground, which means that whenever it comes into contact with the ground it tends to tip over. (I see why a lot of people's experimental rotorcraft have ping-pong balls on sticks for landing gear.)

Some rig that somehow held it safely fixed in place but let it pivot freely through a wide range of pitch, roll, and yaw would be fantastic. I'm not quite sure how to arrange that. Failing that, maybe I should take it to some larger space (the cafeteria at work, on a weekend?) and give it more room to flop about while I tune all the parameters?

I'm reminded of learning to fly the original Draganflyer, a quadrotor with some basic stabilization but mostly manual control. We would practice in the cafe of our New York office, which didn't have particularly high ceilings or open spaces, and mostly be frustrated with our total lack of ability to do anything, so we would keep it very close to the floor and constantly power down just before it ran into the wall. Only here I'm tweaking code, not control sticks.

Sunday, May 10, 2009

The Propeller chip: the good, the weird, and the really weird?

I've been using the Parallax Propeller chip as the controller for my quadrotor. Specifically, I've been using the PropStick USB:



The Propeller is an embedded microprocessor with some unusual features. Basically, it's a controller optimized for bit-banging. It has 8 symmetrical cores called "cogs", each running at (up to) 80MHz, each of which has access to read and write all 32 I/O pins. Instead of dedicated peripheral controllers such as UARTs, SPI drivers, or EEPROM controllers, you just dedicate one of the cores to driving the pins directly. You can even generate NTSC or VGA video output this way. There is literally no interrupt controller! The basic chip can be had for $8 in DIP, QFN, or QFP form factors.

(The PropStick is an expensive ($80) but super convenient option -- it has the same form factor as the DIP variant, but includes power regulators, clock crystal, EEPROM, and a USB connector right on the part, so you need basically no external parts to be up and running. I'm paying for convenience here.)

The Propeller's design is super great in many ways. In my application I need to interface with the XBee radio (115kbps serial), the GPS (57kbps serial), the IMU (115kbps serial), and the motor speed controllers (four 50Hz PWM lines). A conventional embedded CPU such as the LPC2138 might have one or two UARTs and a PWM driver, so I'd still have to bit-bang the remaining serial port -- quite challenging at 57kbps if you want to do anything else at the same time! -- or use an external UART. In any case, I would have to fuss with interrupt controllers and so on, which is always a pain. In the Propeller, I simply spin up three cogs running FullDuplexSerial and one running Servo32v3 (both of which are included in the standard library) and tell them which pins to use. I dedicate another cog to parsing the IMU and GPS data as it comes in, leaving the main cog to do the actual flight control.

There's a catch, though.

The Propeller is normally programmed in this proprietary language called "Spin", possibly with some assembly language mixed in. Spin is a BASIC-like interpreted language, and it's kind of cute, but its limitations get frustrating after a while. The syntax is idiosyncratic; for example, the greater-than-or-equal-to comparison operator is "x => y", and the conventional "x >= y" actually means "x := x > y", because they have assignment versions of every conceivable operator, so you get weird bugs just because you slipped and used normal syntax in an if statement. The language is untyped and unchecked; you're constantly passing around pointers as integer values. There are no data structures, only arrays of primitive types. There is no heap. You can define code modules and instantiate them multiple times, but they must be instantiated statically, and cannot be passed outside the calling module. The libraries tend to be buggy, and people make weird cargo-cult modifications to standard modules (the GPS module, for example, demands a custom serial port module that includes decimal parsing routines).

I could live with all that, but the real problem is that Spin is slow, because it's a bytecode interpreted language running on a processor that's about as powerful as an old 486 to begin with (and a 486SX at that -- no FPU!). For example, I mentioned that I dedicate a cog to parsing the GPS and IMU output; that's because keeping up with 100Hz IMU updates and 5Hz GPS updates in Spin requires a dedicated cog. You can write assembly code, which runs much more quickly, but then, well, you have to write assembly code.

Worse, each cog only has 2KB of local "cog RAM" (there's 32KB of shared "hub RAM"), and instructions can only be executed from cog RAM. The Spin bytecode interpreter takes up all 2K. That means if you want to call assembly code from Spin, you have to send it off to another cog to execute. This means those 8 cogs are used up quickly. In my case, I have three dedicated to the serial ports, one to servo control, one to parsing the GPS and IMU output, one for the main control logic, and because the software floating point library uses assembly code that uses up another cog. That's all but one of the cogs used up already.

I would love a Propeller-like architecture that I could program in compiled C. ICCV7 for Propeller costs $99 and runs on Windows only (yuck). Catalina, a free port of lcc, is in "beta release", and apparently generates rather poor code so far. In any case you're not going to get very far squeezing compiled code into the 2K of cog RAM, so both C compilers use the "Large Memory Model" which copies instructions from hub RAM into cog RAM, runs them, and repeats. That means that compiled code in hub RAM runs 5 times slower than native code in cog RAM. (This is still many times faster than Spin code.) Also, the instruction set is apparently designed for hand assembled code, not compiled code -- there is no explicit support for a stack, for example, nor are there indexed addressing modes. I'm guessing clever compiler designers could work around these things, but the Propeller is a niche product, and clever compiler designers have other things to do with their time.

The Propeller 2 will have 256KB of hub RAM but still only 2KB of cog RAM, so I'm not sure it will make things much better. Maybe things like Catalina will be more mature and optimized by then and we'll just live with variants of the LMM. Apparently the Propeller 2 will have an entire IDE *on the chip*, so if you connect a TV (!) and a keyboard you can write code without ever using a PC.

Saturday, May 9, 2009

Not so good vibrations

I'm back at work now, so progress will be slower.

Last weekend, on Sunday night after BANG 21 wrapped up, I pulled everything together and ported the simple pitch/roll tilt sensor Kalman filter code to the onboard Propeller CPU, hooked it up to a simple PID controller (actually a P controller for now), set the collective to a low value (so it wouldn't actually fly), and... it went crazy. When the motors were running, the orientation reading would vary by as much as 50 degrees, even though the device was sitting on a desk. It would try to compensate for the huge perceived tilt, and the motors would start and stop spastically. Fortunately I had ramped down the power so nothing actually happened, but...

I theorized that the vibration caused by the motors was throwing the IMU out of whack. Today I finally got around to looking into it, and indeed, if you watch my host-based IMU sensor readout graphs when the motor turns on, you can see quite a bit of noise in the accelerometer data:



The motors were off at first, and then turned on; you can see the transition about 40% of the way across the line charts (especially visible on the X axis accelerometer). I did several things to try to make this better. First, I mounted both the IMU and the motors on rubber washers to isolate them somewhat from the frame:





The neoprene washers were from Home Depot, which had a limited selection of sizes, but anyway I got them to fit. Running a similar test, you can see that the vibration is improved somewhat, but by no means gone. Maybe I reduced it by half:



Now, the real question is why this is a problem. The whole design of the Kalman filter is to ignore high frequency transients in the accelerometer data, using it only as a low frequency drift correction for the gyro data. (As you can see, the gyro data is basically unaffected by the vibration.) And in fact, in my host based computations the filtered readout is basically fine. Near the lower right of the test displays, you'll see pitch and roll indicators; the yellow triangle is unfiltered accelerometer orientation, the yellow line is unzeroed gyro orientation, and the white indicator is the filtered result. You can see they both indicate (correctly) zero, even though the accelerometer is reading a significantly nonzero value for roll at that moment.

Anyway, after some debugging, I found a bug in the onboard code. (I was adding two floating point numbers with integer addition.) After fixing that, I got much more sensible results -- still varying a bit, but only by a degree or two (which is about the same as the host-based filter).

I'm not yet brave enough to actually ramp up the power to takeoff, but I powered it up to 30% or so (it takes 50% power to take off) and held it in my hand and moved it around. Lo and behold, I could feel the motors counteracting my motion. There's quite a lag -- I tip it to the side, and maybe a half second later it pushes back against me -- which seems like it would be a real problem for flight stability. Not sure if that's sensor reading or the time to spin up the motors or what. Maybe (hopefully?) the lag will be a lot less for small corrections. In any case I look forward to a long process of tuning the flight controller's feedback loops. (How to do that without crashing a zillion times? Even with my foam pads, I'm sure it's quite capable of inflicting serious damage on itself, not to mention me and my apartment.) Maybe some sort of strapdown test harness?

Anyway, that's good progress for one day I think.

The tomato plants are growing madly, they're like two feet tall now:



I'm not sure quite what to do with them. They're still not flowering, I sort of hoped they would have by now, but I'll be patient. Meanwhile, the little booklet has a few pages of advice about pruning the plant if it gets too high or too wide, but from reading their directions you'd think you just have to clip the errant leaf every few days or something. In reality, the thing seems to grow several inches overnight. Do I really want to be cutting off entire branches? I'm not sure. I trimmed back some of the really excessive growth and some of the leaves that had gotten burned when they grew into the lights (I told you it grew several inches!), but otherwise I'm leaving well enough alone for now. The little beasties certainly suck up a lot of water, I have to refill it with an entire pitcher pretty much every day.

Thursday, April 30, 2009

The art of reduced expectations

After poring through this code for a while, I ended up with more questions than answers. Are they indeed using a Kalman filter, or an "Extended" Kalman filter, or a "Kalman-Bucy" filter? The use of Jacobian determinants implies that it's an EKF, but where's the nonlinearity that would require an EKF? (This is especially puzzling in this simplified version that a lot of people like to crib from, which also uses a Jacobian even though it's a constant.) The use of the "P_dot = A*P + P*A_transpose + Q" form of the covariance update equation seems to imply the continuous-time ("Kalman-Bucy") formulation, but why would they do that for something that's explicitly using a series of discrete measurements?

What nobody really talks about is that most uses of the Kalman filter for accelerometer/gyro fusion are a little bit of a hack, treating the gyro as a control input, treating the accelerometer as a noisy orientation reference (even though vehicle acceleration does not behave anything like the Gaussian white noise the Kalman filter assumes), and various other questionable things.

This poor fellow had some very similar questions and got some really not very useful answers. Several people declared it was impossible because his system was "not fully determined", other people said things like "You just have to get a book on Kalman filters and work through the examples until you understand" without actually answering the questions. The grand winner of the "insanely awful reply" contest was the guy who said
And remember that for stability, the "stable platform" must have an
undamped resonance with a period of 88 minutes. (88 minutes is the
orbital time at zero altitude assuming no air, and also the period of a
pendulum whose length is the earth's radius, assuming uniform gravity.)
Yes, yes, the Earth's rotation is super important to worry about for our little MEMS accelerometers and gyros. Idiot.

It may well be that the "go read some books" advice is good, but I'm pretty sure that would take another month or six.

Anyway, on top of all of that, the way rotations are represented by quaternions in things like the Paparazzi code are also complicated. I mean, quaternions are simple enough, but mixing them with all this Kalman filter linear algebra gets hairy. And porting their carefully unrolled matrix multiplications to my Propeller processor -- which would almost certainly need to be done in assembly -- seemed more than I wanted to undertake just now. So eventually I decided, okay, screw that crap, I'm going to treat this as independent 1-D tilt estimation problems for pitch, roll, and yaw. They're not actually independent, but they are mostly so when the platform is mostly level, and I'm sort of hoping to keep it mostly level most of the time, and this way I can actually have a fighting chance of getting my head around the equations without exploding.

So, I managed to implement the basic tilt sensor Kalman filter thing for pitch and roll in Python code on my computer, and it's simple enough I bet I can port it to the Propeller pretty readily. Here it is in action:


The Kalman-filtered roll indicator is the needle near the lower-right of the grey data display window. You can see that it's perpindicular to the propeller.


See, it's still perpindicular to the propeller, in this new orientation. Of course, to get the real effect I'd have to post a video or something, but I'm just not that elite, so take my word for it that it seems to work OK and updates pretty fast and doesn't go too haywire if I shake the unit (that is, non-gravitational acceleration doesn't confuse it too badly).


Detail of the IMU test app's data display. I had just set it down on the table when this was taken, which is why pitch and roll are both zero. Check out that wacky, wacky magnetometer.

So, the next step is to port the pitch/roll computation to the Propeller, and make sure it's working okay, and then... set up some kind of simple PID controller to try to take off and keep the platform level. Yaw is going to be an issue unless I can get some kind of absolute heading out of that crazy magnetometer data, but maybe I can sort of fudge that for now with manual control.

Monday, April 27, 2009

What's the collective noun for "annoyance"?

I have learned several things the hard way over the last two weeks.

Sensors, such as the accelerometers, magnetometers, and gyroscopes that I'm using, have a significant amount of fixed bias (in addition to a nontrivial amount of variable bias). That is, the zero point of any particular sensor is really nowhere in particular. As far as I can tell from reading about other people's autopilot projects, they just take a lot of calibration measurements. I'm about to write some scripts to do the same.

The magnetometers are extremely fiddly and sensitive to noise. I guess I knew that in theory, but now I know it in practice, too. One thing I wouldn't have guessed is that the radio is hugely disruptive to the magnetometer signal (easily dwarfing the Earth's field -- and it's not a purely AC disruption, either, it adds DC bias as well). After an evening or two of cursing, I ended up relocating the radio to the end of one of the prop arms (yes, the propeller clears the tip of the antenna):

Note that the radio is strapped to the motor speed controller. The first time I did this, I strapped it directly to the heat sink of the speed controller whose auxiliary voltage regulator (aka "Battery Elimination Circuit") I'm using to power the electronics. Turns out that heat sink gets hot and that's a bad idea. Now it's strapped to the opposite side of a different speed controller.

I also wrapped the sensor board in four layers of aluminum foil (sandwiched between paper to avoid accidental shorts):

It looks ridiculous, and I could be imagining it, but it does seem to help the magnetometer noise. The idea is that the nonferrous aluminum shouldn't interfere with the Earth's magnetic field, but should help shield RF noise from the radio.

On the bright side, in testing so far running the motors doesn't seem to bother the magnetometer at all. I would have guessed quite the opposite -- that the 2.4GHz 100mW radio wouldn't bother the magnetometer, but 200W+ of low frequency electromagnetic grunting coming out of the motors would drive it crazy. But other people report the same thing, and apparently modern brushless motors really don't leak much in the way of AC fields. (It probably also helps that they are perched on the ends of arms several inches away from the magnetometer.)

I set up a test rig to calibrate motor speed and thrust:

Features of the test rig include:
  • An elevated stand (EVERT by IKEA) to avoid ground effect
  • A kitchen scale (0.5g resolution) to measure collective thrust
  • A stroboscope to measure propeller RPM
  • Foam sheets (from House of Foam) to soften the impact of crashes
  • A cat, for general assistance and interference purposes
Originally I had each of the two batteries powering two motors. I had been hoping that the speed controllers would regulate motor speed independent of battery voltage, but it turns out that's not the case -- at a given power setting, motor speed varies depending on how charged the battery is. (This was one of the first things I discovered on the test stand.) That meant that two of the motors run slower than the other two, especially since one of the batteries is powering the electronics and the other isn't.

So I spent some time and rewired the power harness so the batteries are in parallel, so everything gets a uniform voltage. Then I measured thrust (in grams, all 4 propellers spinning) and RPM at a variety of power levels (expressed as % of maximum power):

Measurements stop above 55% power because that's where it started to take off. (I could have extended the measurements by weighting down the craft, but I didn't.)

After taking all the measurements (which of course drained the batteries some) I went through the whole set again, which are the "low battery" measurements. The bad news, as noted, is that as the battery drains the speed changes. The good news is that it doesn't change all that much. The other good news is that power level, RPM, and thrust are all (above the zero level) more or less linearly related. I'm certainly hopeful that within the range of 40%-60% power, where I expect to be operating, that I can assume output is basically proportional to input.

Also, the tomato plants are doing ridiculously well:

The real bummer is that I only have a week of leave left, and plenty of puzzle hunt related work in that time, so it's looking increasingly less likely that I'll actually get to the point of stable flight -- witness how the last two weeks were spent fussing with one annoyance or another. Still, I'll see how far I can get.

Sunday, April 12, 2009

First crash damage

A variety of boards and connectors and things came in, so I pretty much finished putting everything together, and verified that the CPU can talk to the GPS and IMU and radio.




Why yes, that is a lovely rat's nest of wires! I will have to do something about the motor's power harnesses in particular, lest they dangle all about and get caught in the propellers.

So, having assembled a mostly complete (but unprogrammed) flying device, of course I had to take it for a spin (so to speak). Somewhere around 30% or 40% power, it lifted off the carpet, rapidly tipped to one side, and fell back down when I cut the power to the rotors. During one of these hops, some damage was sustained:


Yes, yes, it's kind of minor. It was a tab I had glued on earlier (it broke during some assembly mishaps). As designed, the interlocking undercarriage and zip ties held everything together just fine. Still, next time I'm at TechShop I'll make a new undercarriage with thicker tabs. (And then I'll have to cut and re-attach all the zip-ties to replace it.)

Meanwhile, since everything's hooked up and mounted, I can route IMU data through the XBee radio so at least I don't have to fuss around with that godawful Bluetooth interface. Or at least I will be able to once I reconfigure the XBee from 57600 to 115220 baud. Also, I can wave the IMU around for testing without its power coming loose every few seconds (because it's all wired up properly now, instead of using a jury-rigged connector).

Then, back to my schooling in sensor fusion and control theory.