Thursday, June 5, 2014

"Shintanki" project notes

If you played Shinteki Decathlon 9, you may recall the little mini tanks:


I'm excited to drive a tank! (Photo credit Linda Holman.)

Teams got one of those little soldier-pegs every time they solved a puzzle. (Each puzzle was themed after a Stratego rank, so the peg was labeled with that rank: Marshal, Sergeant, General, etc.) At two points during the game, teams were loaned one of the little tanks, and asked to use their pegs to make the tank drive through a maze.

The "tutorial" maze. Note this team's particularly artistic arrangement of pegs!

The final challenge mazes.

The pegs could be installed in the tank (snapping into sockets on top). Each peg had its own "personality" which translated into driving commands. For example, the Scout would go forward, then turn left, then go forward, then turn right. The tank played voice messages as actions took place: "[tank voice] Hello, commander. Let's have a roll call. [Scout voice] Scout on board! [Colonel voice]  Colonel on board! [tank voice] Okay, let's get started. [Scout voice] This is the Scout driving. A scout has to look both ways! You never know what you might find. [Colonel voice] This is the Colonel driving. Let's move forward! [tank voice] Let's take it again from the top. ..."

Teams had to figure out how the tank worked in general and learn what each peg did. Then they had to figure out how to sequence the pegs to solve the problem at hand (drive through a maze, employing particular peg actions at the right time, such as the Miner disarming a bomb).

This blog post is all about the process of building the tanks, and a description of how it all worked.

The idea

When I joined the Shinteki team as "guest GC" for this year, I felt that I didn't have a lot to add to the core game or puzzles. Brent and Linda have probably run more Games than I've played in, Ian has got to be one of the world's most awesome puzzle constructors, the Decathlon format is tried and tested. What I thought maybe I could add was a little bit of "flash" or "pizzazz" -- something a little unusual in the presentation, something that sticks in the memory.

Once I learned about the Stratego theme, I started thinking about what to do. I wrote an e-mail in late December summarizing the tank concept. I said "Here's a truly nutty thought. It's super fancy and blue sky but maybe I can trim it down."

Originally, every team was going to get their own tank, which they would carry along. As they received "drivers", they could figure out what they do. There would be "tank challenges" at several points during the race where you would use what you'd collected so far to accomplish some task. This would require building at least 30 tanks (if we collected them at end of game) or 60 (if we didn't).

In retrospect, this seems totally mad. Even at the time, it seemed pretty mad. But Brent was encouraging and enthusiastic: "So my take is that this whole tank thing could be just awesome if we do it right. And I'm willing to throw a good chunk of the budget at it to make it happen. It's mainly a question of figuring out exactly what it is we're talking about and determining if we can pull it all off in time. But in general, the idea of a motorized toy that learns as the game progresses (whether it's by players earning people that affect the programming, by players learning new ways to use the toy, or by some other means) sounds like a ton of fun to me. I'm excited about the potential!"

So with that, I started to experiment.

Early prototypes

The first thing I did was try to find a tank chassis that would work. I ordered some purpose-built tank robot bases, like the Rover 5. They were fancy, with position encoders and plenty of room for electronics, but tended to be a little bigger than I was imagining, and were quite expensive ($60+ each, and that's not counting any of the electronics or pegs or batteries or anything).

The other approach was to hack a mass market toy. After some searching, I ordered a pair of these "RC Fighting Battle Tanks" from some anonymous Chinese brand on Amazon:

"With Music" is kind of overstating it.



These were $20 (Prime-eligible) for a package of 2 -- a pretty good price. It's actually a cool little toy. They make neat sounds, and you can "fire" a gun which makes a nice boom and kicks the tank backward in a recoil, and there's a sensor on each gun that can tell if it was "hit" (infrared LED/photodiode), and if it was the other tank jerks a little bit, and if it's hit three times it plays a sad little song and turns off.

Of course that was all the stuff I was planning to throw away (though I hoped to reuse the speakers). I was more interested in the basic driving mechanism:



Happily, it was easy to take apart. Once the main electronics board is removed, what remains is very simple -- batteries wired to a power switch, and two DC motors (one for each tread).





As a proof of concept, I used an Arduino Uno (leftover from BANG 28) and a Motor Shield to make the tank drive in a programmed way and annoy our cats. Here's a video I took to share with the Shinteki folks:



As you can see, the Arduino is just sort of dangling on top. But this much success led me to think this could work. I still wasn't sold on the whole enchilada, but figured "some sort of motorized toy" could be fun for puzzle purposes somehow, and a tank fit the theme nicely.

Blind driving

The core assumption was the concept that a tank could be programmed to drive a maze by blindly driving forward, turning left, and so on. I didn't want to have to add bump sensors, position sensing, line following, or any of that fancy robotics stuff, which would have greatly increased the complexity and cost of the tank, and also made it harder to match to a puzzle. I really wanted something where you built a sequence of moves, like in RoboRally or Ricochet Robots.

I knew the robot couldn't be counted on to drive perfectly straight or turn perfectly left or right, but I was hoping the maze design could be forgiving, with walls that would nudge the robot back on track if it wasn't going perfectly straight. But I wasn't at all sure this could be made to work. (In fact, until the day of the game, I was terrified that this wouldn't work.) So that was the next thing to prototype. Here's another status-report video:



As you can see, now there was a surface on top (a chunk of wood I had lying around, taped on) so it could bump into walls without catching the tread. I also got a few bits of lumber and plywood and bricks at Home Depot to prototype a maze with.

At that point I ran into a few issues. If the surface had sharp corners, it tended to get caught on things, turning a sideswipe nudge into a spin or a complete stall. Sanding down the corners of the wood helped that a lot. Also, once it was pressed against a wall, the tank would be unable to turn very well, so I added a little bit of logic to back up after driving straight and before turning. Even then, it had a tendency to get stuck in corners; even after backing up it would slide against the side wall and be unable to turn. So instead of backing up straight, I made it back up toward the direction of the turn, and thus away from the wall, giving the tank room to maneuver.

This seemed pretty promising! But this was a hardcoded program; there were no pegs, no audio, and it was still a pile of Arduinos. Also the fact that I'd made it work one time, in my apartment, with one tank was no guarantee I could make it work reliably with a bunch of different tanks getting (ab)used by teams in the field.

This was the scariest part of the whole project. I kept tweaking and tuning the motor timings, and every time I thought I had it in the bag, tanks would start behaving differently and failing to make any of their turns. During the full playtest, the final tank activities took place later on a chilly evening, and the tanks definitely drove more slowly and completely missed all their turns, creating much despair and anguish in my heart. Fortunately, for the full event, we were able to move the tanks inside the restaurant before it got chilly (which was advantageous in several ways).

There was tremendously variable performance from tank to tank. Many tanks would curve strongly to one side or the other. I tried compensating for the curve, but it was tricky since the response was nonlinear. Eventually we held "tank races" where the Shinteki folks tested all the tanks by hand (before disassembly and conversion). Of the 40 tanks (20 pairs) I'd purchased, the best 15 (straightest, most consistent forward driving) ended up being selected for use in the game. Even then, as some of you noticed, a few of them had a tendency to curve.

To avoid changes in behavior as batteries wore down, the CPU was given a sense line for battery voltage. The motor control is based on PWM, so by controlling the duty cycle to compensate for the battery voltage, the tank is able to maintain constant voltage at the motor (this is a primitive switching power supply). The motor controller was selected for very low dropout to enable the motors to be driven very close to input battery voltage, so we can achieve a reliable 5V output even if weak batteries sag as low as 5.25V under load. If the battery voltage ever fell too low, the tank would shut down and speak "Tank batteries are low. Please contact Game Control". This never happened, even though the same batteries were used for both runs of the event. (Hooray for Energizer Ultimate Lithium AAA batteries.)

Printed circuit board

After the initial prototypes next step was to start designing a PCB to make this more "real". I had been prototyping with the Arduino platform. So far I'd used the Motor Shield to drive the tank and Adafruit's Wave Shield to make sounds, but not both at the same time (in fact they both used the same pins, making that impossible without more hackery). And I hadn't done anything about peg sensing.

But the Arduino itself, motor shield, and wave shield are open hardware, and the schematics are published. So I took those designs as a starting point and built my own circuit board that combined everything I needed, and also added a stab at peg sensing. I used compatible parts so I could still use all the Arduino libraries and bootloader and so on.

This was the first time I'd designed a PCB! So it was a learning experience to figure out how to use Eagle, locate part libraries and footprints, and generally figure out just what the heck I was doing. After several weekends of struggle I had something to send out. I shipped off my design to Advanced Circuits and got a handful of boards back for testing.



This is actually the final tank board design (you can see "Shintanki v4" in the upper left). The initial revision was auto-routed, so the traces were a lot messier, and it didn't have the big ground planes, and there were different choices for some parts, and various other differences.

The board was designed to mount on the tank using the mounting holes, which were placed to correspond with the screw-holes on the tank base (the ones that originally connected the base to the top shell of the R/C toy). The electronic components face down (that's the side you see above). The other side has the contacts to sense pegs (more on that later).

Once I had boards in hand, I had to add parts! This whole "surface mount soldering" thing was new to me. (I was never even very good at regular soldering.) Learning those processes -- I primarily used hand-applied solder paste and a hot air gun -- was fun, but also maddening, because basically nothing ever worked the first time. Every board I made, up through the very end, had many defects that had to be painstakingly discovered and fixed.

Over the course of this project, I accumulated a number of tools for my workbench. During the later stages, desperation started to sink in and I would do anything that seemed like it would help.

This is after cleanup. It was much, much messier during the project,
as was every semi horizontal surface anywhere nearby.

The microscope on the left turned out to be an absolutely vital purchase. I would make a board and then look at it under the scope and recoil in horror at the Mordor-like terrain, and then head in with a soldering iron to try to make it look less like someone sneezed in a steel mill.

Here's what a board looks like after assembly (with the speaker attached, but without the power and motor connections wired up):



Not so bad, right? Most of the design is visible here. 3.3V (for the SD card) and 5.0V (for everything else) LDO regulators in the lower right. Serial I/O header for programming on the lower left. Motor driver above that, CPU (square) above that. Between the CPU and the SD card is the 5.0V<>3.3V level shifter. Above the CPU is a resistor network with 10K pullups for the pegs (see below) and other purposes. To the right is an analog mux for the pegs, the small chip below that is the DAC for audio. Hiding behind the white speaker wire is a tiny but quite powerful audio amp.

The metal discs are the magnets, which are used to hold pegs (which come in on the opposite side) in place. The magnets are held down with discs of VHB tape. Speaking of which...

Peg sensing

Motor driving and audio output are both normal things people do with Arduino type hardware, so there were plenty of designs I could crib from. But I was on my own for reliably distinguishing the type and position of 10 little pegs in 9 different holes.



I always intended to use wooden peg pieces with round bases. They're widely available, cheap, and once I found the helmeted dudes I thought they were perfect. I also planned from the start to use a PCB base with electrical contacts. I used "tab routing", a common technique for making many small boards where the PCB manufacturer creates a larger board with many "panelized" small boards on it that are nearly but not quite cut out, leaving "tabs" between them which are broken apart by the customer (me).

The parts of a peg: Tab routed PCB fragment (upper left), wooden soldier
(upper right), magnet (lower right), assembled base with spacer (bottom).

Close up.

In my case, the boards came in sheets of 40 pegs arranged in an 8-by-5 pattern, which had to be broken apart (this can be done by hand). That leaves rough tab stubs (as you can see in the fragment above), which we then sanded off with a Dremel (thank Linda for diligently sanding the tabs off hundreds and hundreds of little round boards).

Originally I had not planned to include any components on the boards, but instead to have several contacts, and use different board designs that connects the contacts together in different ways to indicate which peg is present. Because the round pegs can rotate freely in the socket, the contacts all need to be rings. I couldn't think of any way to reliably distinguish 10 different pegs with fewer than 5 rings (maybe you can?), and that seemed to be pretty excessive.

Even three rings is pushing it. (This is an obsolete design.)
Fully assembled final peg design.

Eventually I settled on using two contacts (a central disc and a ring around it). These are connected (on the top side) by a resistor. The resistance determines which type of peg it is. On the tank side, there are three contacts to provide tripod stability for the peg, but two of the contacts are wired together and both touch the same pad on the peg. Each of the nine sockets is wired to a 10K pull-up resistor and ground, forming a resistor divider with the 10K and the peg resistor. By sensing the output voltage from each socket (via analog multiplexer and on-chip ADC), the tank can tell which pegs (if any) are installed.

The contact side of the board (the top side on the tank). 
Closer view of the contacts.

But that means each and every peg base needed a resistor (of the appropriate type!) soldered to it. I did that before the panelized boards were torn apart. (I forgot to hot-air one of the boards, oops. All the resistors came off in handling with unmelted solder paste, and I had to redo them later.)

Also, the resistor makes a bump on the top side of the peg board. I made laser-cut plastic spacers including a cutout for the resistor as well as a slot for the magnet. I felt the magnet would be needed to make good contact, and I was right; when we forgot to install the magnet in a couple pegs, they just fell immediately out of the tank at the slightest touch.

The whole stack of board, spacer, magnet, and peg is assembled with glue (cyanoacrylate). Pegs were manufactured as a big joint effort by many people, including intrepid volunteers (thanks!!).

Problems

So far this maybe sounds like everything just worked. Ha ha! Not so.

The first few revisions of PCB were absolutely plagued by electrical glitches. Running the motors would cause sound playback (from the micro-SD card) to fail. Playing full volume sound would cause the CPU to reset. Doing both at once would cause errors all over the place. This was really, really distressing to a software-head like me who is terrified of analog considerations.

On the advice of a EE friend, I made a variety of changes over the iterations. I cleaned up the trace layout (doing it by hand instead of using the autorouter -- this took many hours, even for this simple board, due to my inexperience). I used thick, heavy traces for all power lines. I added ground planes everywhere that wasn't a trace, and made sure they were all connected together. I rechecked my decoupling capacitors, and added ferrite beads to quell high frequency noise on the motor and speaker wires. I twisted the motor and speaker wires to reduce electromagnetic interference (otherwise clearly visible by waving a grounded oscilloscope probe anywhere in the vicinity). I don't know which of these measures was the most important, but collectively they made the glitches go away, and I didn't feel inclined to try removing any of them.

Peg contact unreliability was another head-bashingly frustrating problem in the first iterations. Peg sensing would work great... around 70% of the time. 30% of the time, when a peg was inserted, it just wouldn't read (empty socket). Twisting the peg a little bit in the socket, or just pushing down on it a bit, would cause it to read properly.

Eventually I did three things: I had the peg PCBs done with gold plating, I increased the strength of the magnets, and most importantly I replaced the spring contacts I had been using.  Pogo pins worked great but cost 50c each, which is kind of a bunch since there were 27 (9 x 3) on each board. Fortunately, I found a product that worked well and only cost 15c or so in the quantity I was using.

Contact under a microscope (low zoom).

There were a couple failures to sense pegs during the second run, but none on the first run -- I'm guessing some of the peg bases had gotten a bit dusty by the second run. Wiping them off and re-seating them seemed to help.

Final assembly

For a while, I was really worried about making some kind of nice enclosure. Laser cut plastic was my fallback plan. But when I made an initial stab at a "rough" laser cut plastic surface (mostly to test peg reading and general operation), I thought it actually looked quite nice and decided to run with it.

There were two layers of plastic. The top layer had the Shintanki name and speaker grille. The bottom layer had cutouts for protrusions from the PCB (pins poking through from connectors on the other side) and a mounting hole for the speaker. Both layers had holes for the pegs and screws, and collectively were thick enough to provide a good "socket" for pegs to fit into.

The top and bottom layers, and speaker.

Layers assembled with the speaker in place.

Once assembled, the layers and PCB made a stack which screwed into the tank base (with stand-offs). The end result seemed quite solid! The PCB electronics were exposed on the underside, but nobody seemed too disturbed by that.

The shape changed several times to improve the tank's ability to nudge around corners and straighten itself against walls. In this demo video you can see an earlier version of the design:


Software

For me, the software was the relatively easy part, since embedded software is what I do for a living. Also, there was already an open source library to do the tricky work of accessing the SD card, reading the FAT filesystem, decoding WAV files, and sending samples to a DAC on a timed basis to play sounds in the background.

This is pretty impressive because the chip (an ATmega328P, selected for Arduino compatibility) is really underpowered! This is a chip that runs at 16MHz and has 32KB of program memory (flash) and 2KB of RAM. Yes, that's kilobytes. My first computer (an Atari 400) in the early '80s had 8 times as much RAM! 2KB doesn't leave a lot of room when you're opening FAT filesystems and reading WAV files. But the library works well and leaves a few hundred bytes free, which is plenty for the relatively simple game logic, peg sensing, motor output and so on.

The curious can browse the project here: http://svn.ofb.net/svn/egnor/dec9/
This is the main program and core game logic: game.cpp

People often ask about the tank voices. They are all synthesized voices, from AT&T's "Natural Voices" demo page or from www.fromtexttospeech.com. Originally these were placeholders, and I intended to record live human voices, but the synthetic voices actually sounded pretty good, and distinct, and meant I didn't need to bother with getting several distinct male and female voices and cleaning up recordings and normalizing volume levels and so on.

Puzzle design

Master design sketch. Note that some behaviors got changed (Captain and Sergeant swapped).
The "MAJOR BUG" is a note to myself to fix a software bug found in playtesting.

As I mentioned before, originally every team was going to get a tank, and there would be several "tank challenges". Eventually it became clear that making that many tanks would be impractical, and staffing several stations would also be impractical.

I also wanted to avoid integrating the tanks too deeply into the game. Until the very last minute I wasn't sure this was going to work! Literally two days before the game, I still had only four working tanks, which was nowhere near enough. (The day before the game, I finished up nine more.) So I wanted to make sure that even if the tanks were a flop, I didn't ruin Decathlon.

So ultimately we settled on having a "tutorial level" at clue site #4 (the Bay Model), and then a final mission at the end. Originally the final mission was going to be one huge maze, something like 6' x 8' in size (the size of the two final mazes put together). I was struggling to make a final maze design that would incorporate most of the characters in some complicated way, but Ana talked me out of it. So I settled on the two final missions that ran in the game: One to extract the Spy, and the final one to use the Spy to capture the flag.

I would have liked to use more Stratego-thematic elements. When I was planning a final mega-maze, I had notions that it would resemble a Stratego board. But I was unable to make that work in any kind of reasonable way.

In the end I had an intended solution for each of the mazes, but I was willing to accept alternate solutions if they worked. Indeed, a popular alternate solution (first employed by the Burninators) for the final maze used a repeating sequence that incorporated both the Miner and Spy in each "hazard" section. A different approach (employed with particular hilarity by CRANEA) was to use the Marshal mid-maze and hope it came out in a decent direction. The Marshal was added mostly because I couldn't figure out what to make the last peg do, and decided to do something that would be amusing, fit with the Marshal's personality from the puzzle, and theoretically serve some purpose (ensuring contact with the flag).

The other goals were to make sure each peg did something unique, and that there were some interesting "modifiers" (the Sergeant and Major). Many teams noticed that combining those two directly (they both repeat the previous action, in the Major's case with a left/right swap) was expressly forbidden. If this was Dominion there would be some careful rule about how that works, but it wasn't needed for the solution, and I didn't want to have teams waste time thinking about it, or make it more confusing to work out the effect of those pegs.

Finally, we wanted to give each peg a distinct "personality" based on the voice and messages, and where possible match it up to the puzzle. We made sure the genders aligned (Colonel Bogey is male, Lieutenant Uhura is female, etc).

Emotional impact

Most people who help run a puzzle hunt knows it can be a weird emotional roller coaster. This one was especially so for me. I ran into really bad timing with a major crunch of job related activity; I spent most of the last two weeks before the hunt flying to Michigan for work. I knew the crunch was coming and tried to get everything lined up in advance but things never quite work out that way. I wasn't sure until the end whether it would even work; the initial yield rate on tank manufacturing was abysmally low (something like 2 of the 13 tanks worked the first time in testing, the rest required some degree of repair); playtest teams ran into slow-cold-tank behavior that I thought I understood (based on refrigerating tanks) but wasn't sure of; all kinds of total meltdown scenarios played through my head. And in the end, it was just a huge amount of work.

Final production verification.


On the positive side, the rockstar team of Brent, Ian and Linda had everything else about the game completely in hand (and you don't hear them whining about the workload, despite pouring in more than me!), so I was able to focus on nothing but tanks.

Once the game is over, there's always a bit of a letdown for me. Suddenly all the stress is over, which is great! (I can return to being anxious about the other stresses of my life.) Running the hunt is a rush, but after it ends there's no real payoff commensurate to the work put in, and there's some whiplash from the sudden release. The Shinteki team run Decathlon every year (and plenty of other private hunts to boot) so they're tough as nails, but I'm still a sensitive greenhorn.

So if I knew then what I know now, I would actually have scaled this project back a bunch. There was just a little too much stress for my fragile constitution.

That said, this was a fantastic learning exercise, it was an amazing privilege to work with the best in the biz, I'm honored to have been part of an absolute core tradition of the puzzle scene, and hopefully I made the hunt a little zingier for everyone who participated. I'd like to give a big thank you to all the players, especially to all the volunteers and helpers, and I the kind comments warm my heart.

I'll close with a few more photos I managed to grab: