Recently, I've been developing a Raspberry Pi prototyping board - the "Wombat".
I'm now in the final stages of getting it ready for manufacturing, by the same company who build my PIC training boards. And the final stage of manufacturing is testing - every board must be properly tested at the factory before it is shipped. And of course I have to create the test procedure for the factory to use.
How hard can it be to create a test procedure? Just check that everything works, right?
Sounds easy, but the testing has to be done in a repeatable way, without too many manual processes, because we want to limit the opportunities for human error and keep costs down. I've wanted to automate as much of the testing as possible - ideally the test procedure will be: plug the shiny new Wombat board into a Raspberry Pi, run a Python program that puts the board through its paces, press the pushbuttons, check that the LEDs light, and look for "PASSED" on the screen; move onto the next board and repeat. If it says "FAIL", go fix that board.
That's the theory behind the test setup shown here.
The Pi connects to the Wombat via a 40-pin IDC cable, in the usual way. That's great for testing the LEDs, pushbuttons and ADC, but what about the Wombat's USB/serial interface?
That's what the USB cable is for... By connecting the Wombat's USB port to one of the RPi's USB ports, it's possible to check that whatever serial data is sent to the Wombat board is exactly what's received, and vice versa. That tests the micro-USB connector, the FTDI USB/UART chip, and associated passive components and PCB traces, including the connections to the RPi's RXD and TXD pins.
Excellent - so what's left? Power!
The Wombat board includes a 3.3V regulator, able to supply up to 500mA. It is powered from a 5V rail that is supplied by either the micro USB connector on the Wombat board, or from the Raspberry Pi (via the 40-pin IDC cable).
So, if we're not getting 3.3V from the regulator, either the regulator (or an associated capacitor) is faulty, or it's not getting a 5V supply, meaning a fault with either the RPi's 5V supply, or the 5V coming from the Wombat's USB connector.
We're testing the Wombat board, not the Raspberry Pi, so we want to make sure that what we're testing is the USB power on the Wombat board, not the 5V coming across from the RPi.
So, to test the power supply, we shouldn't connect the Wombat to the RPi's 5V supply pins.
That means that we have to "break" that 40-pin IDC cable somehow, so that the 5V pins aren't connected.
To do that, I made up some stripboard, with two 40-pin connectors, as shown. It would act as a "pass-through" connector, sitting between the IDC cable and the Wombat board, but only connecting the pins I want to have connected - limiting what Wombat pins connect to the RPi.
And, as we'll see, it also makes it possible to add extra components, such as resistors, inline with or between some of those pins.
So now, if there is some power-related problem on the Wombat board, including no 5V power supplied via its micro USB connector (maybe caused by a dead or missing ferrite bead), there will be no 3.3V output from the onboard regulator.
How to test for that?
The MCP3008 ADC chip is powered from the 3.3V regulator.
If that chip has no power, it can't do any analog to digtal conversions, so the ADC tests will fail, right?
So, that would be a way to confirm that the power supply is working, right?
If the ADC tests pass, the Wombat's 3.3V regulator, and the 5V supply from the Wombat's micro USB connector, and all the associated passive components and PCB traces must all be ok, right!
Yeah, I thought so, too. It stands to reason. No power, no ADC readings.
But I was wrong. At least at first.
I connected the Wombat board to the RPi, via the pass-through connector, and with the USB cable in place (supplying 5V USB power to the Wombat board) as shown. All tests worked. Great!
Then I unplugged the USB cable.
With no power coming from the Raspberry Pi via the 40-pin cable, and no power coming from the micro USB connector, the Wombat board has no source of power and therefore wouldn't work.
To my amazement, the ADC tests worked just fine.
Of course, I checked that the 5V pins really weren't connecting to the Rasperry Pi. My continuity tester failed to beep...
Then I measured the 5V rail on the Wombat board: around 2.5V. Weird.
I also measured the output of the 3.3V regulator: also around 2.5V. Also weird.
Where was this 2.5V coming from?
Whatever it was, it was enough to power the MCP3008 (which has a minimum supply rating of 2.7V, proving again that devices often do manage to operate a little outside their rated limits...).
Maybe there was some pathway for power somewhere that I wasn't seeing?
Some strange effect caused by something else on the Wombat board, perhaps? Unlikely...
Nevertheless, I took a "back to basics" approach, making up only the ADC section of the Wombat's circuitry on a breadboard, as shown here, to ensure that there was nothing else affecting either the power supply or the ADC circuit.
Taking no chances, I used a bench power supply for the 3.3V rail, and of course this little circuit worked just fine. More importantly, when I removed the 3.3V supply, it stopped working - just as a well-behaved circuit should!
So, there must be something different about this same circuit, as implemented on the Wombat board - or something else on that board having an effect. Something that will make this circuit work with no power supply...
And then I realised - the Wombat board has a capacitor on the output of the 3.3V regulator!
Adding a 1uF capacitor across the "power supply" rail (in quotes because, by all rights, there is no "power supply" here!) on the minimal breadboarded circuit, as shown here, recreated the strange behavior. The ADC circuit worked, passing the tests, even with no power applied to the Vdd pin on the MCP3008.
All it needed was that capacitor between Vdd and ground.
Ok, so where was this power coming from?
Clearly, not out of thin air, so it had to be coming from the Rasperry Pi - and the only connections were the SPI interface pins: Din (MOSI on the RPi), Dout (MISO), CLK (SCLK) and *CS (CE0).
It took a while, but eventually I realised what was happening - and felt really foolish for not seeing it earlier, especially given the evidence of 2.5V appearing on the "3.3V" supply.
Most modern integrated circuits include protection diodes on their digital input pins.
An example is included in the generic I/O pin circuit shown in the PIC16F1824 data sheet, and reproduced here.
Although there is no PIC on the Wombat board, the MCP3008 (and indeed the FT232RL USB/UART chip) has a similar arrangement on its digital input pins.
The idea is that if a voltage higher than Vdd or lower than ground appears on the input pin, one of the internal protection diodes will conduct, restricting the votage at the input to a safe level, and protecting the chip. They are one reason why CMOS ICs are much less sensitive to static discharge (ESD) than they once were.
Like most silicon diodes, the protection diodes drop around 0.7V when conducting, so the input voltage is actually limited to between -0.7V and Vdd+0.7V or so.
And so, finally, it all makes sense!
In the absence of a power supply, the protection diodes feed power from the digital input pins to Vdd.
The *CS pin (connected to CE0 / GPIO 8 on the RPi) is normally held high, that is, at just under 3.3V. This provides power, via an internal protection diode, to Vdd. The diode drop of around 0.7V explains why I was seeing around 2.5V (= 3.3V - 0.8V) on the "3.3V" rail.
Power goes in at 3.3V through the *CS pin, via an internal diode, coming out of (not into!) Vdd at around 2.5V.
When an ADC sample is taken, *CS is pulled low, removing the chip's "power supply", and the ADC fails.
BUT - when a capacitor is added between Vdd and ground, the story is a little different. Now, in between samples, CE0 is held high, supplying 3.3V to the *CS pin, which in turn supplied 2.5V via an internal diode. This then charges the capacitor on Vdd to 2.5V. When the ADC sample is performed, *CS is pulled low, but now the capacitor suppies current to Vdd, keeping the chip alive long enough to complete the ADC sample.
Now, how to make the ADC fail, with a capacitor on the supply rail (as on the Wombat board), when the power supply is removed?
A simple solution is to place a resistor inline with the *CS pin, to limit the current that the RPi's CE0 signal can supply; when the MCP3008 chip has a "proper" power supply, the *CS pin should not draw much current (no more that 10uA, according to the data sheet), so an inline resistor should not affect normal operation.
So I tried placing a 10k resistor between CE0 (GPIO 8 pin) and *CS, but to make that work I also had to increase the pull-up resistor on *CS ('R7' in the circuit diagram) from 10k to 100k. This pull-up ensures that, when jumper JP10 is removed to disconnect the CE0 signal from the MCP3008 (as you would need to do if you wanted to use GPIO 8 for general purpose I/O, or if you wanted to use a different chip as device 0 on the RPi's SPI bus), the *CS pin is held high, placing the MCP3008's digital pins into a high-impedance state, effectively disconnecting the chip from the SPI bus. A 100k pull-up is still effective, and is a high enough resistance to not interfere with the CE0 signal, even when that signal is connected via a 10k resistor.
That all worked fine on the breadboard, but even with the 10k resistor inline with *CS and after changing R7 to 100k on my prototype Wombat boards, they still failed to fail when the power supply was removed...
But now it was easy to see why. The FT232RL chip was doing the same thing, feeding the 3.3V signal from the RPI's TXD (GPIO 14) via the RXD pin and through an internal diode to come out of the FT232RL's Vcc pin and onto the "5V" rail at 2.5V - explaining why I'd seen 2.5V there as well.
To address that, I took a different tack - I placed a 100R resistor between "5V" and ground, via the pass-through connector. That creates an additional 50mA load on the 5V supply - not enough to affect normal operation, when there is a "real" power supply, but it's enough to drag the signal-derived supply down enough that the 3.3V regulator will no longer function, stopping the supply of power to the MCP3008.
So, finally, after days of investigation, with the addtion of a 10k resistor on the CE0 signal, a 100R resistor across the 5V power rail, and increasing the pull-up on *CS from 10k to 100k, the Wombat board failed, as it should, when 5V USB power was removed!
Which all means that, if a board has a faulty regulator or micro USB connector, it will (reliably?) fail the testing, and I can be confident that a faulty board won't be shipped to a customer.
Who would have imagined that failure would feel so good? :-)