Counterfeit parts are a big headache.
It’s been a long day. This morning we picked up about 1000 PCBs for Logic and Logic16 and got to work doing the final assembly for the first 200 units so we could ship our back-order tomorrow (Thursday, August 11th). I hate being out of stock, it’s such a huge pain for our customers.
We noticed that many more boards than normal were failing the functional test (on the order of 10+%) but we pressed on using the good boards, not seeing anything wrong with the assembly (which is super clean and precise). Once we had the previously mentioned 200 Logics/Logic16s completed, we took a closer look at what could be accounting for all the failures. We soon noticed that the USB chip was running hot. In fact, it was running hot on the good boards as well as the bad ones! We measured the current of the boards, and double checked what it was supposed to be from a couple of boards we had from the last batch. Much higher. Note that this is the exact same PCB in both cases, and we are seeing this same current issue with both the Logic and Logic16 boards.
We had sourced these USB IC parts from Asia as we have for some time, and we immediately started to wonder if these parts were really what they seemed. I had always imagined that counterfeit parts were complete duds – basically no silicon die. I reasoned that there’s no way that it would be worth it to actually reverse engineer an ASIC and reproduce it in a functioning form. There was little or no information online about this part, but I did uncover some random snippet that said the part was flagged for possible counterfeit activity just two months prior. Not a good sign. Searching around further, we learned that a huge category of counterfeit parts are what’s called “reclaimed” parts, where discarded electronics are stripped of any high value components and then resold in fake packaging as new. This includes cleaning up the part and relabeling it in some cases with a new part number / batch number. Apparently this is a staggeringly large illicit business. http://www.hq.nasa.gov/office/codeq/trismac/apr08/day2/hughitt_NASA_HQ.pdf
We suspected that this might be what was going on – that the part in fact was a reclaimed (used) instance of a really old revision of the part we use – with a different part number — and one that used far more current – explaining the run-hot behavior and high current measurements. We dug into the various datasheets for this part and its prior revisions / part numbers, and found that there was a hard coded “REVID” register which in a roundabout way reported what revision of the part it was. We rigged up some firmware to pull this register and pull it over to the PC. Sure enough, every last part was an old revision corresponding to a different (obsolete) part number – the parts had been relabeled with a modern part number.
Since we are out of stock and people have been waiting anxiously, we did briefly consider shipping the high-current, but otherwise working boards. But clearly would not be the right thing to do. We have no idea what reliability to expect and what differences it might cause in the firmware in edge cases – plus it runs hot and usesconsiderably much more current that are specification indicate.
The timing of this is terrible, in that we’re desperate to get high quality product to our waiting customers – and now we need to rework almost 1000 boards. It’s also going to be pretty costly, and not just for the rework. We don’t have any idea if we can get any compensation from the supplier for these parts – in part because it’s so damn hard to prove that they are counterfeit, you kind of need to be an engineer who knows a lot about this part to even tell. In addition we’ll need to order replacement parts to the tune of $10K. Not cool. But I don’t care about the cost nearly as much as I do about missing a ship date and upsetting our customers.
We’re going to do rework over the weekend on this and hope to be able to ship existing orders on Tuesday. Distributors with open POs are going to have to wait a little longer I’m afraid, but hopefully only a few extra days.
We’re refunding everyone’s shipping and I’ll be sending out an email to eveyone shortly. We’re also offering to upgrade the shipping speed of anyone who asks. We very much realize this does not make everything okay, but hopefully it will help a tiny bit.
Back to it.
Doesn’t look counterfeit does it?
