Counterfeit parts are a big headache.
It's been a long day. This morning we picked up about 1000 PCBs for Logic and Logic16 and got to work doing the final assembly for the first 200 units so we could ship our back-order tomorrow (Thursday, August 11th). I hate being out of stock, it's such a huge pain for our customers.
We noticed that many more boards than normal were failing the functional test (on the order of 10+%) but we pressed on using the good boards, not seeing anything wrong with the assembly (which is super clean and precise). Once we had the previously mentioned 200 Logics/Logic16s completed, we took a closer look at what could be accounting for all the failures. We soon noticed that the USB chip was running hot. In fact, it was running hot on the good boards as well as the bad ones! We measured the current of the boards, and double checked what it was supposed to be from a couple of boards we had from the last batch. Much higher. Note that this is the exact same PCB in both cases, and we are seeing this same current issue with both the Logic and Logic16 boards.
We had sourced these USB IC parts from Asia as we have for some time, and we immediately started to wonder if these parts were really what they seemed. I had always imagined that counterfeit parts were complete duds - basically no silicon die. I reasoned that there's no way that it would be worth it to actually reverse engineer an ASIC and reproduce it in a functioning form. There was little or no information online about this part, but I did uncover some random snippet that said the part was flagged for possible counterfeit activity just two months prior. Not a good sign. Searching around further, we learned that a huge category of counterfeit parts are what's called "reclaimed" parts, where discarded electronics are stripped of any high value components and then resold in fake packaging as new. This includes cleaning up the part and relabeling it in some cases with a new part number / batch number. Apparently this is a staggeringly large illicit business. http://www.hq.nasa.gov/office/codeq/trismac/apr08/day2/hughitt_NASA_HQ.pdf
We suspected that this might be what was going on - that the part in fact was a reclaimed (used) instance of a really old revision of the part we use - with a different part number -- and one that used far more current - explaining the run-hot behavior and high current measurements. We dug into the various datasheets for this part and its prior revisions / part numbers, and found that there was a hard coded "REVID" register which in a roundabout way reported what revision of the part it was. We rigged up some firmware to pull this register and pull it over to the PC. Sure enough, every last part was an old revision corresponding to a different (obsolete) part number - the parts had been relabeled with a modern part number.
Since we are out of stock and people have been waiting anxiously, we did briefly consider shipping the high-current, but otherwise working boards. But clearly would not be the right thing to do. We have no idea what reliability to expect and what differences it might cause in the firmware in edge cases - plus it runs hot and usesconsiderably much more current that are specification indicate.
The timing of this is terrible, in that we're desperate to get high quality product to our waiting customers - and now we need to rework almost 1000 boards. It's also going to be pretty costly, and not just for the rework. We don't have any idea if we can get any compensation from the supplier for these parts - in part because it's so damn hard to prove that they are counterfeit, you kind of need to be an engineer who knows a lot about this part to even tell. In addition we'll need to order replacement parts to the tune of $10K. Not cool. But I don't care about the cost nearly as much as I do about missing a ship date and upsetting our customers.
We're going to do rework over the weekend on this and hope to be able to ship existing orders on Tuesday. Distributors with open POs are going to have to wait a little longer I'm afraid, but hopefully only a few extra days.
We're refunding everyone's shipping and I'll be sending out an email to eveyone shortly. We're also offering to upgrade the shipping speed of anyone who asks. We very much realize this does not make everything okay, but hopefully it will help a tiny bit.
Back to it.
Doesn't look counterfeit does it?
Update August 11th, 2011
Some images to document the issue:
December 1st, 2011 - 14:08
Counterfeiting of silicon is rampant. I’m told that companies like Silicon Cert (http://www.siliconcert.com), which does reliability testing, are seeing a booming business thanks to all the fakes out there that are making it into the supply chain.
September 4th, 2011 - 07:08
Good Grief –
First hand accounts help drive home how real this is. I need to be more cognizant of this with my own products.
Sorry this happened – Looks like you took the high road did the right thing in reworking and compensating customers. Well done.
August 16th, 2011 - 05:50
“Counterfeit parts are a big headache.” – surely, it is. But it is resolvable, although it cost time (and money).
Much worse is if someone makes the fake copy of your product. The only protection is protect your product by ICs which fakers can’t copy. Any soft protection (like registration of product name, company brand name, intellectual property …) will not help you, if fakers are from China.
Take care.
August 15th, 2011 - 21:49
Just a little note … the deference in texture is most likely due to sanding which is cheaper then solvents …
August 14th, 2011 - 07:44
Wow. Really sorry to hear that. You guys really go above & beyond for your customers.
About 5 years ago I worked for a company that built data acquisition devices. I did all the hardware & firmware design & implementation, but all the manufacturing – which was moved from Taiwan to China to save money – was handled by a pair of other people (the sourcing, PCB fab, board stuffing, test procedures, shipping & customs stuff, etc.)
Sure enough, after the first production (QTY:10K) run in China, subsequent runs started having problems passing test. The supplier was providing counterfeit parts (memories and glue logic chips). Whether or not they were complicit, we could never determine. (I’m sorry if this sounds racist, but never play poker with a Chinese businessperson, you’ll lose your shirt.)
We moved back to Taiwan, and things cleared up. Never had to bring everything back stateside, but the owner was close to sourcing through DigiKey ($$$) and manufacturing in TX or SD. I’m sorry to say that stories like this are not uncommon.
Anyway, I hope this incident isn’t too painful for you guys, you certainly don’t deserve it (but, to quote Clint Eastwood from “Unforgiven” – “Deserve’s got nothing to do with it” :-( )
August 12th, 2011 - 15:00
what a disappointment to find out about the chip relabelling, good detective work though. The rough surface texture of the bad one to me suggests the false chip was sanded down prior to laser-relabelling. Seems like a lot of work to do, but I guess labor is cheap.