lol, now I'm going to go check and make sure none of my Raspberry Pis have chips that say "Broodcom"
@jolness1
2 жыл бұрын
Mine says “breadcom“, is that not right? Lmao
@vamwolf
2 жыл бұрын
Mine says bootycom
@d00dEEE
2 жыл бұрын
I have a SuperMacro server, but I can't get into the room with it to check anything, it's too big.
@HiltonT69
2 жыл бұрын
Oohhh, I have one that says "Broadcon"
@digitalta
2 жыл бұрын
maybe one of them has mistakenly been installed with a chip that magically supports gpus!
@hjohnson7801
2 жыл бұрын
Look ma, I'm on TV! Great job tracking this all down; I loved the timeline. Rephrasing my new comment from the other video as well since it's more relevant for the discussion here: It's definitely easy to miss, but I also wonder how many people (in the ODM partner, Dell RMA, etc) saw this, had the same thought I did ("oh, this must just be a typo, oops") and decided to not rock the boat. I almost didn't comment because I figured "surely they must know about this already." I figure they did have some good traceability of where the chips were coming from - if your sole defense against counterfeiting is a printed label there's a problem - but it seems to speak to either some gaps in their attention to detail, or to their management structure making it feel hard to bring up an issue like this internally. This is almost the sort of thing that you'd do on purpose as a test of your QC folks.
@jhulbe
2 жыл бұрын
Classic Harry Johnson.
@jannegrey593
2 жыл бұрын
Well done! Now you can tell people that you were instrumental in checking US Military security and you found it lacking. When people ask for details, just tell them that you can't talk about it 🤣. On more serious note - superbly to see that you basically kickstarted the investigation into something that big. I would probably not commented that, just thinking it was a typo or perhaps a label that was supposed to misdirect people as to what exactly is inside. Though the latter would only apply if I saw it on a big show that shows things that are going to government customers or are in prototype phase. Not something that you sell to your customers.
@DavidMiller212
2 жыл бұрын
Good catch dude
@RobTownley
2 жыл бұрын
I am glad you brought attention to supply chain security. Thank You.
@jkp910
2 жыл бұрын
If you caught this, nice job. Feather in your cap.
@menhirmike
2 жыл бұрын
"I know a genuine Panaphonics when I see it. And look, there's Magnetbox, and Sorny!"
@omnipresent990
2 жыл бұрын
Simpson’s never fails
@Rickety3263
2 жыл бұрын
Doh!
@picklerix6162
2 жыл бұрын
My buddy used to own a Shrap calculator. I used to ask him if I could borrow his Shrap.
@TheMhannah100
2 жыл бұрын
Hey I watched this on my Scamsung TV.
@JeanFrancoisDesrosiers
2 жыл бұрын
I saw a Samsnug fridge... No joke.
@danmerillat
2 жыл бұрын
Sounds more like "we've got 50,000 boards with the wrong sticker already built and in the pipeline all over the world, going to just say "it's fine" rather than trying to hunt them all down and re-sticker them" than "we need to save money on printing a new batch of stickers" Great find and research!
@AC-cg4be
2 жыл бұрын
Government contractor here for almost 15 years working in (usually) the civilian sector. . . In all the civilian agencies I've worked in, no one inspects the hardware. Ever. Unless it's dead, it trips IDS/IPS systems, or the tape on the box looks broken, nothing is checked. The usual reasoning is "we need this now" or "who cares" and the hardware just gets hooked up. Things get scanned once OSes are installed, but that's it. Hell, most of the agencies I've been in had people in a mail room or asset management breaking open the boxes to slap asset tags onto gear before anyone that could tell you what was in the box touched it. I've wondered for a very long time how easy it would be for a foreign entity to pull an NSA move, intercept equipment, then put some sort of device in it to siphon data off. Honestly on the civilian side, it wouldn't be *too* hard (probably one of the reasons the NSA got away with doing it for so long). Hell, at one place I worked, they insisted FIPS compliance was enforced on all their gear. It caused issues with Cisco switches and the line cards would randomly fail because the IOS wasn't FIPS certified even though Cisco said it was. Neither Cisco nor the contractors I replaced could figure that one out; I only discovered it by sheer accident of looking at configs applied to a working switch and "nonworking" switch. Anyway, bottom line is that security in the IT industry is a farce and a half all the way from the suppliers to most end-users. It's really sad we're pumping out all these "security" people from universities yet very few people have enough knowledge and acumen to say "uh hey. . .this doesn't seem right" and then ask questions.
@MrAsmith1583
2 жыл бұрын
Management, Executives, and Owners do not care. The whole idea of cybersecurity is foreign to them because they haven't had to really deal with it in the past. Cybersecurity is costly and they really don't like that. If the company's costs go up they can't underbid the competition and win contracts. Most of the management/execs are just trying to cruise into retirement without making any waves. Ransomware will be the only thing that will get their attention and by that time it will be too late and their reputation will be damaged.
@esotericjahanism5251
11 ай бұрын
You make a great point and that's concerning. I recently saw a video about Chinese consumer appliances like these Knockoff Android TV boxes and a lot of the implications were surrounding them were shocking. People are filling their homes up with tech made in a foreign nation that's hellbent on over taking us as the dominant world power and hooking them up to their home networks without a second thought, and we wonder why our infrastructure is under constantly threat of cyber attacks.
@thatspsychotic
2 жыл бұрын
What's even crazier about this story is that, assuming that these stickers ended up in all kinds of models from lots of different vendors, *no one* in the entire IT security industry noticed this or at least reported it.
@UrSoMeanBoss
2 жыл бұрын
Probably the EULA effect. I've been conditioned for years to mentally censor and ignore legal labels.
@jfbeam
2 жыл бұрын
And you fully disassemble and x-ray every server, hard drive, and ram stick going into your racks?
@propellerhead2000
2 жыл бұрын
@@jfbeam It seems like the DoD and its vendors should be. Do they?
@SupremeRuleroftheWorld
2 жыл бұрын
i have seen it everywhere but i just assumed it was just some tax dodging scheme.
@aliabdallah102
2 жыл бұрын
@@propellerhead2000 You expect someone employed in the US government to be competent?
@Enjoymentboy
2 жыл бұрын
Lesson I learned here: Next time this is caught be sure to register the company name, copyright it and register the domain names before you let anyone know.
@leland818
2 жыл бұрын
Even if you did copyright (trademark*) it, they could argue previous use. And regardless whether or not you could win on merit would probably be pointless considering they can outspend you in legal fees.
@hubertnnn
2 жыл бұрын
Yeah, copyrighting is pointless, since in court the winner is never the one with copyrights, the winner it the one that has more money. To be more specific, the one that will not run out of money before the case will end. So in fight between huge companies, the copyright owner will win, but in fight between small people like me or you, money = law.
@ZiggyTheHamster
2 жыл бұрын
This has a super easy explanation: they did a print run of ~100k stickers with the misspelled company name, and doing a re-print would be a ton of work. AMI has a history of misspelling their company name on stickers or having ancient copyright dates that don't agree with the age of the actual software. I've seen BIOS updaters that said "American Megatends". It's sloppy and more of a reflection of their commitment to quality than anything else. It also doesn't help that the people being tasked with doing this work don't speak English most of the time, so are less enabled to discover such a mistake. The supply chain discussion we all should be having is whether the software itself has been written using the most secure development practices, or if they let a team of folks just out of college loose and hoped they'd make something that appeared to work. Embedded computing is typically the latter. This is why IoT devices can end up being vulnerable to really basic attacks.
@0bsmith0
2 жыл бұрын
When there is a complete lack of commitment to quality with tech in general it isn't surprising.
@AnthonyLoPrimo
2 жыл бұрын
Thinking about it, I don't think I ever had an issue with anything that utilized American Megatrends stuff (was so tempted to make a joke referencing a misspeling). I feel like I've seen misspellings before but can't remember, but now I want to look this stuff up haha
@mndlessdrwer
2 жыл бұрын
This is a very good point. A deep dive security test of various BMCs would certainly be interesting, because they're just SO ubiquitous in enterprise equipment and a lot of them are probably less secure than we are led to believe. I will say that there's also a lot of instances where they start out more secure than they end up simply by necessity. For instance, Cisco's CIMC comes with certain features disabled by default for higher security, such as SSH and Telnet both being disabled to prevent low-level access through a terminal to the CIMC module. Many labs will turn those back on for the convenience factor and will rely on their firewall to prevent those ports from being accessible outside of their networks.
@lahma69
2 жыл бұрын
While I certainly have little to no (emphasizing the NO here) faith in the software of any of the big mainstream suppliers of industrial equipment, one thing I can say, being a member of the reverse code engineering community, is that these remote administration interfaces are a very attractive vector for exploitation and that there are likely many talented people looking at these things on a fairly regular basis (if for no other reason than getting a good Defcon presentation out of it). That's not to say that necessarily all of these people are going to be reporting such vulnerabilities in a responsible manner or anything.. but all I'm saying is that it is a very attractive prospect for many people in my field.
@xani666
2 жыл бұрын
@@0bsmith0 When your manager hounds you for deadlines quality goes out of the window. When the customer shops by cheapest, quality goes out of the window When you hire for cheap and don't bother to train people in security and good software practices, quality goes out of the window
@vtheofilis
2 жыл бұрын
So much for the supply chain control from HPE and Dell EMC. These switches aren't on the average home network, they are part of the infrastructure in big and important organisations. Today it's a misspelled label, tomorrow it might be a counterfeit IC with backdoors in place.
@semicolontransistor
2 жыл бұрын
I mean if it's an organization with the resource to counterfeit these things, they are not going to make a typo on a label.
@SomeAngryGuy1997
2 жыл бұрын
Don't worry, genuine parts also have backdoors
@vtheofilis
2 жыл бұрын
@@SomeAngryGuy1997 it' one thing to have backdoors in a country with free press, a democratically elected government, where you can vote, where whistleblowers exist, and another to have backdoors controlled by an autocratic state. It is a lesser evil situation, sure, but the world isn't all sunshine and rainbows...
@vtheofilis
2 жыл бұрын
@@semicolontransistor Yeap, and it would be waaayyy better hidden. Imagine how difficult to would be to take even the slightest notice, if someone would want to hide the fact the chip was out of spec.
@excitedbox5705
2 жыл бұрын
Just think how easy it would be to get a root kit into a bunch of infrastructure systems. Buy misspelled domain. Offer fake firmware updates. Sit back and wait. It might take a while, but as soon as a tech runs into a bug with the BMC and decides to update the BMC firmware, they will land on the fake page and deploy your backdoor for you. This is a much bigger deal than it would seem at first glance because it could jump the barrier into air gapped systems.
@SmokeytheBeer
2 жыл бұрын
I like the way you did the events/response timeline for each group involved. I hadn't seen that kind of interaction timeline displayed that way before.
@ServeTheHomeVideo
2 жыл бұрын
Added that this morning actually. I woke up and realized it was hard to follow without something like that. Still hard to follow even with that chart.
@allhandsonberk
2 жыл бұрын
@@ServeTheHomeVideo You're going to love that you misspelled "rogue" as "rouge" in the timeline.
@ServeTheHomeVideo
2 жыл бұрын
@@allhandsonberk There are Easter eggs in every video.
@NathanGracie-Raitt
2 жыл бұрын
@@allhandsonberk Damn, you beat me to it. Also, TIL my annoying habit of pointing out other people's typo's isn't annoying pedantry afterall. It's high end security consultancy that I've been doing to documents when I should've been doing it to motherboards.
@mdd1963
2 жыл бұрын
I saw a 80 foot tall cylindrical tower at a cement manufacturing plant in Okinawa with large, professionally painted 6 foot tall letters painstakingly printed on the side of it that proudly proclaimed, "Safty first!"
@absalomdraconis
2 жыл бұрын
Safety first, spelling eventually!
@hawk_7000
2 жыл бұрын
What I would like to point out here is that it's impossible to know if a sticker like that is genuine regardless if the text on it is spelled correctly or not. It does indeed look suspicious when it's misspelled (and it's fascinating that no one seems to have noticed/cared earlier), but I don't think the argument that "now you can't tell if it's genuine or not" really holds water as you couldn't tell if it was actually genuine even if it had been spelled correctly in the first place.
@PBRichfield
2 жыл бұрын
A true professional thats well funded and executing a targeted attack wouldn't be so obvious. It's probably more likely this is a false flag to remove these from the supply chain. The last time I reviewed any hypothetical, potential, or real occurrences the sign outside the cyber potty/little hackers room had NOT been flipped to "occupado".
@dangerousmythbuster
2 жыл бұрын
The reason they are continuing to use the misspelled stickers is because someone went into the warehouse, saw that they had millions of the wrong sticker and said "screw it we can't throw out that much money on printing new ones".
@ServeTheHomeVideo
2 жыл бұрын
How much would you guess one of these stickers costs? I have no idea so just wondering.
@dangerousmythbuster
2 жыл бұрын
@@ServeTheHomeVideo I actually have no idea, but I wouldn't be surprised if this was at least partially why they're going to keep using them.
@PrivateUsername
2 жыл бұрын
@@ServeTheHomeVideo To do them today, somewhere in the two cent range. The parent comment is right; they did a lifetime buy on these back 20 years ago after going through a qualification of the vendor regarding the sticker material, the adhesive properties, hot/cold cycles, etc. It's a potential loose part that could take down critical infrastructure if the glue failed and it gummed-up an exhaust fan, for example. And it might change the ESD characteristics of the part. Lots of unplanned RnD would need to be spent. Doing that again today would result in a not-insignificant cost since they would be buying another 20+ years worth all at once.
@n8theb
2 жыл бұрын
Also, it appears that the sticker is on an IC that is on a subassembly that is in larger and varied products and it is quite possible they have work in various stages of various branched processes that use the IC with the misspelled sticker. It would be expensive to trace all of that inventory down and update the sticker in all of the work scattered throughout all of the different stages of production. Generally, sourcing a new batch of stickers for new IC productions seems a little annoying, but doable. Retrofitting a bunch of work in various stages of process is probably many times more expensive (I'm tempted to say an order of magnitude more expensive) and difficult and it might take some time for such work to make its way through the production process.
@survivalSC
Жыл бұрын
Less the 1 cent. And I got that from a friend that is an Account rep that sells stickers to companies. His company makes the stickers everything from soup cans to auto parts and computer parts.
@crikey4336
2 жыл бұрын
I'm certain that I saw a Megatrands[sic] sticker in a server or switch or some piece of hardware about 10 years ago and joked about it with one of my coworkers. At the time it didn't seem totally out of place and we chalked it up to typical english misspellings that were (and still are) frequently appearing in various tech products coming out of china and taiwan.
@nullvoid564
2 жыл бұрын
The Virgin American Megatrends *THE CHAD AMERICAN MEGATRANDS* The virgin Kenwood *THE CHAD KENWEED!*
@KomradeMikhail
2 жыл бұрын
This is not the first time they did this... I have specifically seen American _Megatrands_ before, in a late Socket 7 Pentium MMX era BIOS. Early Socket 5 Pentiums still used the ID of 586, but AMI misspelled it as _568_ . And I've seen _America_ Megatrends laser etched on a 486 BIOS chip.
@danlandia4399
2 жыл бұрын
+1 I've seen Pentium era Megatrands labels Edit: Pentium 66Mhz era
@SupremeRuleroftheWorld
2 жыл бұрын
i have seen it everywhere but i just assumed it was just some tax dodging scheme
@Troppa17
2 жыл бұрын
I had a Deja vu as well but not sure were I saw it before. I know I've seen it in the BIOS itself misspelled before and I think it was an Medion or FujitsuSiemens laptop probably Centrino Pentium M or Core Solo era cause I used to scrap tons of these.
@absalomdraconis
2 жыл бұрын
I've seen much the same.
@azuwan
2 жыл бұрын
It would be cooler if the typo is Megaglands
@jacobharvey2946
2 жыл бұрын
Speaking of American Megatrends… they’ve been around forever. And I feel like I know nothing about them. Every time my computer boots and their logo pops up I think, “one of the retro computer channels needs to do a mini documentary about them. I would watch the hell out of that.” Kinda like how you can find random deep dives on Commodore, Apple, etc.
@booti386
2 жыл бұрын
For their logo on your computer, it's because they are one of the big BIOS/EFI firmware solution providers (others being Insyde, Phoenix (and I probably forgot some)).
@ewenchan1239
2 жыл бұрын
LOVE this! Who would've thought that STH is going to get into investigative journalism in the tech world just because of some guy via a KZitem comment. LOVE IT!!! I think that for your readers and viewers, it would be awesome if you guys did more exposé pieces like this, if, for no other reason, just to make people aware of it. And the fact that AMI is going to CONTINUE shipping products with the incorrect labels/stickers WITH the typo in it - that's just nuts!!!
@arthurmoore9488
2 жыл бұрын
Don't forget that the only reason why they even purchased and reviewed the Dell switch in the first place was because someone there basically asked "Why should we work with you?"
@ewenchan1239
2 жыл бұрын
@@arthurmoore9488 Welp...Dell wanted to know why STH mattered. Now they know.
@PBRichfield
2 жыл бұрын
@@ewenchan1239 next we will be wondering why Bloomberg bought them out for an inflated price and suddenly shut them down. smdh... I hope thats not ever a true statement because i'll be gone too! So for the record, I am not now, nor ever been depressed or had suicidal thoughts. JS...
@marcogenovesi8570
2 жыл бұрын
Nobody knew, and nobody checked it. Most of that goes through so many layers of subcontracting that you could literally attach Peppa Pig stickers to stuff and nobody would notice for years
@802Garage
2 жыл бұрын
Amazing a single typo on equipment the vast majority of us will never touch can make for such a compelling story. This chain of human fallibility is now part of history. Great work! Now, if one visible error on a sticker is cause for concern, imagine how many potential mistakes there are in millions of lines of code, each line only ever seen by a handful of people. ;)
@JamesKao3D
2 жыл бұрын
It's interesting that the number at the bottom of the sticker appears to be a serial number rather than a model number. Then if each sticker printed is unique, and batches of serial numbers are already distributed to different manufacturing sites, decommissioning all of the typo stickers could be more complex than just re-printing them. Existing numbers may already be in numerous databases, existing ranges may have internal meaning, and a new set of ranges that don't overlap and still have the same meaning would have to be re-created.
@dudindmitri
2 жыл бұрын
good point
@PBRichfield
2 жыл бұрын
Trypophobia intensifies......
@gblargg
2 жыл бұрын
So basically, the remaining stock of stickers in all the various places serve to keep track of which serial numbers haven't been used. To replace these without duplicating numbers or leaving any out would be time-consuming and costly.
@willcoder
2 жыл бұрын
Proof that monitoring and management is the LAST and LEAST priority for networking gear manufacturers (recall this is the BMC, not the packet processing backplane). From my years of writing networking monitoring software, I could volunteer as a character witness. It's no wonder the hysterically awful quality of embedded management systems is manifested so thoroughly that it escaped the silicon and affected the freaking label too.
@wmopp9100
2 жыл бұрын
if your royalty sticker says megatrEnds, it is obviously counterfeit. the genuine ones say megatrAnds, easy :D
@TheSleepyCraftsman
2 жыл бұрын
It's called, "It was easier to update the engineering to match the typo than fix the typo".
@Uvirra
2 жыл бұрын
I have been in electronic manufacturing where we were using vendors(OEM/ODM) to make boards for our system. We normally won't be checking on every single labels as those inspections should be done by the vendor themself. As long the system can boot and pass all the test, no further inspections are needed. We would do certain inspection if a particular component had some issues in the past and we want to make sure that it is the correct part or batch. So, the board manufacturer is the one who supposed to do their inspections and should have rectified it with AMI. Or at least they should have reported this to the company who is in charge of assembling the units. Most likely AMI would have pre-printed this by millions and they didn't want to waste money to rectify it. They might have had informed the board makers of it and they would have accepted AMI's explanations. Thinking about my current job as software developer, it is similar case for those who use other libraries as dependencies. We don't really check if the dependencies of the dependencies are trust-able or not. We assume someone down the line had done their due diligence.
@colonelmustard2652
2 жыл бұрын
Swamp gas reflecting off of Venus, much?
@berndeckenfels
2 жыл бұрын
I Hope you check your checksums and signatures and I also hope engineers do sample testing and QA inspection
@a4000t
2 жыл бұрын
Assumption is the mother of all fuckups :)
@ThisIsAGoodUserNameToo
2 жыл бұрын
How many people looked at that sticker and thought "meh"
@ServeTheHomeVideo
2 жыл бұрын
I 100% looked at these and did not notice. Hard to tell unless you are looking for it.
@chubbysumo2230
2 жыл бұрын
@@ServeTheHomeVideo it's funny that it didn't get caught until now, if it had been caught in 2018 during the height of the Chinese spy chip fear, Dell would have likely been forced to recall and repurchase every single switch that had been sold. That said, it still sounds like these should be used as a demonstration at how easy it is to poison a supply chain and have the general buyer never know for years. The idea that the Chinese have put spy chips on all of our computers, sounds all the more plausible when you look at things like this, this has been in the wild since 2018, and not a single person noticed. I bet these are knockoff a speed controllers, but amI has given them their blessing right now because amI cannot keep up with demand if these were needing to be replaced with genuine aspeed chips right now, which would cause both Dell and ami massive money headaches...
@jmr
2 жыл бұрын
That's exactly the kind of thing I would have bitched about as quality control and management would had said "No one will see it. These have to go out. We can take a quality hit but we can't take a hit on delivery."
@alexcrouse
2 жыл бұрын
Some Intern ordered about 500,000 of these labels, and replacing them is more work than it's worth. And the 7 year old building these switches has no idea who American Megatrends is.
@jolness1
2 жыл бұрын
This is awesome. Good investigation. Crazy this is just being disregarded and continuing to use them as well.
@jolness1
2 жыл бұрын
@Conor Hanley yes that seems the most likely explanation for why they would do it, still seems wild to me that they won’t at least change the stickers going forward. Would be one thing to not issue some sort of bulletin or whatever to have them replaced when the machines are serviced or whatever but to do absolutely nothing is 👎🏼
@PBRichfield
2 жыл бұрын
@@jolness1 Nothing to see here... move along.
@kestutissnieska
2 жыл бұрын
I just had a thought, this may be on purpose, as a hologram - those unaware, would write properly and this way one could catch a counterfeit.
@PBRichfield
2 жыл бұрын
@@kestutissnieska good ole false flag operation? Security through obfuscation?
@redtails
2 жыл бұрын
It's super easy to claim "yeah it's a typo, easy to miss, get over it", but counterfeit chips are a HUGE security issue, especially if it's a management engine. Imagine if it was some knock-off chip from a rogue state, now potentially in control of top-secret military servers
@planixxx
2 жыл бұрын
What suprised the most neither Dell nor AMI offered to read out the firmware of the BMC for sanity checking. That makes me wonder if a label thats visible can slip trough, how about non visible things like the firmware loaded on those things ? Are they even validated checked ? I think its more than a licensing issue!
@PBRichfield
2 жыл бұрын
It's called a crc and the hw won't boot without at least that level of assurance. Other vendors are starting to take vastly deeper paths up this mountain but despite their best efforts there is one totally undetectable and unremediable way and it would require a significant, but not, impossible penetration into the supply chain. I'll tell you exactly what it is but I have to go jump out of this 47th floor window first ......
@planixxx
2 жыл бұрын
@@PBRichfield lol. You have no idea what crc is, what is it used for and how it works. I can craft completely different message/firmware that can result in same CRC. CRC is generally used for detection of firmware corruption not for validating firmware. What you meant to say probably is signature verification. In this case root of trust/chain of trust. I did not checked if the BMC has a rom that can do this but even assuming it has there are also points in the supply chain that can be attacked. Just as an example can you see the paints on top of the chips (small dots) they generally mean that those chips are programmed of board (possibly by some other manufacturer). Use your imagination. From an embedded software engineer who worked on similar stuff and "might" have an idea about the issues with the supply chain attacks. I hope I have saved you jumping from that high :)
@PBRichfield
2 жыл бұрын
@@planixxx your ad hominem is 👍 and I enjoyed reading your post as I was waiting on the clipped little winbond 25q128fv in an hp 800 to finish writing. I'm not the one that forgot the password but that's why it was free! Checksums are lame I agree but it's better than nothing otherwise one or several bits get flipped and borks the whole thing! Funny thing about radhardness... But not the point. The point is industry is moving towards better validation albeit at a snails pace but if there's a guy/gal/non-binary/whatevs that's disgruntled or enjoys sabotage at the foundry then garbage in garbage out. I am not at liberty to go any further nor do I have any better ideas so Happy Holidays!
@christ2290
2 жыл бұрын
In other words, the ODM got caught using bootleg stuff and the ODM / Dell paid AMI whatever penalty / fine under the condition they disclose it to the public as a typo. If you contact HP, they'll contact AMI and get the same canned response now.
@JGott0001
2 жыл бұрын
Probably this
@alakani
2 жыл бұрын
That would be foolish of AMI to accept legal liability without examining the die if they're 3rd party chips
@christ2290
2 жыл бұрын
@@alakani they're not third party chips, they're janky min-spelled license / royalty stickers. Watch the video. Same as if you bought a PC with a "Wandows" COA sticker.
@irvingchies1626
2 жыл бұрын
@@christ2290 pretty sure it was really cheap and fast back then to just ship the original stickers, so the bootleg chips are honestly a more plausible thing to happen than it looks like, the scary thing is if they're not only "bootleg" but actually some modified chips, since some of those switches go on very critical infrastructure
@electronicsandewastescrapp7384
2 жыл бұрын
If they could clone the chip... couldn't they clone the sticker?
@TheTigero
2 жыл бұрын
If a STICKER with a clear misspelling on it could infiltrate an entire supply chain for multiple vendors, then we can be SURE that counterfeit or compromised chips or firmware would be able to as well. This almost seems like a canary - someone was testing how much scrutiny any of these components get…
@ServeTheHomeVideo
2 жыл бұрын
That is the big picture question
@ccmangb
2 жыл бұрын
The thing you have to realize is that most technology is not manufactured in English speaking countries. They're manufactured in countries where English is not a priority, where it is used very often but normally not checked for grammar or spelling. This leads to very public and prominent errors that any proficient English speaker would notice immediately. Basically it's par for the course, and when it does happens they just pretend it didn't happen.
@logikgr
2 жыл бұрын
This is probably how extraterrestrial life will be discovered; some guy watching a space video on KZitem saying, "WTF is this smudge?"
@davidgunther8428
2 жыл бұрын
To keep using the stickers with the wrong company name seems a little nuts to me.
@Fee.1
2 жыл бұрын
Couldn’t you argue they’re failing to protect their brand/IP by not correcting this? They have a hard time challenging a company that uses counterfeit stickers like this in the future
@leland818
2 жыл бұрын
They are likely going to register the “American megatrands” mark as well
@Fee.1
2 жыл бұрын
@@leland818 I don’t believe they can. Not with the full protection of the law they’re getting otherwise at least. It’s a nightmare regardless especially for China related issues
@kelownatechkid
2 жыл бұрын
Yeah this is a letter that clearly states they won't attempt to protect their mark in this instance... Someone should get a lawyer on the line lol
@RevenantMike
2 жыл бұрын
As someone who has dealt with supply chain and QA departments... Multi-billion dollar companies not noticing a letter for years? Not as rare as one may think. It's honestly good this was found out and good that AMI confirmed this isn't some trickery of some sort. I'm perplexed why these stickers "will continue to be implemented and shipped" unless that specific sticker format is somehow tied to a qualification/certification of some sort? Just seems...weird.
@marcogenovesi8570
2 жыл бұрын
My guess is that they have a warehouse full of those stickers and they can't be bothered to deal with it
@morosis82
2 жыл бұрын
Or possibly automated QC that they'd have to retrain.
@morosis82
2 жыл бұрын
Would be a classic case of defining the test based on output, rather than the other way around.
@RevenantMike
2 жыл бұрын
I'm not so much confused of them just using the stock they have, I'm more confused they aren't giving a cut in date for revised, fixed, stickers. Deviations are expected from time to time, but usually the deviation period is defined. Maybe just a case of them trying to get a statement out ASAP and no one thought to ask the planners for a date the stickers will change to a corrected version. As I think about it more, I'm guessing different teams got different letters. I'm positive AMI is going to be getting people info on what lot these first appeared, when they expect to change back, and also some supplier and internal corrective actions going around.
@arthurmoore9488
2 жыл бұрын
@@morosis82 Hey I've seen that happen. In a far worse method than this one. Smaller, but far worse.
@OneBiOzZ
2 жыл бұрын
We had someone misspell the company name on a design 6 years ago, it was noticed during an repair and its still there, even through a revision (they forgot to fix it). It's just not really worth anyone's time.
@dvdemon187
2 жыл бұрын
Wow, I just watched about a half an hour long video about a spelling mishap on a sticker on a chip in a switch and I'm not regretting it.
@thejo6331
2 жыл бұрын
Harry Johnson is the real MVP
@pete3897
2 жыл бұрын
I do enjoy the often unspoken humour in your delivery :)
@seylaw
2 жыл бұрын
As a Tech nerd and being a law professional, this is a really fascinating and entertaining story. Great work and a really great video! What a fiasco in supply chain due diligence. By the way, I use a Chinese branded X99 board and I guess my i350-T2V2 NIC is also not the official one, but hey - I knew what I was up to when I bought both over Aliexpress. These people buying expensive switches probably did not.
@ServeTheHomeVideo
2 жыл бұрын
I never practiced, and never want to, but I have a JD.
@seylaw
2 жыл бұрын
@@ServeTheHomeVideo Ah, that is the final piece of the puzzle as the other day I commented on your approach to categorizing SmartNICs which was methodically very similar to my daily work with the law. Keep it up, I really love your channel!
@davidjohansson1416
2 жыл бұрын
What is it that really happened? Also why is it bad?
@seylaw
2 жыл бұрын
@@davidjohansson1416 It really depends on which narrative you want to follow as we don't know all the details yet, if it is just a typo on a sticker (and these things happen), it is about why no one bothered to take notice sooner as such typos are usually a hint for counterfeit products and as Patrick said in his video, supply chain security seems to be an afterthought if such a typo got unnoticed for so long. After all these companies employ people who should ask these questions to their supply chain contacts and provide answers to upper management and eventually to journalists quickly. If the AMI letter was part of a cover up to protect the reputation of all companies involved, it would be a totally different story and heads would roll. But there is no proof of that.
@davidjohansson1416
2 жыл бұрын
@@seylaw so it's worst case a patent/copyright infringement. and else a "harmless" typo?
@colonelmustard2652
2 жыл бұрын
There are a literal boat load of American Megatrends pirated BIOS firmwares on devices coming over from manufacturers in China through even trusted channels like Amazon. They’re rife with misspellings and “evaluation purposes only” disclaimers. This is discomforting given how much access the BIOS has. Who knows what other fun modifications have been cut into that.
@rabidbeaver3250
2 жыл бұрын
Old people are so fucking stupid, how do they now see this a huge security risk and scam.
@DoctorWhom
2 жыл бұрын
@@rabidbeaver3250 Well, those Sorny TV's weren't invented by The Simpsons.
@stefanl5183
2 жыл бұрын
Yeah. and this makes it all the more dubious to me that this is simply a "spelling error". Big companies like this have to know this is going on and how widespread it is. So, why aren't they doing better quality control? Because they're in on it. They save a few bucks here and there which maximizes profit and if they get caught, they simply pay off AMI to take the blame and claim it's just a "printing error".
@thatspsychotic
2 жыл бұрын
Lol... AMI may as well officially change their company name to American Megatrands now! Would be less confusing at this point...
@DrewRushmer
2 жыл бұрын
Great job holding them to task, ultimately an anti-climactic answer that definitely has a few loose threads. You laid out the series of events really well and did a great job, thank you. Looking forward to updates on this one.
@gblargg
2 жыл бұрын
So a counterfeiter could slap the same American Megatrands sticker on without trademark infringement, because that's not the name of any company. This seems stupid for them to continue to do. It's like banks sending emails that look like scams and ask the user to click on a link and enter their login info. It trains users to ignore signs of fakery; it reduces the distance between the authentic and the counterfeit.
@IanBPPK
2 жыл бұрын
There was a company that made knock off PlayStation DualShock controllers that would badge the controllers with FONY instead of SONY. At least they were honest😂.
@ELREASON44
2 жыл бұрын
I would imagine that a sticker on the outside is considered far more important than an internal sticker, as far a spelling mistake-and provided it is fully acknowledged. The way this 'MegaTrAnds' events unfolded is exactly how fiascos begin. If there was actual malice involved, I would expect the label to simply lie and have said MegaTrends rather than one of the 'admitted' fakes by misspellings you find prevalent in such places as China where patents and copyrights have little to no weight.
@j2simpso
2 жыл бұрын
I’d be more worried about The Farmer & The Dell and Hewitt Pickard machines if I were you! 😂
@stefanl5183
2 жыл бұрын
"I’d be more worried about The Farmer & The Dell and Hewitt Pickard machines if I were you! " Why? they probably have genuine parts in them. For them getting busted would be far more risky and far more costly. AMI is not going to publicly embarrass a large customer like Dell. Instead they are going to cut a back room deal and cover for them as they clearly did here.
@SISSYPUSS
2 жыл бұрын
American Megatrends gets a new stock keeping unit, servethehome gets a new stock keeping unit, EVERYONE gets a new stock keeping unit
@SenileOtaku
2 жыл бұрын
in my brief time working for a QA department at Cisco, I remember some security team specifically looking for counterfeits slipping into their own supply chain, as well as completely counterfeit units being sold outside of it. And I'm sure the likes of HP and Dell would prefer to just cover the story up rather than risk tarnishing their worthless names by admitting their supply chain could readily be compromised. Now if the BMC firmware is readily updatable, I wonder if any units the CCP compromised could be readily re-flashed to properly secure versions?
@PBRichfield
2 жыл бұрын
@@Android-ng1wn Too late. Planned obsolesce was invented by Maytag shortly after they were acquired by Chevrolet who continues this age-old craft but with modern flair!
@oldtechdude
2 жыл бұрын
Exceptionally well done. Did a post on LinkedIn to spread the word. Shows that QA is not what it should be, across the entire supply chain. It highlights just how easy it might be to inject something unintended into critical infrastructure hardware.
@ServeTheHomeVideo
2 жыл бұрын
Thank you Bruce.
@DasIllu
2 жыл бұрын
So we can now officially rename one company to "American Megatrands" and the other to "Dull" 🤣
@jasoncherry2508
2 жыл бұрын
...Now that commenter is the BIG "Harry Johnson"!! Way to go bud, keen eye.
@plonk420
2 жыл бұрын
I feel I've seen MegatrAnds on a desktop board before, but I have a pretty low confidence in that 😂 edit2: i only found AMIBIOS and a massive socketed AWARD chip... oh well.
@marcogenovesi8570
2 жыл бұрын
It's Microsoft "WinDOS"
@ServeTheHomeVideo
2 жыл бұрын
You are more creative than I am!
@nekomakhea9440
2 жыл бұрын
Actual investigative journalism still exists! Not all heroes wear capes
@ServeTheHomeVideo
2 жыл бұрын
Thank you for the kind words.
@scottb721
2 жыл бұрын
Had a similar thing where I pointed out to a European streaming radio manufacturer about one of their products in an Australian store with what I think were some questions. They were like wtf, that's a clone. We need to look into that, other something to that effect.
@Pr0toc01
2 жыл бұрын
PLOT TWIST: "Harry Johnson" is actually the person that designed the incorrect sticker...
@BansheeBunny
2 жыл бұрын
It was nice of you to snatch up those domains. I too wouldn't cert them, no IT professional worth their salt would except the risk.
@Graham_Wideman
2 жыл бұрын
Of course, an IT professional who wouldn't "except" the risk probably wouldn't notice a typo in Megatrands.
@dudindmitri
2 жыл бұрын
My personal oppinion is that it is not proportional comparison when we speak about "Wandows" logo that anyone can easily observe and a sticker on the switch chip, that is not exposed to a regular customer or user. In addition most of the switch components are non-FRU and those that are, do not require you to disasemble the switch so this sticker can really be noticed when you ship this switch back to vendor for repairs... My biggest frustration is how all those HP, DELL, CISCO.. switches manufactures deal with switch FANs, that are - non FRU and no information on the model or part number is mentioned in vendor manuals. Without this ''amazing'' feature your switch would serve you ''forever'', just replace fans and SFPs from time to time... So vendors just found the reason to sell new switches with same non-FRU fans. My collegues usually dispatch a new switch for such cases while i always go for cleaning the switch and replace all fans. Great video. Subscribed.
@benjamindrake6065
2 жыл бұрын
I deal with labeling issues all the time. Normally a supplier creates a label, no one double checks it, it gets found years later by a new assembler and we burn tons of hours contacting the customer to let them know it is out of spec. Then we look at what we have in stock and on the manufacturing flow and decide what to do going forward. It is a lot of fuss for a $0.02 sticker
@xiphias256
2 жыл бұрын
AMI doesn't bother with this because it's a sticker inside equipment not typically opened and looked at by the end user. Especially switches without any user serviceable parts. A "Wandows" sticker on the outside of a PC, for everybody to see, is another story. I'm pretty sure Microsoft would have had to take action to regain its reputation. :-)
@dsebrock
2 жыл бұрын
Missed opportunity to reference Michaelsoft Binbows
@brainyquizchallange
2 жыл бұрын
Crazy! I've worked in OEM licencing, Dell being one of them, if there was even one thing out of place on a sticker we would destroy the whole lot and start over.
@UltimatePerfection
2 жыл бұрын
While it may be a fake, it could be also a mistake during printing of the label. In the company I work at, recently we got some labels for our products with a spelling mistake similar to this. The owner told us to just use it up because he didn't want the money he paid for printing those go to waste. And so there are some packaging that says "Hleb wiejzki" instead of "Chleb Wiejski" floating around. Obviously we've changed the printing company after that.
@gleep23
2 жыл бұрын
Excellent investigation! Good on you for respecting the business involved and their own internal investigative processes. You deserve alot of respect for not announcing some exaggerated claim on day 1.
@the_beefy1986
2 жыл бұрын
So they're not fake chips? And AMI is totally okay with letting this confusing situation continue? WHAT?!
@tacticalcenter8658
2 жыл бұрын
Follow the money.
@absalomdraconis
2 жыл бұрын
This is _not_ the first time AMI has made a typo like this. This is not the first _century_ that AMI has made a typo like this. In fact, at this point, it's possible that AMI spelling docs for these stickers actually _say to make_ these typos.
@stefanl5183
2 жыл бұрын
Think about it. Dell is probably a huge customer for AMI and the customer is always right, especially if the customer promises future big money contracts in appreciation for you falling on your sword to save their reputation. ;) Expect AMI to get lots of future business from Dell and for that business to be very profitable for them.
@stefanl5183
2 жыл бұрын
@@absalomdraconis "This is not the first time AMI has made a typo like this." Translation: "This is not the first time AMI products have been counterfeited."
@Braiam
2 жыл бұрын
I would tack this to the "humans read coerrlcty worlds even if they are on different order" category. They are probably more preoccupied with having the correct feature set and no undesired feature in the systems that everyone treated them as a kind of black box and trusted upstream that they checked it out.
@ServeTheHomeVideo
2 жыл бұрын
That is rihgt!
@noahluppe
2 жыл бұрын
the ole pattern recognition saving energy
@DoctorX17
2 жыл бұрын
AMI has made this type of error before, lol. I had worked on machines in the 90s that said “AMERICA MEGATRENDS” on the BIOS chips. I think it was Dell? I bet people have noticed, but just said “huh, that’s funny”, maybe mentioned it to a friend, but didn’t think anything more about it… we just got lucky that you saw a comment where someone noticed Also, I’m new here, hi
@ServeTheHomeVideo
2 жыл бұрын
Hello and welcome!
@DoctorX17
2 жыл бұрын
@@ServeTheHomeVideo thanks :D I think I'm gonna like it here!
@curranhouse
2 жыл бұрын
You should make up some t shirts - STH American MegaTrands "Typo-Mini computing"
@XshlomoX
2 жыл бұрын
I just love the content that you put out. It's so niche, but so well made. And for someone like me, who is actually interested in these things, this is just perfect!
@ServeTheHomeVideo
2 жыл бұрын
Thank you!
@jfkastner
2 жыл бұрын
How would the Copyrights / Trademarks hold up legally if they do NOT replace those stickers?
@ServeTheHomeVideo
2 жыл бұрын
Good question
@johnturnbull7798
2 жыл бұрын
Forgive me for being a cynic but think about this. Sure this is an upstream supply issue but can we really be certain that a third party contract manufacturer is not using cloned parts? Would they admit it and would either Dell or AMI admit to the problem if they thought they could get away with it? Yes it could be a typo and maybe it is, however, how many of us are bombarded by IT departments telling us that typo's and misspelling are the first indication of companies masquerading as someone else. This is a PR disaster for them and registering those names after the fact doesn't cut it.
@JSLEnterprises
2 жыл бұрын
I've seen american megartrends stickers in older dell switches that said American Megatrands... as well as some mellanox core switches as well. That typo has been shipping out on stickers for quite some time.
@udirt
2 жыл бұрын
Imagine the guy who ordered 500m stickers and didn't notice until accepted and stocked and shipped to clients. Uhhhhh...... Oooops....
@absalomdraconis
2 жыл бұрын
@@udirt : Imagine being the guy that has _kept_ making typos since at least the 90s.
@nagi603
2 жыл бұрын
Pure comedy :D "Hey, we spent about $100 on printing a metric ton of these stickers, why waste that?"
@EsbenBloch
2 жыл бұрын
Brb starting a new company. We already have product out it seems. Also looking into magatrands being a thing.
@mrmarkom
2 жыл бұрын
Unlike windows stickers that are in front of our faces, people usually don’t open their machines to see the typos on chips.
@eastwood4
2 жыл бұрын
@ 9:18 "rouge[sic] reseller"? STH getting in on this new trand? 😂
@ServeTheHomeVideo
2 жыл бұрын
:-) easter eggs in every video
@FinalSentinel
2 жыл бұрын
Really strikes me as something adjacent to the Mandela effect and the Berenstain Bears. Can be easy to misread and flip e and a in words like this. The human brain doesn’t read individual letters, but the overall shape of words, so not surprised so many people could skim by it.
@lahma69
2 жыл бұрын
This has got to be one of the most interesting.. umm.. "random, happen-stance" type tech videos I've watched in quite a long time. It was almost like watching a murder investigation documentary or something (which by the way, I do not like) that keep you wondering until the last moment. Personally, I find it VERY bizarre that American Megatrends would publicly say that not only is this their sticker but that they will continue using it moving forward, despite being aware of the issue and (surely?) knowing that this does not engender feelings of certainty concerning supply chain security in IT professionals. These stickers can't cost more than a couple of pennies each.. Why denigrate your brand and its image over such a miniscule expense.. Do they really just not care?
@PBRichfield
2 жыл бұрын
Or its a massive coverup? Things happen every day for reasons you, or I, don't or won't ever know the true motives, methods, or actors behind. There exists a field of study that aligns itself with Science and some Ancient Astronaut Theorists agree that it not only exists today but has for millennia!
@MattSuguisAsFondAsEverrr
2 жыл бұрын
Every computer in my former school has American Megatrends BIOSes megatrands really made me intrigued
@ryanfoley8035
2 жыл бұрын
well on an SGI octane they misspelled copyright those were $20-80000 graphic workstations in 1996.
@wolfgangloll2747
2 жыл бұрын
Honestly, they must have several thousand boards in circulation and no one is keeping track of when they ordered another sticker. They will now order new stickers and fix the error, but they won't do a recall because of it. the stock gets used up and no one opens the boxes and sticks on new labels. And who has a comprehensive QM for internal stickers?
@sofascialistadankulamegado1781
2 жыл бұрын
A sticker with a spelling issue got through so many processes unnoticed. How easy would it be to sneak in a counterfeit chips with backdoors. Wow, just wow.
@ServeTheHomeVideo
2 жыл бұрын
That is the question
@CCL13CN
2 жыл бұрын
I work for one of the major cloud platform. What I can say is we have more than just this sticker, and a lot of random errors in the hardware manufacture. The sticker to be honest is just one very minor thing, and that's why the manufacturer just don't want to know about it.
@GalvayraPHX
2 жыл бұрын
You're probably right...The quality these days is definitely 'Wandows'-level....
@SytheZN
2 жыл бұрын
I can quite easily believe that this went unnoticed. 'a' and 'e' are the same shape, just rotated. Captial letters temd to make most words just a block as well. Unless you're using AI/ML, and specifically running spellcheck on every component on your boards, chances are the only place they coud reasonably have been noticed was human eyes. We're not really programmed to read individual letters unless we're trying to pronounce or interpret something unfamiliar, rather we recongise the overall shape of words...
@jfbeam
2 жыл бұрын
99.9999999999999% of people don't read it. You see the "MegaRAC" line and notice the "(c) 2003" below it without looking, and then your eyes focus on the serial number, completely skipping ("filling in") the rest.
@thomasflake1404
2 жыл бұрын
Human eyes can skip right over misspelled words. I read your post twice after seeing “temd”. I found “Captial”, “coud” and “recongise” too. (I’m just joking around!)
@AndrewBucklin
2 жыл бұрын
Great video! This was better than any episode on Investigation Discovery channel! And purchasing those domains was an epic move! Now you just need to trademark that name (since I doubt AMI is going to go through that trouble, even though they decided to continue using the stickers, lol).
@ServeTheHomeVideo
2 жыл бұрын
I think they do a better job. This is not what we normally do.
@AndrewBucklin
2 жыл бұрын
That just makes it even more impressive. I was on the edge of my seat and was very impressed with the way you handled everything with the parties. And the presentation of the facts in the video and the timeline was 💯! Too bad these characteristics are so uncommon in the media industry these days. Kudos! 👏 👏 👏 Keep up the amazing work!
@nibrobb
2 жыл бұрын
3:50 Couldn't help but think about Michaelsoft Binbows
@carlkamuti
2 жыл бұрын
So it IS a typo? Well, I've got to give it to you: Journalism break of the century dude. Not in living memory have I seen a scoop like this. Not since the likes of Snowden have we seen the corridors of power groan under the weight of technological journalism!! I'm calling it right now, Pulitzer!! Boys, this will be your epitaph, mark my words. Very well deserved.
@cll1out
2 жыл бұрын
Can't imagine how many heads would have rolled if Apple released the iPhone 13 and it said "Designed By Epple in California" on the pamphlet in the box
@ServeTheHomeVideo
2 жыл бұрын
More like the product itself, not even the pamphlet. Then the question is if it was not discovered for two years after release
@KillaBitz
2 жыл бұрын
Just opened my Dill desktop looking for similar mistakes but didn't find anything. Thank Dog for that.
@summerlily2123
2 жыл бұрын
*with fams atm for Xmas dinner.... showed this to my dad & uncle, my uncle [former military, etc] has knee-jerk reaction "Has anyone contacted the military to make sure we haven't been compromised with spy tech!!" I was LOL'ing so much I started crying! My dad looked at me with raised eyebrow and stern face which made promptly stop laughing, then I said to them both "Keep watching.. this is classic interesting"!! *I winked at Dad* And when it was done, We all had the same question... "So was it a typo?" My Uncle STILL thinks there's some conspiracy brewing here and is already on his phone making calls. lulz Dad just sighed and said "Hey Kitten, why are u even watching this? Isn't this out of the realm of your anime and pop idol friends doing streaming?" I giggled and said "I left the auto-stream running while I went shopping with Mom awhile ago, this was playing when I came back, however, U have to admit this is an interesting story, right? He laughed and patted my head and said "Good job finding something interesting enough to take you away from your V-tuber friends!" ❤️😌😂❤️
@ServeTheHomeVideo
2 жыл бұрын
Ha! Merry Christmas.
@jfkastner
2 жыл бұрын
Would be interesting to look at eebay product photos and see how often this happens ...
@ServeTheHomeVideo
2 жыл бұрын
Few people open these up
@TheDalen2016
2 жыл бұрын
Outstanding find. Thank you for sharing.
@NetRolller3D
2 жыл бұрын
Now I have to go look at the sticker on my motherboard, to make sure this is not another Berenstein/Berenstain Mandela effect thing...
@AidenPryde3025
2 жыл бұрын
Bet you they have millions of these stickers already printed and AMI didn't want to eat the cost to fix the typo.
@Harani66
2 жыл бұрын
It may well be that they are going to change the label. but by releasing a statement saying they're still going to ship with the old one, means they do not have to recall possibly thousands and thousands of warehoused units that are already manufactured but not shipped to end users yet
@forbiddenera
2 жыл бұрын
@3:50 total missed Michaelsoft Binbows reference opportunity
Пікірлер: 895