Blows my mind that it happened before with the same guy. George Kurtz, CEO of Crowdstrike, was serving as chief technology officer of McAfee when that company released a patch that crashed many of its client's computers.
@MrDarkoiV
2 ай бұрын
Do not worry. He will get his golden parachute. After all it's never fault of executives! They just do too good of a job. It's always these pesky developers and their realistic timeliness, and those useless QA with their bugs and issues, just a waste of money really. Who needs those when you can promise someone else doing the job??
@jbutler8585
2 ай бұрын
Can't punish CEOs, they make all the important decisions! Like cutting costs to the bone so hard it ends up undermining the entire company forever!
@moonasha
2 ай бұрын
I'll never understand how these people consistently fail upwards. Same situation with EA being driven into the ground by a CEO who was then appointed the CEO of Unity, who then proceeded to drive that into the ground before he was fired (Unity is in a much better position now with him gone)
@Kwazzaaap
2 ай бұрын
@@moonasha Because success for them is not about sustainability of a business but how much they could bleed the company before they leave, and then it's not their problem. Hiring an outsider to CEO is all about that, investors want someone who will squeeze and they choose a professional squeezer and then they all collectively move on to doing it to a different company once it starts sinking. Vampires all of them.
@autohmae
2 ай бұрын
Important: he was also co-founder of Crowdstrike, which means he probably created the company culture !
@FeekOps
2 ай бұрын
It's hard for me to explain to people who don't work in technology, or work in devops/data center work how big of a deal this was. It literally took me 3 days to bring some of the systems up. This disrupted so many downstream applications and pipelines for us, we still haven't recovered all the way.
@ThePrimeTimeagen
2 ай бұрын
my friend, if you are defcon, mention you being hurt by crowdstrike, and i'll buy you a beer
@FeekOps
2 ай бұрын
@@ThePrimeTimeagen I didn't end up going this year, but I can only imagine how awkward it must be at Blackhat with Crowdstrike's usual extravagant booth. They typically have a setup as grand as Oktoberfest, complete with indoor fireworks, magicians, and Star Wars LEGO giveaways, among other technical conference attractions that somehow still draw people in. Meanwhile, I'll be continuing my work on the Star Wars Death Star from Invent 2022.
@mwwhited
2 ай бұрын
CrowdStrike is going to have a hard time defending their "limited liability" based on how badly they ignored industry best practices.
@mwwhited
2 ай бұрын
As a note for your distance comment... when I was doing IT it was not considered offsite unless it was out of nuke strike range. If you datacenter and backup build be hit by the same nuke it didn't count as an offset backup. This really isn't too much to ask as 5 miles would be more than enough.
@AlbertCloete
2 ай бұрын
I love how giving a third party vendor kernel level access to your company's computers is considered best practice.
@black-snow
2 ай бұрын
Let's call them "off blast site backups" from now on. We're talking about blast radii anyway.
@tylerfisher4969
2 ай бұрын
@@AlbertCloete that's how drivers work
@EwanMarshall
2 ай бұрын
Even in the US, gross negligence gets you past any such waivers.
@ericmyrs
2 ай бұрын
When I purchase offsite backups, I generally require them to not in fact be at the same site. Hence the name.
@ianbelletti6241
Ай бұрын
The distance off site that's reasonable is calculated by looking at available hazards. In buildings close enough for fire to jump from one building to the next is obviously too close.
@ericmyrs
Ай бұрын
@@ianbelletti6241 generally you want them not even in the same county.
@kasuto-no-machi
Ай бұрын
For our Asia data centers we have to keep them in cities far apart enough that a huge earthquake won’t damage both, hundreds of miles. The idea that a fire jumps from your primary to your DR site and wipes out both is wild.
@elmo2you
2 ай бұрын
I have pointed this out since day 1, because apparently many/most people seem to miss this (even up to now). Many of the affected machines were operating in certified environments/organizations (think e.g. ISO27K, PCI, etc.). Many of these certifications strictly forbid stunts like the kind Crowdstrike pulled here. It's not even about it going wrong, it's simply forbidden in the first place. Either Crowdstrike lied about how its products opreate, or a lot of companies got certified by providing false (or at best incomplete) assessments. That should cause a serious storm of questions in its own right. Not to mention that Crowdstrike should be removed from those environment immediately, or the companies operating them have their certification revoked until that has happened. If Crowstrike indeed officially refused to implement best practices after request for that from clients, then this can quickly turn into a criminal prosecution and even incarcerations. Maybe no in a banana republic like the US (sorry chaps, but unfortunately you really are one), but in plenty of jurisdiction the company will be toast.
@PedroBentoIT
2 ай бұрын
Clowdstrike's lawyers are gonna have to start invoicing in billable decades
@krisavi
2 ай бұрын
Bad for them is also that judgement in one jurisdiction only applies to this jurisdiction, so they have to fight all over again in different countries.
@mattcraftien974
2 ай бұрын
To be fair French judges have a very wide mandate to request independent technical assistance when the subject matter is outside of their knowledge.
@indierodo
2 ай бұрын
what a wild idea, it even sounds like common sense
@mattcraftien974
2 ай бұрын
That's why I don't get the "lawmakers and judge are to incompetent to rule on software development" argument. Lawmakers don't know anything about building a tower with preconstrained concrete slab but the regulations on construction are essential.
@rusi6219
2 ай бұрын
@@mattcraftien974those kind of arguments are made by arrogant Americans who know nothing about the world outside of their own backyard yet still consider themselves entitled to comment on anything that happens ever
@satibel
Ай бұрын
@@indierodo for all the bs we have, france is actually pretty good on IT law, GAFAM (google amazon facebook apple microsoft) often get taxed (read multiple 100M fines) when they try to pull off bs. source : my aunt works at collection and casually strolls in google france with a bill that's like "hey you owe us 650M for illegal data collection" also you can ask a business to nuke any digital data they have on you by basically just saying the magic words "loi informatique et libertes" even for government things, for requesting an id you can ask them to delete your fingerprints from their database after they checked you aren't wanted.
@RMDragon3
2 ай бұрын
If the same accident (flood, tornado, fire, ...) can destroy both your data and your backup, that's not an off-site backup. Avoiding that is literally the point of having an off-site backup.
@tamirsalem76
2 ай бұрын
i guess depends on the scale and the contract. Your off site backup NEEDS to be in another building, but does that entail another state or city? Hurricanes can hit way more etc
@ImperiumLibertas
2 ай бұрын
@@tamirsalem76in the context of a cloud provider yes it is reasonably to assume the off site backup would be sufficiently far enough away to not be vulnerable to natural disasters
@monad_tcp
2 ай бұрын
thus if you did put both in the same place, it was intentional negligence, thus it is totally a crime
@monad_tcp
2 ай бұрын
@@tamirsalem76 yes, it totally does. even when I had an small operation, I stored my backups 400km away from the primary building. before that I had backups, they were stored in another building, but it was in the same city (it was 10km apart), but I still didn't consider it proper off-site backups, so every 6 months I would send a courier with a couple of hard drives to be stored in a vault in another city, 50km away. at least the company could resume operating instead of shutting down in the worst possible scenario the entire city burned down (the city was only 20km, if it was a big city, 50km is not enough).
@monad_tcp
2 ай бұрын
can't we just define a number : offsite is 100km away, easy, done, by law.
@mmmhorsesteaks
2 ай бұрын
We used to have tornadoes, but they got legislated out of existence in the 2007 Treaty of Lisbon (Treaty on the functioning of the European union). Free movement of goods and services also, notably, does not include tornadoes who would be subject to local tariffs equal to or no less than 15%.
@jonahhekmatyar
2 ай бұрын
OVH tried to argue their customers should have used best practices and have an offsite backup while also not following best practices themselves and not storing backups off site? Lmao
@black-snow
2 ай бұрын
Fair point, actually. Just doesn't help the fact they didn't do their job xD
@Jarikraider
2 ай бұрын
I'm sure there's some malware creators out there somewhere that could only dream about inflicting the level of damage that CrowdStrike has caused.
@joseoncrack
2 ай бұрын
At such a scale, that's for sure.
@punkysnarks
2 ай бұрын
The guy(s) who programmed ILOVEYOU must be _fuming_ with jealousy.
@sirius4k
2 ай бұрын
“I felt a great disturbance in the Force. As if millions of computers suddenly crashed and were suddenly silenced.”
@andrew_ray
2 ай бұрын
Your peers are not expected to be knowledgeable on all subjects. That's why expert witnesses exist. They basically come into court and say, in this case, "I'm a professional IT specialist and I have a degree from such-and-so and worked for 12 years at this-or-that company and I can testify from experience that it's standard industry practice for redundant datacenters to be sited away from each other. The reason for that is so that if one datacenter goes offline due to a loss of power or network connection or a fire or other disaster, the other datacenter remains available and the data are still recoverable." Gross negligence is the perfect example of when you want an expert witness because the whole goal is to demonstrate that the defendant has done something so outside the norm that it rises to that level, so you need somebody to testify about what is the norm.
@SG_01
2 ай бұрын
Yeah, this ^^ It is unreasonable for the court to assume it can become an expert by reading up of things, and there would be way too many things they would need to be experts on otherwise.
@thewhitefalcon8539
2 ай бұрын
How far away is negotiable. Often they're on opposite sides of the same city.
@satibel
Ай бұрын
expert witness : *pinches bridge of nose* what? judge: gross negligence it is then.
@ProfessionalBirdWatcher
2 ай бұрын
CrowdStrike is literally the South Park cable company
@isocuda
2 ай бұрын
Ahhhhh they employed BBM, the Boeing Business Model.
@PhoenixtheII
2 ай бұрын
Why the F*** is your elevator directly hooked up to the internet? WTF
@Rynios222
2 ай бұрын
To get Updates! Why does it need updates? Because it's on the Internet!
@autohmae
2 ай бұрын
My guess is the article is a bit hyperbole.
@pilkjaer
2 ай бұрын
We have coffee machines in the office with 3G connectivity and 12inch screens that can show you adds/slides. No wonder there is semiconductors shortage everywhere. Before you just pressed one button to get a cup of coffee. Now you need to navigate a multilevel menu to do the same. Coffee tastes as terrible as before...
@Jabberwockybird
2 ай бұрын
418 I'm a teapot
@joelv4495
2 ай бұрын
@@pilkjaerheck that sounds like smart TVs these days. Wanna keep it air gapped and never connected? No problem, they'll just auto join any open wifi network. 😬
@aajas
2 ай бұрын
People say "don't attribute to malice what can be explained by incompetence"... but when the incompetence is so extreme - it's easier to believe "oh maybe this was simply malice"
@arthurmoore9488
2 ай бұрын
SolarWinds is proof that a company can do massive damage and customers just don't care. One of the major things for companies will be laptops for remote workers. If 15 restarts does really work, then that's an easy fix. However, if it does not, then things get bad. They would need to mail their PC's back and get a replacement mailed to them. Except there are no replacements available!
@MrDarkoiV
2 ай бұрын
Muratori with his 30 milion line problem was right. There is way too many devices that run linux or windows where they should not be. BSD and minix exists. We use way too generic systems for everything, which leads to insane attack surfaces.
@autohmae
2 ай бұрын
Well, that's easy to explain, software is often not cross-platform or limited supported, the people who make business decisions often don't want to spend time on it, let other people spend time on that.
@YaroslavFedevych
2 ай бұрын
Don’t worry, if your computer has an Intel CPU, it’s running Minix too, all the time.
@Moon-zo6hu
2 ай бұрын
@@autohmae then they should be sued and they should lose, unfortunately it's really easy to pretend that you did everything you could do when you really were just doing things as cheaply as possible.
@TrimutiusToo
2 ай бұрын
Us-east-1 is 5+ locations which are more than 80 miles apart from each other
@eventhorizon853
2 ай бұрын
everything under 20km distance can not be considered georedundancy and therefore not a proper off-site backup
@watcherquek263
2 ай бұрын
The industry standard is at least 50 miles/ 80 km.
@digiphaze
2 ай бұрын
What blows me away, is not that they pushed an untested update.. Its that they have an AUTOMATED deployment pipeline that doesn't have any testing in the pipeline. And computers running crowd-strike are typically in environments that require high levels of QA and tight change control guidelines. Poo on companies for being ok with software that can push updates without review in environments that need the change control. And Poo on Crowdstrike for knowing who their customer is, yet STILL not doing a true staged and tested delivery process.
@grokitall
2 ай бұрын
to be fair to the companies, a lot of them were shocked that the n-1 and n-2 policies provided by the configuration tool did not also apply to the live update definition files.
@satibel
Ай бұрын
also they're bypassing microsoft driver certification by dynamically loading executable files in a kernel driver. usually kernel drivers require extensive testing, but that takes time, and so what they do is make a simple loader which is tested and certified as stable by microsoft, and they update the definition files (which are effectively executable files) with the kernel driver. why is that bad? your regular software crashing is handled by windows because it has higher privilege and can block bad behavior, software running in the kernel can nuke everything, so windows does a full stop and bsods. in addition to that a crashing driver should be disabled, but it isn't unloaded because crowdstrike is defined as a driver required for booting. (I think that's where the 15 reboots might come from, windows might finally try to disable it anyway after that many fail boots in a row.)
@henson2k
2 ай бұрын
AWS says Availability Zones in a Region are meaningfully distant from each other, up to 60 miles (~100 km) to prevent correlated failures, but close enough to use synchronous replication with single-digit millisecond latency. Regarding argument that CrowdStrike protected somebody from something I'm not sure about that at all. I have tons of different computers around, some have CrowdStrike, some don't. Literally no difference. Same situation like snake oil antiviruses that do nothing.
@ProfessionalBirdWatcher
2 ай бұрын
Prime really loves CrowdStrike - giving them the benefit of the doubt left and right
@autohmae
2 ай бұрын
I'm also amazed this was possible, it has to be a mistake right ? And not just 1 mistake, disasters happen when multiple systems fail. But who knows... it's just amazing if they found the right technical people without any morals doing this.
@stephanbranczyk8306
2 ай бұрын
He just doesn't want to get sued himself. You have to watch your words carefully on youtube.
@pluto8404
2 ай бұрын
frankly microsoft should be held liable.
@disk0__
2 ай бұрын
I wish he was this understanding when he talked about the Bethesda unionization effort (I'm noticing he took that video down? lol)
@autohmae
2 ай бұрын
@@disk0__ I had noticed the same thing a few days ago... I think he most have realized: not a good take
@marc-andreservant201
Ай бұрын
There was an aircraft incident in the Netherlands a while back, NLM CityHopper Flight 431 flew into a tornado and the right wing fell off due to the forces involved, so yes Europe does get tornadoes but not nearly as often as the US. All 17 souls on board were lost, and an eyewitness to the crash had a fatal heart attack.
@willblanton3120
2 ай бұрын
Regarding OVH, several buildings on the same lot should be considered a single data center with multiple buildings.
@keithmanfredi
2 ай бұрын
Remember in 1995 when people trusted 'Gatekeeper'? 'Offsite' means the same disaster like a meteor won't affect both, let alone the same fire.
@autohmae
2 ай бұрын
depends on the size of the meteor, because if it's nearby: it's just the other side of the city or the next city.
@paulthomann5544
2 ай бұрын
i guess we finally found an actual reason to build on mars :P
@1DwtEaUn
2 ай бұрын
@@paulthomann5544 Iron Mountain's Olympus Mons secondary storage facility ...
@satibel
Ай бұрын
@@paulthomann5544 the old game singularity forsaw that, though it goes up to building in another universe iirc.
@xoso599
2 ай бұрын
An offsite backup would need to be far enough away that a local disaster can't damaged both locations. So a fire might not jump a street, but a flood would. I wouldn't call offsite within the same city.
@unowenwasholo
2 ай бұрын
14:15 No, no, no. That comma is connected to the following comma in a parenthetical clause. It would have been better for them to use the em dash (-), but alas. "There are multiple occurrences of negligence-mistakes or questionable practices in how OVH was operating the service-which lead to the issue."
@Salantor
2 ай бұрын
"I should just read one more line!" Every time, Prime. Every time.
@nicolaschasteler
2 ай бұрын
My company was heavily effected by the Crowdstrike issue. I woke to my laptop on a BSOD which took around 5 minutes to fix after just googling. It then took a full 2 days for minimal operations and the rest of the week to get back 99% of devices. Most remote servers which had no easy physical access could be recovered remotely, while others needed to be completely re-imaged and re-deployed, requiring remaking a lot of custom configuration files. There is still one server which is used for testing in a remote location which is completely inaccessible and has not been recovered yet almost 3 weeks after the initial incident.
@mastersword2829
2 ай бұрын
They better hire a damn fleet of lawyers.
@ProfessionalBirdWatcher
2 ай бұрын
I like how Prime slowly realizes the absurdity of defending CrowdStrike lol
@ianbelletti6241
Ай бұрын
The problem with crowdstrike is that it operates in level 0 but it's definition file is updated in higher levels. Because of this crowdstrike needed a check in its software to check for bad references. It was a bad reference in their definition file that got through the nonexistent software checks in turn activating the Windows crash protection.
@lashlarue7924
2 ай бұрын
Even in the US where corporate law is designed to help manage business risk, many jurisdictions will not honor attempts to self-indemnify for sole negligence, on the basis that it's against public policy. And that isn't even gross negligence, it's just ordinary negligence that they won't let you wriggle out of. ClownStrike has so many different jurisdictions that they operate in that eventually one will stick something on them somewhere, and beyond that the legal fees will be enormous irrespective of win or loss.
@FuturisticFolk306
2 ай бұрын
They likely wouldn't be liable in the United States because of binding arbitration agreements. Binding arbitration is usually used to completely absolve large corporations of responsibility for damages.
@JavaProgrammingify
2 ай бұрын
France mentioned
@Jabberwockybird
2 ай бұрын
Oui oui Baguette. Ça dire de France. Bee-soup!
@geroffmilan3328
2 ай бұрын
Backups are usually in place for Business Continuity Management, and I kid you not, they are usually designed to factor in large-scale natural disasters & even tactical WMD. So they're usually in another city, which wull be what that court looked at.
@Drazil100
2 ай бұрын
I know exactly the type of people who use windows for elevator panels. It’s the type of person who for no good reason hates Linux with an absolute passion and will brute force windows into working even if it’s way more expensive and less convenient. It’s the type of person that believes “I need something that just works and don’t want to spend forever troubleshooting”. It’s people so far into their windows Stockholm syndrome that they believe absolutely nothing will be as easy to deal with as windows is.
@Satook
2 ай бұрын
It's not Offsite if the same, conceivable event could take out both the primary and "offsite" location. That's how we always operated when considering Distaster risk and recovery. Example events being a truck crash, flood, fire, plane crash, building collapse, etc.
@ingowalz9271
2 ай бұрын
Rule of thumb for DC distance is roughly the distance between the 2 nearest airports. Door 2 door DCs really don't make sense (same cabling, base infrastructure etc)
@headlessserpent17
2 ай бұрын
24:50 - In some cases reimaging the device was the only solution. A family friend is an ASL interpreter for hospitals - they can sign like a character from Naruto but does not have the technical proficiency required to unbrick CrowdStrike's gaffe. As a result, their company had to expedite ship them a new computer and intake the old one. Not sure how the company handled removing the corrupted update but the bottom line is the end-user had to obtain a whole new device.
@TinBane
2 ай бұрын
Just on court rulings, they often get expert witnesses in on both sides, and they argue it out in front of the judge, with rebuttal etc. So it’s not a judge just deciding, it’s often shaped by experts. I work in ops on cloud platforms, the norm is you have availability zones that are ideally isolated in space and by geography. Ie you don’t put them all on different sides of the same volcano or all next to the sea. They are situated to reduces systematic risk. Having all AZs or DCs as backups of each-other, but colocated is insane. And if you sell that as redundancy, that is fraud.
@ericw.1620
2 ай бұрын
WRT the law and tech, I generally would not trust congress to write laws, but I do trust the judicial system to determine what's "reasonable" and what's "gross negligence", at least in most cases in the US, and especially when it comes to contracts and liability. The process for determining liability involves looking at industry standards and known best practices, interviewing expert witnesses, etc. Since this doesn't rely on some stupid law written in the 70s by congressmen that spent their free time chewing on lead paint (ahem copyright law), jurors and judges are free to make decisions based on the arguments put forth by lawyers and their witnesses, and the legal standards and precedents from prior cases. This means that the actual people that making claims about what is reasonable best practice are the expert witnesses that actually know what they're talking about. All they have to do is convince the jury/judge. This is in contrast to, say, a copyright case where there are laws directly regulating it. Even if a judge thinks it's stupid to say that Google committed copyright infringement by implementing the Java API, their hands are tied when it comes to their decision because they're making determinations against a law, not a reasonable standard.
@grudley
2 ай бұрын
i, a canadian, am personally responsible for multiple tornadoes. can confirm
@henryvaneyk3769
2 ай бұрын
Too much beans?
@Aliamus_
2 ай бұрын
Did you get credits on the twister movies?
@Jabberwockybird
2 ай бұрын
I thought they were caused by building a trailer park
@c128stuff
2 ай бұрын
Offsite backup, suitable locations have to be far enough from the primary location it becomes unlikely they get involved in the same (natural) dissaster. How far you go with that, what you really cover for, depends on who you are, but that is the general concept. As I am in the Netherlands, and having written requirements for exactly this for the national government here, we typically consider the likelyhood of both getting flooded at the same time (and we now have a 3 backups requirement, but 2 of those can be at the same location). But then, we also have a requirement of different underlying storage technology, to cover for technology related failures. For another job I did a couple years ago, ensuring locations were far enough to not both get caught in the same plane crash was a relevant consideration (as we were directly below an arrival route for a large airport). The exact considerations will be different, but, putting thought into this is absolutely a thing if you actually care about disaster recovery. For my private backups, I keep duplicates about 100km away from where I have the primary backup, and actually both are independently made, and use completely different technologies. Duplicates of backups of my cloud hosted things I keep on the backup server I run at home. But then... doing backups right looks easy, but in reality many organisations don't do this right, and I have helped with data recovery from damaged hardware way more often than I'd like (even if it does pay very well). Doing it right means thinking things through, and not just blindly implementing some 'best practise' without clearly understanding the how and why of it, and having gone through which risks are and especially are not covered. Not because you can cover everything, but you better know about it.
@renerpho
2 ай бұрын
Far enough apart that they're not likely affected by the same disaster, but close enough to not cause latency issues if your sites are communicating with each other. 100 km is a good compromise.
@hanro50
2 ай бұрын
I'd say if a hard drive has the same postal code as the offsite backup, then it isn't offsite.
@bjorn9875
2 ай бұрын
The OVH judgment from the court as you read it about the good backup practice, it seems like the court said "OVH (defendant) says it's good practice to have offsite backup, and OVH said they provided backup, thereby OVH says that had they followed good practice the backups would have been offsite." That does not mean the court even has to understand what a "backup" is lol :D
@MitzaMaxwell
2 ай бұрын
It was not enough to start up the computers manually, there were also some files that had to be searched for and deleted manually and the computers could not be accessed remotely, so the computers also had to be searched on foot, so it took much more than a second to start the computers, so there were far more work-intensive than you mention.
@ZT1ST
Ай бұрын
I think the better guideline for "Backup must be stored in a separate location" is "The exact same cause that brought down the original must not also be able to bring down the backup.". So because it was the same fire for OVH, that was an issue. If it was a tornado, and it destroyed the main building, proceeded to go through more than two street and avenue intersection, and then *also* destroyed your backup, that could be considered reasonable steps. Though ideally, best practice would be for the backup to be in a different city altogether - or at least some of them.
@sakuyarules
2 ай бұрын
"You can't just cheat once and be like: 'it's fine it only happened one time.' ". Tell that to Dr. Disrespect.
@bigfootisjustreallyshy
2 ай бұрын
The funniest part is the broken update was for telemetry. They have hardware and software data from all their users. So, if they aren't running tests on a system configuration they know 8.5 million of their customers are using, then that's gross negligence IMO.
@collinoly
2 ай бұрын
Crowdstrike should definitely pay. But I’m not sure how much. I don’t see Microsoft, aws, or AT&T reimbursing people when they cause outages. Maybe they should be? It also seems like companies like Delta should have their own contingency plans and those obviously failed.
@DaM_Cdn
2 ай бұрын
Wow, I wouldn't have thought The Primeagen would be such a strong advocate for the Oxford comma, but he's all about it!! (cf. ~14:00) Come for the tech discussion, stay for the hot takes on English grammar!!!
@Big3Taxi
2 ай бұрын
14:10 reading comprehension issue, the grammar is correct (a comma would also be "correct" [English is correct if it is understood, not if it follows THE RULEStm] but would be an Oxford comma, an accepted breaking of comma rules for clarity, not a normal comma)
@B20C0
2 ай бұрын
I can give you anargument Crowdstrike probably used against staged rollouts: "Since updates to the threat definitions are time critical (as in bad actors actively using them as attack vectors), the rollout of these updates is time-critical." I'd imagine they'd use something like that.
@satibel
Ай бұрын
unless the company is actively targeted delaying by 30 minutes and using a canary-watchdog is not an issue and if you're actively targeted "cut cable to activate firewall"
@davidmcken
2 ай бұрын
13:07 - Um... Not sure I agree with your assumption, 2 points above "the court acknowledged it was good practice". This is exactly where experts get called in, each side gets to put up their own experts and if anything the judge makes a judgement call based on the competing experts testimony / reasoning. The court at best is assessing a contract and terms / conditions within it. As for how far is off-site, my take is off-campus at the very least. There can be a competing regulation where say health / banking information can't leave the country limiting how far the off-site can be. Having them right next to each other is a clear violation, the argument that the customers should be following best practices when they out-sourced it to OVH scares me and I guess is whats driving the whole multi-cloud push.
@liamconverse8950
2 ай бұрын
US East 1 is like 15-20 buildings in the same area
@szirsp
2 ай бұрын
26:10 "Interesting" math there. Apparently sysadmins fork 24 hour days. But they work "shorter" years (364). And they also fix an order of magnitude more computers than they need (84M instead of 8.5M).
@marcof1430
2 ай бұрын
To be fair in Europe (civil law) lawyer and Judges don't cover the technical issues: expert in the field are used by judge. Basically the expert has more power in this kind of issues than a judge...
@goury
2 ай бұрын
There are a lot of tornadoes in Europe, they just build houses and other stuff the way it's not a big deal. Tornado is just a kind of strong wind, if you don't let it pick up and throw around all kinds of stuff, it won't hurt anyone.
@ZT1ST
Ай бұрын
@26:44; I mean, even that is skipping a step - rebooting the computer into normal mode to make sure that the fix worked, and is sticking. You *could* just immediately move to another computer and apply the same fix without testing that it worked, but...that's what CrowdStrike did.
@YaroslavFedevych
2 ай бұрын
Re: display kiosks, the ones I’ve seen running Windows were also actually displaying some remote desktops. I’ve seen an electronic queue thing boot, it actually had a jumble of batch files that started TeamViewer and remoted into some host that was running the UI for them. Given that your queue number was printed on the kiosk’s local thermal printer, there was some black magic fuckery going back and forth. Also don’t underestimate those machines, if I were a hacker, I could do a lot of nefarious shit with timetable displays. Make people of interest miss their flights and other subtle things.
@someidiotwithnoname
2 ай бұрын
Courts in Europe have a mandatory advisor council on anything the judge or the court doesn't understand. The advisors should be (I say should be because sh*t happens) a party not affected in any way by the outcome of the case (non competitors), a party not affiliated to the accused or the prosecution, a party that is an expert in the field. Those can be University professors or experts from the private sector.
@georgegrundv9933
2 ай бұрын
Id say for offsite backups a natural disaster should be able to occur (flood, tornado, earthquake) and shouldnt affect any offsite backup. ex: 1 offsite backup should have 1 natural disaster in 1 region, 4 should have 4 disasters, and still have backups.
@VallanMandrake
2 ай бұрын
In Germany, there is no class action lawsuit*. Everybody will sue individually. And, as far as I can tell, Crowdstrike is liable (through multiple ways) for all damages.
@Jabberwockybird
2 ай бұрын
It's the fault of businesses putting all their eggs in one E-basket. E-commerce should not be the only way of doing business. Brick and mortar should always exist, and doing business via paper should still be doable.
@renerpho
2 ай бұрын
If you want to do business via paper, come to Germany. Don't forget to bring your fax machine.
@These_Old_Engines
Ай бұрын
The appropriate distance between data centers is about 100 feet more than fire can cross.
@szirsp
2 ай бұрын
27:05 Yeah, sure: It's "sad" that how killing just one person can ruin your reputation after years of not killing anyone... What's sad is that governments let software companies operate this way. If it was hardware and was sold endangering health and safety of users it would be a different issue. There is a reason why CE marking and tests are required (in EU*), or selling deadly poison as food is not allowed. You cannot just do whatever the F you want and claim "provided as is" exemption from liability. I'm not fond of the idea, but it might be required that if you want to sell software in a region it has to undergo some kind of regulatory approval, compliance testing, software safety checks, certification... *USA has similar regulatory requirements: Consumer Product Safety Commission, FTC, FCC, FDA...
@ZT1ST
Ай бұрын
@1:39; I guess that's effectively using RowHammer to fix the issue that CrowdStrike did?
@foolish3art
2 ай бұрын
CrowdStrike will single-handedly employ the next generation of lawyers
@iAmMeAndYouAreYou
2 ай бұрын
Tornados and cyclones are all close to equator. Heat isnt the main thing, its how the air collides and where it is...basically.
@LabelsAreMeaningless
2 ай бұрын
If OVH could be held accountable for a fire.. there's no way that crowdstrike couldn't be held accountable for intentional actions.
@autohmae
2 ай бұрын
OVH wasn't accountable for the fire, they were accountable because off-site backup wasn't far enough away. Small detail I know, but they did something wrong, knowingly.
@blenderpanzi
2 ай бұрын
There are tornadoes in Europe, but much much more infrequently and usually quite a bit smaller. But who knows what climate change will bring.
@YdenMk-II
2 ай бұрын
To be fair about that earthquake comic, if someone's tweeting about an earthquake, it's a fairly minor one and not one where you would need to find shelter for.
@DePhoegonIsle
2 ай бұрын
Data centers where remote backup & redundant data storage is part of the package, it is unreasonable for them all to exist in the same location period. If something wipes out power for a city block and it hits all your data centers at once. That is tooo damn'd close. If a single localized event can destroy all your redundant storage... They are not being stored in a remote location. Plain and simple.
@thekwoka4707
2 ай бұрын
A lot of the liability is going to demand on to what degree negligence was involved. And I think from Crowdstrikes own review, it was very negligent.
@davidmcken
2 ай бұрын
17:25 - Imagine your oxygen machine updating crowstrike in the middle of surgery, the anesthesiologist would sh*t themselves... Windows update is bad enough.
@1990Cookie
2 ай бұрын
Germany has tornados, nut many and not as strong as the US, but we have them. Our brick houses mostly ignore them.
@Dylan_thebrand_slayer_Mulveiny
2 ай бұрын
18:09 how can they possibly refuse to do staged rollouts? Step 1) Have a bank of test computers. Step 2) Create update. Step 3) Push update to said computers. Step 3) Did said computers blow up? If so, roll test machines back and go back to step 2.
@bigpod
2 ай бұрын
In regards to how far away should backups be my opinion always was distance that would protect from semi localized natural disaster
@mz-pd5hw
2 ай бұрын
and to notice the stock prices historically, right now is far from the lowest even in the last year and seems like is already recuperating; I don't think this will have any noticeable effect in the company, some fines to be paid in a period of a couple of years, some will quietly reduced in time, a little "workforce adjustments" to compensate, some accounting magic to move those payments to "expenses" and discount them from the taxable income, some price adjustments in certain jurisdictions, and voila, nothing happened here. Business as usual.
@q1joe
2 ай бұрын
You will never get a jury and judge that will understand CI/CD enough to assign gross negligence
@seancooper5007
2 ай бұрын
We don't have tornadoes in the UK we have Will-o'-the-wisp
@JMurph2015
2 ай бұрын
The French courts seem pretty based to be honest. That's almost as based as Judge Alsup learning Java in order to rule on the Oracle v Google case. I hope CrowdStrike gets their butts handed to them in court, they deserve it for crashing the infrastructure of several countries. With great power comes great responsibility.
@trixer230
Ай бұрын
By the logic in this video (or of the courts) if you are providing back ups, you have to own at least 3 physical buildings (all of which cost millions of dollars) that cant be effected by the same act of nature. 1st - User facing server and services 2nd - User provided back up service 3rd - Company back of customer back ups Say good bye to your low hosting prices lmao!
@stephanbranczyk8306
2 ай бұрын
Actually, the MarketPlace article also says that they're liable. They're just saying that their insurance coverage will take care of it. Personally, I'm doubtful of this. Sure, the insurance will take care of it, but if gross negligence can be proven, which I believe it can, the insurance company will try to recoup its losses from Crowdstrike.
@grokitall
2 ай бұрын
actually there have been multiple cases in multiple jurisdictions, which pretty much all agree that if you are guilty of negligence, willful blindness gross negligence or various specific types of illegality, the insurance company does not have to pay.
@373323
2 ай бұрын
if their testing procedure was anything less than industry standard , they can be found liable i think, if it was, but mistakes where made, as in the testing did detect, but was ignored , or the faulty patch was shipped in error , there is still reckoning to be done, imho ( this is my humble opinion and i have no legal expertise )
@pluto8404
2 ай бұрын
but why are the companies auto updating their systems? You wouldnt update your dependencies in prod. Surely they should be held liable too for not testing updates first.
@373323
2 ай бұрын
@@pluto8404 this what they are supposed to do, basically updating what amounts to a virus signature file , you want these threats updated for immediately, so testing should match the criticality of the task ( its not a virus signature, more like a threat detection signature file )
@grokitall
2 ай бұрын
@@373323there are a number of companies who were running n-1 or n-2 versions of the driver, which crowdstrike support, but the issue here is that it was company policy as stated by the ceo to immediately push the signature files out to everyone in one go, without further testing. the information from crowdstrike is that the engineer in question picked up an untested template, modified it for the case in hand, ran a validator program against it which had not been updated to cover that template (and thus should have failed it), and once that passed, picked up the files, and shipped them out to everyone with no further testing, as per company policy. it then took them 90 minutes to spot that there was a problem, and do a 2 minute fix to roll back the update to stop the rollout and fix any machines with the bad update that had not yet rebooted. it took them 6 hours from the rollout to have a solution to the problem of how to fix the rebooted machines, but it only really worked on basic desktops which did not need security. at least one company reported spending 15 hours manually rebooting and fixing 40,000 machines. some were worse.
@hgbugalou
2 ай бұрын
I think they will be held liable as gross negligence almost always trumps EULAs. The only way this isn't that is if this was done maliciously.
@AUATUWVSH
2 ай бұрын
no amount of "har har this software is offered with no warrantee that it will work properly!", you wouldn't fly on a plane or spaceship where flight critical software is not garneted to work, continue to work (even after update) 110% of the time.
@canoozie
2 ай бұрын
Canadian here. Yes, we're horrible. We ruin everything. You know you're talking to a Canadian when they mention something about how you should thank Canadians for the fact you have a smartphone. Or for the discovery of insulin, or any one of the other things we're all ingrained to know about Canadians contributions to the world. We're great at parties. 😂
@matthamende6359
2 ай бұрын
seems really extreme to reimage, my work used S1 so we were unaffected but my wife's work PC from her company was, I was able to bcdedit into safeboot and just delete the file and it was fine, though her remote IT insisted on sending a new computer so she left that part where her husband already fixed it out because she's been wanting a new laptop for a while, lol
@mcsquared920
2 ай бұрын
14:09 Primeagen Oxford comma enjoyer confirmed
@Purkinje90
2 ай бұрын
I’d be really surprised if Crowdstrike get more than a slap on the wrist in the US.
@adamrak7560
2 ай бұрын
I think the reasonable distance is that you cannot get both locations with the same nuke.
@Chag69420
2 ай бұрын
What's with the bouncing? You got a new Sybian?
@pelic9608
2 ай бұрын
Guy made some good observations, bit really lost me at empty bullet point. The nerves to demand Crowdstrike stage their deploys, but not even unit testing his paragraphs. That's just too much.
@demmidemmi
2 ай бұрын
Most countries do not do the same clown show as the US and UK of having some John Does off the street make important legal decisions. Instead they have highly trained and experienced legal experts that do nothing but evaluate complex legal cases all day everyday make these decisions.
@gdwe1831
2 ай бұрын
Ive got a meterology baclround, and there was a tornado in the channel islands, it was actually only a mile or so away from a datacentre which we once worked with. This is an historically incrediby rare 'blackswan' event and tornados are practically unheard of in europe. However they may become more common due to climate change and extreme weather events are likely to become a more prominent risk globally which will be exacerbated by rising sea levels.
@PedroLeite-q6q
2 ай бұрын
The empty bullet point is a dereference dereference?
Пікірлер: 266