Thanks for watching! I thought I would try something slightly different with this video, focusing a bit more on telling a story. I had a lot of fun with it. Open to feedback from y'all, as well as suggestions for future videos (vulnerabilities, breaches, exploits, anything really). I'm doing a bit of travelling next week, so it might be a bit longer until my next upload. JOIN THE COMMUNITY ➤ discord.gg/WYqqp7DXbm Also, I think I finally fixed my intonations LOL Thank you for all of the support, I love all of you ♥ *EDIT* - At 2:23 those timestamps were meant to be 9:35am, my apologize for the mistake. I thought I fixed it, however I must have ended up uploading the wrong render. *EDIT #2* - 2:00 should have been "two and a half minutes", rather than seconds. Thanks to those who pointed this out!
@ImperialRoads
8 ай бұрын
I love pegging
@BillAnt
6 ай бұрын
Fascinating, love it. :)
@DanielBoctor
6 ай бұрын
@@BillAnt love you more
@renakunisaki
5 ай бұрын
I enjoyed it. Good pace and explanations.
@stockstreamtwitch
5 ай бұрын
Great work dude. Glad this video fell into my recommended.
@TheEnlightenedMalignancy
5 ай бұрын
Heavy engineer: “It costs thirty million dollars to run my code for twelve seconds”
@slomnim
5 ай бұрын
Put that on your resume xD
@TheNefastor
5 ай бұрын
Underrated comment ! 😂
@Fossil_Frank
5 ай бұрын
Well, on a good day it probably earns them more than that. It's part of the risk game, you accept having a robot making transactions for you and earning money at lightspeed, so you have to accept that sometimes it might lose money at lightspeed too.
@JohnSmith-ox3gy
5 ай бұрын
@@Fossil_Frank Just no. These are very low margin services. In 45 minutes they spent 4 years of profits of the entire thousand employee business. Your comment is off by a factor of over 1000.
@Fossil_Frank
5 ай бұрын
@@JohnSmith-ox3gy I don't know where you're from, but on my home turf any kind of stock transaction costs a hefty premium to perform. Granted, these kind of automated services probably discount it for the customers, but those customers are more than likely exclusively high rollers who bring in truck loads of money. I find it hard to belive they would charge them flat rates.
@sanderbos4243
8 ай бұрын
Imagine the stress of the engineers trying to identify the problem, knowing their company is losing 2.5 MILLION dollars per second
@DanielBoctor
8 ай бұрын
At least they ordered pizza LOL
@BillAnt
6 ай бұрын
@@DanielBoctor- Give new meaning to "When the sh*t hits the server fan". Ha-Ha
@supersat
5 ай бұрын
I'm a little surprised NYSE doesn't have a mechanism to block trades when something is obviously going wrong
@DanielBoctor
5 ай бұрын
They actually do, however they were not that helpful for Knight as they were designed for price swings, not trading volume. Mary Schapiro (the SEC chairman at the time) did end up reversing 6 of Knight's transactions, as they reached the cancellation thresholds outlined below: The SEC required more specific conditions governing the cancellation of trades. For events involving between five and 20 stocks, trades could be cancelled if they were at least 10 percent away from the “reference price,” the last sale before pricing was disrupted; for events involving more than 20 stocks, trades could be cancelled if they deviated more than 30 percent from the reference price. You can read more about this at Henrico Dolfing's report linked in my description.
@cambrown5777
5 ай бұрын
It's actually $292,000 per second, if the title is correct (440M / (28*60)). Still absurd.
@zdrux
8 ай бұрын
At first I thought, how could anyone be this stupid?.. Then I got to the point in video where they were given a month to design and deploy a whole new piece of software, and everything made sense.
@DanielBoctor
8 ай бұрын
yep, that will do it
@ghutkamukesh
8 ай бұрын
@@DanielBoctor😂😂😂
@paraax
5 ай бұрын
Yes, getting the software wrong is understandable. Not knowing how to turn your software off is however not understandable. The new routines worked within their existing framework. Deploying and decommissioning should be literally one of the first things they learned. Turn it off, and don't let it out of test mode again until you are sure it works.
@exponentialcomplexity3051
5 ай бұрын
@@paraax I am still confused. Why couldn't they shut it down? Just pull the plug in worst case no?
@BAmalakas
5 ай бұрын
@@exponentialcomplexity3051distributed system
@orterves
6 ай бұрын
Blaming the devs for losing the money when the company pushed for the release in a month, through procedures that involved manual unverified deployments, classic.
@spacemanmat
6 ай бұрын
You know that Devs complained and weren’t listened to. Would be surprised if the company had a history of doing this. This time it the company paid the price.
@tr7zw
5 ай бұрын
I mean, it was a failure on all fronts. 1 Month to implement this, no kill switch, broken deploy scripts, at that point 7-year-old dead and dangerous legacy code in the codebase, being able to "reuse" a flag that causes "dead" code to revive, no plans in case of emergencies... There's a lot at fault here.
@snorman1911
5 ай бұрын
Similar to my company. Each rush job builds on the tech debt of the previous rush job, with the whole system getting worse each time. Then management demands to know why everything doesn't work optimally. Every objection is met with "If we don't get this out by next week, we'll miss the market!"
@Gideonrex1
5 ай бұрын
I’m a dev ops engineer and my employer made me delete our dev environment bc he didn’t see how it was needed and was costing money. So I could see a company literally just having prod and the devs have no say.
@voidspirit111
5 ай бұрын
@@tr7zwthere was a kill switch. There allways is. They just wantes to.keep operationa going. The handling of thw situation ia more of a management and risk management failiure. But in America managament is rarely blamed. Based on the story they wanted to recover while being online so they don't lose face as a MM. They always had a hardware kill switch. A server kill switch would.have been a nicer option. I say it's a crisis management issue, because they were doing live debugging and troubleshooting while losing so much money and apperently nobody in the chain of command said " Stop take it offline". Beaides the fact that they had to.ve contacted from outside... like.. nobody was supervising that???
@KandyWrongIncognito
6 ай бұрын
The part where the engineers were engaged in live debugging on a production system made me cringe into the next dimension. That's like trying to perform open heart surgery on a marathon runner as they're running the race. What an absolute disaster. Great video.
@DanielBoctor
6 ай бұрын
well said. glad you liked it
@Entropy67
5 ай бұрын
it must have been torture, considering that there was no bug in their new code, it was a deployment issue 🤣
@gingeral253
5 ай бұрын
@@Entropy67The worst situation. Everything looks like it’s right and the problem turns out to be somewhere you never looked.
@thomquiri9860
5 ай бұрын
not gonna lie, that's a new fear unlocked for me as a future software engineer
@LesserAndrew
5 ай бұрын
I'm surprised they couldn't figure out how to kill the servers. Once, our CTO fixed a production issue by driving to the colo and unplugging a network connection.
@wieliewiel2630
5 ай бұрын
"I don't always test my code, but when I do.. it's in Production" - 😅
@RudolfJvVuuren
20 күн бұрын
This made me lol
@beyondfubar
16 күн бұрын
Crowdstrike? Is that you?
@BrumBrumBryn
9 күн бұрын
Everyone has a test server, it's just sometimes people are lucky enough to have a separate Production server as well.
@LordHonkInc
5 ай бұрын
If you're losing ~$150M a minute, there is a kill-switch. It's called the server room breaker
@ratulsaha9487
5 ай бұрын
Exactly. Don't know why they didn't shut down all operations until they figured it out. They wanted to continue business as usual and ended up losing the entire company. What a bunch of dim wits.
@eyeofthepyramid2596
5 ай бұрын
What does that do, never heard of it ?
@mortyrosenstein4211
5 ай бұрын
It’s the power circuit for the room. A light switch essentially. You just literally turn off the power for everything in the server room so the servers immediately stop melting the company down. The most basic and sure fire way to stop the problem. You just hit the big power button.
@astr0man573
5 ай бұрын
These days it would be in the cloud and no one would have the credentials to nuke the account 😂
@AccountInactive
5 ай бұрын
@@eyeofthepyramid2596Same as the breaker in your home. Turns off power to any given room or circuit (like laundry machines or stove)
@drakerp2
5 ай бұрын
"There is no kill switch," my man, unplug the server, throw a bucket of water or a cup of coffee on the buildings circuit breaker, litterally anything will cost a lot less than 2.3 million to fix.
@EntityVsEntityInteractions
5 ай бұрын
2.3 BILLION*
@baktru
5 ай бұрын
You're assuming you have physical access to the servers involved. In the stock exchange world, most likely in 2012 you don't. Those servers were likely co-located in NYSE's datacenter.
@Hunter_Bidens_Crackpipe_
5 ай бұрын
@@baktrulog in with a console and shut down the server or the app instance.
@CycloidalHeadache
5 ай бұрын
@@Hunter_Bidens_Crackpipe_yeah, hunter bidens crack pipe, I’m sure if you were there with your infinite wisdom, this never would’ve happened
@pepega4844
4 ай бұрын
@@baktru very likely. Im sure theres SEC rules that has regulations about where these systems can be held and likely has a secure facility access level requirement to run it. By the time they made it to the servers on site, they would have been bankrupt. I'm also sure it's probably not a server you can shut off remotely either even if you intentionally wanted to. They want those servers up 24/7 and isolated.
@thenayancat8802
6 ай бұрын
Seems like the "buy high sell low" code is not something you want on your production machines, but then I'm not a financial expert like these folks
@JacobSantosDev
5 ай бұрын
Well, they did have it behind a feature flag. But reusing a feature flag was a huge mistake.
@thenayancat8802
5 ай бұрын
@@JacobSantosDev Again seems like just simply not pushing the test code to production machines is the safer option, but I'm not a financial expert/CS wizard like they are
@JacobSantosDev
5 ай бұрын
@@thenayancat8802 oh sorry. The entire purpose of a feature flag is to be able to turn on and off features that you are testing in production. Just because you tested something in other environments does not mean the feature will work as expected in the production environment. The point of a feature flag is to facilitate the feature used in a live environment where you will want to turn it off. Technically, it is the "kill switch" and based on the limited information, turning that switch off would have saved them. Except it doesn't sound like anyone had training or didn't have access to the feature flag. Different teams are going to have different procedures for how feature toggles are switched. Better of it is a page where product can manage but might be entirely engineer owned. "Not deploy test code" is a non sequitur as depending on how you define it, all code is test code. The correct terminology would be "dead code" as the code should never run but because there existed a condition where it could, once it is revived, fuckery happens. You never want dead code to revive. I have never heard of good things happening when dead code suddenly runs.
@jajordan2106
5 ай бұрын
It depeneds borrowing a stock at a high price selling the stock at a high price then buying when the stock price falls allows you to make some money although the potential losses are infinite
@cheesesniper473
5 ай бұрын
The problem here was that there was no distinction made between user and development software. This PowerPeg development software should have never been on a live program server at any point in time. It belongs on a dev server or stored on a HD somewhere.
@ally6438
8 ай бұрын
As a coder who recently broke out with shingles due to the stress of being given far to short a deadline for something that ships to 9 million people I feel for these devs, that’s insane
@DanielBoctor
8 ай бұрын
I know, I find that most of these production nightmares are due to time constraints being pushed on the devs
@JorgetePanete
6 ай бұрын
too*
@everythingpony
5 ай бұрын
Isn't shingles deadly?
@TokyoXtreme
5 ай бұрын
@@JorgetePaneteYou found the misspelling, yet missed the comma splice.
@ally6438
5 ай бұрын
@@everythingpony I don't think so, but you're not really supposed to get it until 50+. I'm early 30s
@Lyokou
5 ай бұрын
As a software engineer as soon as I heard one month, I was like yep. Been there, done that.
@scottpaul7427
6 ай бұрын
Interesting. The way I've heard it, Power Peg was not "intentionally lose money for testing", but a production option to try and "peg" stocks to a given price (even if that would lose money). After being deprecated, it was judged too difficult to remove without effecting other production code. It was still being tested in builds until the 2005 changes caused those tests to fail, and they were removed. There was a script to automate deployment to the servers, but one was down for maintenance so the connection timed out and it was skipped without logging an error (or nobody bothered to read the log.) During the event it was obvious that something was wrong, but not obvious that it was causing huge losses until after the rollback accelerated things.
@Smaylik03
5 ай бұрын
And that's why investing into code quality and tests is important.
@Entropy67
5 ай бұрын
I just wrote about 120 tests over the past week or two for what I'm currently working on. Good to know that someone will just delete them when they fail in the future instead of actually figuring out why they are failing... eh, if the tests failed when the code is broke my job is done.
@restoreleader
5 ай бұрын
As a QA, i would like to comment on this: hehe
@fredmercury1314
5 ай бұрын
@@Smaylik03 Google, Facebook, Twitter, Microsoft, Apple, and Adobe have entered the chat... "Testing is what users are for."
@SuppressedOfficial
5 ай бұрын
@@Entropy67 Meh. A test that never fails tells you nothing. A test that always fails tells you nothing. The only good tests are basically coin tosses, but then why write code that only works sometimes? =D I fuckin' hate tests. lol
@shubashuba9209
5 ай бұрын
South Park: "Annnd it's gone."
@Nope_handlesaretrash
6 ай бұрын
Oh no wont someone PLEASE think of the poor high frequency traders. Lol.
@Xilladan093
5 ай бұрын
@natmarelnam4871 who , you?
@counterleo
4 ай бұрын
Any donation link?
@Bramble20322
4 ай бұрын
@natmarelnam4871 Stock markets are literally leeches on society and bring literally no value. Prove me wrong.
@dale3478
6 ай бұрын
Other rollback: "finally sh*t has stopped hitting the fan" SMARS rollback: "holy sh*t! There's now 8 times more sh*t hitting the fan!" But seriously, for a software that runs at a scale of thousands of requests per second and work with millions of dollars, there should definitely be some sort of kill switch or feature toggle built in from the start. Although the "rollback cause even more problem" is definitely a first for me
@MarcosAlexandre-no3qx
5 ай бұрын
probably someone saw every problem and said that they need more time to work on solutions and the company probably said. Just send anyway, it wont happen and we put in our next update, that never came.
@user0K
Ай бұрын
there should've been a metric, showing volume of orders from each server. but yeaah, probably timing issues first
@manankpatel2759
5 ай бұрын
As a developer i want to just say to all the companies and their executives, dont push too much for a little gain, give enough time as per the requirements or it might happen that the company may not even exist after launching that product.
@MKVideoful
5 ай бұрын
Doesn't matter, they will never listen.
@GamesFromSpace
5 ай бұрын
Also, don't try to pay the devs as little as possible. Because then this shit happens, and a good salary is a rounding error in comparison.
@enginerdy
5 ай бұрын
@@GamesFromSpaceI think they are probably well-paid, but there should have been 5x as many…
@txorimorea3869
Ай бұрын
@@enginerdy Or just one more guy: a deployment/build master responsible for making sure dead test code, or any other sus code or files don't make it into production builds, and production environments are properly deployed.
@whenmullet2674
17 күн бұрын
Executives aren't dev's, they will never think or act like a dev. Time is money, we can't waste the opportunity to make money here especially when its not ours.
@BigJMC
5 ай бұрын
Something similar nearly happened to my father. He works at an investment bank and was called in to solve an issue. Their network was being clogged by packets and it seemed a cascading effect has been caused between every server as they were stuck in a loop. 10 minutes before the stock market opened he went into the server room and just started pulling out ethernet cables and disconnecting servers and 5 minutes before stock market opened he pulled out the right ethernet cable and disconnected the server causing the issues. They could’ve lost billions of dollars that day.
@Xalgucennia
5 ай бұрын
Lol I was wondering why thsrs guys weren't doing the same, literally fixed in 5 minutes
@camiscooked
5 ай бұрын
@@Xalgucenniacloud computing exists... They may have been unlucky enough to have gone cloud computing only
@ggsap
5 ай бұрын
@@camiscookedDoes cloud computing not have an off button?
@camiscooked
5 ай бұрын
@@ggsap well yeah. No off button, someone else is running the server and you have to access remotely. Most companies will only have a few people who truly know the steps to stop function.
@ggsap
5 ай бұрын
@@camiscooked Its not that hard to stop the server. Its literally a giant red button in most cases, especially more easier an than diagnosing the issue
@MonsieurSansHonte
5 ай бұрын
Deploying code from dev to prod without a QA staging environment or subsequent smoke testing, is a recipe for disaster!
@Stettafire
20 күн бұрын
Yup. No DR env either
@beepbop6697
14 күн бұрын
I've always wondered: does the NYSE (and other exchanges) have dummy/test environments for these HFTs to test their algorithms against? If not: these guys are always "testing in production" -- hopefully with a very small account/budget limit in case the new code goes haywire. It is nuts that the NYSE allowed them to place orders that couldn't be filled -- no circuit breakers in any of it. Try to do any of this on Robinhood or other trading app, and that app will prevent you from making trades that your account can't cover.
@tomr6955
6 ай бұрын
So an average person makes a mistake and gets fired, but these guys lose half a billion and get bailed out.
@Jackson_Zheng
5 ай бұрын
they lost a company that took them 17 years to build from the ground up.
@SianaGearz
5 ай бұрын
@@Jackson_Zheng and nothing of value was lost?
@coren5911
5 ай бұрын
@@Jackson_Zhengwho cares, you could say the same for a small business owner
@AEVMU
5 ай бұрын
@@Jackson_Zheng They did already make many millions themselves though.
@bigsussyopsec
5 ай бұрын
They lost a bullshit company in a bullshit virtual economy that has no value with bullshit jobbers @@Jackson_Zheng
@ryansamarakoon8268
8 ай бұрын
Really appreciate the amount of detail here! Most other coverage was surface level but you went into a lot of great detail here
@DanielBoctor
8 ай бұрын
I'm glad you liked it! Thank you for the support I appreciate it
@wilkamania
5 ай бұрын
Gotta love the corporate Ops mindset, with directors that probably have more political experience than technical experience. It's like asking someone to cook a thanksgiving turkey in 30 minutes, and then blaming the cook for everyone's food poisoning. While the damage wasn't nearly as bad as this, I worked at a company that wanted to replace our entire system with a brand new one in 6 months. This was a legacy system integrated into EVERYTHING , and being a publicly traded company everything had an even longer process. My Sr. Manager and director kept using the stupid analogy of "the consultants are building us a ferrari and handing us the keys". I ended up leaving the company during this period, but heard that the Sr. Manager and Director started point fingers at each other when the project wasn't even 50% complete by 6 months. The Sr. Manager left before he could get fired, and the Director got fired. Neither of them really knew how the system worked.
@spacetime3
3 ай бұрын
That hit hard... "Hit the Kill switch (for the love of god !!) ...... There is no Kill switch....."
@ychentt
6 ай бұрын
Laughed so hard. Just subbed. Can't believe for this quality you only have 11k subs.
@DanielBoctor
6 ай бұрын
thank you for the support ❤
@loszhor
21 күн бұрын
This is why you review your builds BEFORE launching!
@ryanpalo
6 ай бұрын
This makes all of the times I screwed up prod feel so much better. Thanks for the indepth analysis on this.
@DanielBoctor
6 ай бұрын
thanks for watching!
@MangaGamified
4 ай бұрын
You can't screw like them.. **loses 100k per second **
@wwbcwp
5 ай бұрын
This reminds me of a time I accidentally uploaded an older version of a report I'd been working on in college, overwriting the new one and setting me back days.
@iliketacos6067
Ай бұрын
9:52 The engineers did a good job though. rarely are problems solved in that first 30 minute window.
@user-uy8yt7ku4w
5 ай бұрын
I still don't understand why they didn't just stopped all their servers and cancelled all the order's that weren't filled. That would probably take a couple minutes, instead of half an hour
@chunkyMunky329
5 ай бұрын
They could certainly have cut the power, which would have been fastest but I don't think it would have been possible to cancel the orders that have already gone through
@ME0WMERE
5 ай бұрын
@@chunkyMunky329 better to cut their losses than continue to lose $2.5 M per second
@chunkyMunky329
5 ай бұрын
@@ME0WMERE Thats what I'm saying. Except I was saying that they should cut the mains power to the building instead of manually switching off each server
@SianaGearz
5 ай бұрын
Do you know where the master breaker is for your whole building? Well apparently they didn't either.
@sandworm9528
5 ай бұрын
@@SianaGearz Yep I do, and if we were losing 2.5 million a second I'd probably hear someone yelling 'kill the power' and i would flip it
@TydiriumPilot
Ай бұрын
Crowdstrike: "HOLD MY BEER!"
@jh5131
20 күн бұрын
😅😅
@voidsp
5 ай бұрын
2:23 that's a bit of a fallacy IMO. There is a kill switch almost always. If affected services are on-prem - kill their Internet connection. Pull a plug on whole office/building if you have to. And if it's a datacenter, do basically the same - DC support can disconnect your servers/racks from the Internet.
@EatMyAstro
5 ай бұрын
Market-making (Knight's entire business) means being the middleman for every trade possible... with their infrastructure and resources, they were the #1 and were making a killing. IIRC Knight was responsible for nearly half of the volume across all exchanges in the stock market. If they seized operations here, they would lose everything. Their job is to remove friction between buyers and sellers by being a middleman, and as their reputation grew (along with their systems), it was paramount to always be online. It is not as easy as unplugging a box, and in many cases doing this would only make matters worse, logistics and business-wise. At the end of the day, yes there should've been a killswitch, and yes it should've been engaged. This was one of the first (if not the first) blowups in electronic markets that the industry had ever seen. And from the sounds of it, Knight was understaffed in their engineer/ops departments.
@MrAntiKnowledge
5 ай бұрын
I remember an old video where they switched a telephone network over and had to take the old one offline first. It involved 30 or so people with bolt cutters :D
@samramdebest
5 ай бұрын
I remember that video too Went looking for it a while back Couldn't find it anymore
@DanielBoctor
5 ай бұрын
Is this what you are referring to? This is fascinating kzitem.info/news/bejne/1JeIz6Zvbpt6iqAsi=uBbpgRjyGvrHR1_S
@samramdebest
5 ай бұрын
Yes it is And now I'm confused as to why I couldn't find it, apparently it was already in my likes? I must have been pretty tired when I went looking for it last time (or maybe it was set to private for a while 🤷♀️)
@ninjaasmoke
5 ай бұрын
God! Imagine rolling back to a stable software and losing even more money. That would have sent people nuts!!!! Losing close to 20m every second.
@darrennew8211
5 ай бұрын
They didn't roll back the flags, though. That was their mistake. The problem started when they flipped the flag, and then they left the flag and replaced the code.
@ninjaasmoke
5 ай бұрын
@@darrennew8211 thank you captain obvious
@alexander1989x
Ай бұрын
*uncomments the code *casually loses $440 Mil.
@mariuszmoraw3571
4 ай бұрын
If you don't have software kill switch, you always have hardware kill switch. Disconnect malfuctioning algorith from web and run tests until reason is found and fixed.
@josephduenas4718
5 ай бұрын
No kill switch or procedure to resolve this? Sounds like the devs got 2 days to brainstorm and the company said "yup! autobots roll out" 😂😂😂😂😂😂
@648
6 ай бұрын
Wow, no way this only has 3k views. Keep it up!
@DanielBoctor
6 ай бұрын
I'm glad you liked it! Thanks for the support 😊
@derekhettinger451
6 ай бұрын
Your content is excellent Was invested by the end of the video for sure Subbed, keep kickin ass man
@DanielBoctor
6 ай бұрын
thank you for the support! glad you enjoyed ❤️
@ronatola
5 ай бұрын
@@DanielBoctorYep. As a fellow IT Tech, I've seen a lot of these debacles during my career (even caused one myself once - shhhh) SUBBED
@feuermurmel
5 ай бұрын
Why didn't they stop their software, stopping all trading, when they noticed something was wrong? I mean, this is not Star Trek where a sentient program can refuse to be terminated.
@spyr0guy
5 ай бұрын
"Software will handle it!" Software:
@yashshende2786
5 ай бұрын
That's why chaos engineering and DR testing is important... They will surely build a kill switch now 😂
@Stettafire
20 күн бұрын
Proper version control...
@somebodythatiusedtoknoooooooow
3 ай бұрын
That's the issue with arbitrage bots, when they fail they lose years of profits in just minutes. Title should be "Manual Deployment ends up costing $440 Million. Maybe we need to hire some devops? "
@Stettafire
20 күн бұрын
Hire DevOps. QA. More than 1 dev. Good working practices. Estimation sessions from devs (no crunch time allowed outside of a P1). Proper pipelines and proper version control.
@-na-nomad6247
5 ай бұрын
Two lessons to get from this : 1- Software development is research, you CANNOT rush it, if you want to build faster, get more builders, don't pressure the ones you already have with tighter deadlines. 2- Being a good software engineer doesn't mean not making mistakes or knowing every function and library in existence, being a good software engineer means you clarify your code, document it, ask for reviews and testing and push back when management tries to give you unsustainable goals. it's 25% programming skill, 25% planning and 50% politics.
@AdityaWaghmare
5 ай бұрын
@@Bobrystoteles This 💯 percent
@aliciasueyee
5 ай бұрын
@@Bobrystotelesaccurate. The Mythical Man-Month is such a good read!
@guerra_dos_bichos
4 ай бұрын
2 women dont birth a baby in 4.5 months...
@QuicksilverSG
4 ай бұрын
@@Bobrystoteles You're assuming it's a unified development process like back when everything was in-house. Each layer of devs is hired to build new systems on top of the old systems. So it's more like a mother giving birth to a mother giving birth to a...
@Stettafire
20 күн бұрын
Yes but also. If you think you don't need QA, you do. Pair programming? You need QA. Dev testing? You need QA. QA is your friend.
@miss_adventure
5 ай бұрын
Production tip: the endings of these episodes feel so abrupt, it’s kinda jarring. I think it would be lovely to have more of an intentional outro - maybe summarize the topics discussed, or talk about some takeaways and how things might be improved in the future or something. Also a pause between the end of the script and the start of the “if you’ve made it this far” to give an indication that we’ve reached the end. Love how well you talk about these topics!
@sierragutenberg
6 ай бұрын
Bro, in this case the problem which caused all of this was literally naming. Looks like I'm not the only guy who struggles with naming things 💀
@LevelofClarity
8 ай бұрын
Great video. Would love to see some follow-up stories relating to HFT industry. Read Flash Boys years ago and absolutely loved it. I hope too see more of this from you in the future 😎
@DanielBoctor
8 ай бұрын
Thank you! It's definitely a area that I want to dive into. Thanks for sharing - I actually never heard of Flash Boys before ❤️
@LevelofClarity
8 ай бұрын
@@DanielBoctor Flash Boys is a great book. If you’ve never read anything from Michael Lewis that would be a great one to start with. The last couple of years I’ve mostly been listening to Audible. Hopefully you’re able to check it out.
@kylebroussard5952
5 ай бұрын
*If you have $440M and you can lose even 10% of it in a single day, you've done a terrible job. This right here is abominable*
@rightwingsafetysquad9872
5 ай бұрын
No kill switch? Like there was no circuit breaker to flip, no power cord or network cable to unplug? If I were the acting executive I would have walked into the computer room with cable cutters or an ax or something and just started chopping. I understand there could be large penalties for failing to complete market orders left pending, but it can’t be worse than $2.5 million per second.
Server probably is not even on the premises and they might not even have access to it.
@Ch17638
5 ай бұрын
Wait who hold on ..... Major deployment left to one person ? And then when trading started not a single engineer was monitoring trades just to check if everything was working as expected ? Then the CEO takes a break on launch date. We once deployed a new process service for company payroll, and on day one we had all leads and seniors monitor the system with several layers of safety introduced (limits to transaction amounts, limits to amount of transactions on first run) that data got checked to with an inch of its life, then the next set and the next with reports filled by the dev managers that had to be signed by the CIO before the remainder of the transactions could go through but even then as it ran at a staggered rate we had someone ready to pull the cord if anything seemed off. There were redundancies for redundancies as this system could empty 3 bank accounts in no time.
@mrgyani
5 ай бұрын
Yeah. Makes no sense. 😂 What a clown-show it was.
@Dr_Larken
Ай бұрын
2:26 I can only imagine how that phone call to customer support went!?? “Thank you for calling customer support, Knight in shining armor! How may help you?”? “OMG, We’re literally losing millions every second and can’t figure out WTF is going on…HELP!!”! “Um oh dear, Okay sir…um.. Have you unplugged your router, plugging it back in after 10 seconds?”
@msmith2961
4 ай бұрын
Doing rollback and making the problem 8 times worse made me lol so hard.
@susanoo2.042
21 күн бұрын
God, the worst part is thinking the new code was the problem and then reverting all the servers back to the old code, only to lose cash 8x faster. From 2.5 mil a second to 20 mil a second!
@Denni55
22 күн бұрын
"You think it's a bad idea to let a few big firms manipulate the entire market?" "Nah, it will go great."
@williamchamberlain2263
5 ай бұрын
Software development vs software engineering : the latter starts with system requirements that you have to be able to verify before production, the former can start with "you have one month..."
@weakend
5 ай бұрын
What a crazy story... so insane to me to run a company moving that much money and not have integration testing. On first glance you'd want to blame the engineers here, but the majority of the blame would have to be on engineering management/upper management to allow prod code on financial systems to be deployed sans integration testing. This story is a great anecdote as to why infrastructure as code/virtualization is so critical.
@AryanKumar-jo1pz
5 ай бұрын
The editing and effects are amazing. Reminds me of Lemmino really well done
@DanielBoctor
5 ай бұрын
wow, I never thought that my content itself would be compared to the legend himself. thank you for the support ❤️
@gswdeclan
21 күн бұрын
Losing over $100 million in a minute? Even Cathie Wood couldn't do that, truly impressive.
@debasishraychawdhuri
5 ай бұрын
why did not they turn off the computers? just pull the cables.
@danser_theplayer01
5 ай бұрын
Imagine getting Power Pegged for -440 000 000 dollars💀
@ronanoke
5 ай бұрын
Fantastic vid! A complex topic made simple, great job
@DanielBoctor
5 ай бұрын
Much appreciated!
@teenspirit1
5 ай бұрын
"There is no kill switch" doesn't make sense to me. sudo poweroff and you are done.
@renakunisaki
5 ай бұрын
There's a kill switch on every power strip!
@wulfbak
21 күн бұрын
This is like Skynet becoming self-aware. Engineers in a panic try to pull the plug, but are unable to.
@danielklimek6320
5 ай бұрын
How they couldn’t hit ctrl+c or kill the processes in the background remains a mystery
@redluck01
5 ай бұрын
There is no kill switch? No dual human control that checks the deployment? The CEO is out during a major brand new trading deployment? This is all lies from the CEO trying to save his job. Anything that goes right or wrong is the CEO's responsibility.
@zamoqi
6 ай бұрын
Enjoying your content big time. Appreciate the work that you put in!
@DanielBoctor
6 ай бұрын
thank you for the kind words - glad you like it ❤️
@JorkForkenson
Ай бұрын
Having worked in American Fintech for over a decade I can tell you exactly where this came from. There is zero room for nepotism in IT but I see it all the time. People who have no clue how anything technical works are placed in the highest ranked positions because they are buddies with someone or are great at blowing smoke up peoples rears.
@kucingoyen1
5 ай бұрын
Deployment without internal testing in all of the live servers? Lmao
@pierce9019
19 күн бұрын
Fuck a kill switch, I'd trigger the fire alarm in the server room
@Xezlec
14 күн бұрын
My jaw just crashed through the floor. You're telling me a company that writes code managing BILLIONS of dollars automatically not only doesn't use proof systems to prove the code is correct before deploying it, not only doesn't take their time to carefully vet any code changes through multiple levels of review, but actually pushes developers to rush out code and deploy it as fast as possible with no review, and even has no way to turn it off?! This level of incompetence is beyond what I would have thought humanly possible.
@Hebdomad7
5 ай бұрын
The problem was management all along. But they got golden parachutes as punishment ... The sooner these wallstreet types becomes personally liable for these kinds of screw ups the better.
@shea455
20 күн бұрын
They could have literally disconnected the network faster. It does not have to be automated. Even if in the cloud, a black-hole route is easy to create. As for process, this is classic: "Technology is a COST center" thinking. Cut budgets reduce time to deliver, and reduce talent in the technology pool as the best technology people have the easiest time replacing their employer.
@Orionbae
5 ай бұрын
Its pretty cool to actually see what quant firms do behind the scenes great video 🔥
@thrashwerk
5 ай бұрын
Good to know the financial sector works in the same crappy ways as the rest.
@gn0my
20 күн бұрын
As a software engineer, Im sometimes horrified at the practices other companies have. Its SO easy to keep the power peg algorithm, but not have it in production. Things like that is just astonishing to me.
@007gunlogo
20 күн бұрын
VERY interesting content! The only thing I'd suggest is to slow your narration speed down a bit, as the story kinda comes at the listener like a fire hose. I look forward to your next video...thanks!
@vistalover9607
5 ай бұрын
Sorry if I don’t get the details but why couldn’t they literally send a so command out little pull the plug. It was not days of the cloud yet so wouldn’t they be running their own servers via some sort of enterprise setup?
@DudeWatIsThis
5 ай бұрын
The day Knight Capital got power-pegged.
@ShayneReigns
4 ай бұрын
I said out loud just now that I WOULD like to watch your videos about cybersecurity. It was thrilling to know about this story and I’d love to hear more about the aftermath. Great video!
@daverei1211
5 ай бұрын
At some point they should have just put in a firewall rule to block connections with the trading server so no more trades could be made while they’d figure it out. Better to do this at $40m than $400m…..
@RobTheQuant
6 ай бұрын
No kill switch? WFT? That's the like the first thing you implement...
@alzeNL
4 күн бұрын
Very Good ! Enjoyed this and will watch more of your content.
@dustondoesit3913
27 күн бұрын
This video was very well produced and executed, great content. Easy sub.
@ForkCandle123
5 ай бұрын
It's the fault of the regulator allowing firms to buy shares that aren't actually available at that time. And it's the fault of the firm doing such a thing. Incompetence of the firm involved not to have taken proper precautions. Such profiteering shouldn't be allowed. As it was, they lost out. They'd not have complained if they'd accidentally made that sum rather than lost it.
@QuicksilverSG
4 ай бұрын
So you're saying the regulators shouldn't have allowed them to build a house of cards out of a house of cards?
@KertaDrake
5 ай бұрын
Should have just ran into the server room and started yanking cables when it came to that level of money loss...
@guerra_dos_bichos
4 ай бұрын
Its hard to do that when your servers are spread all over and owned by some cloud provider... companies on that level use high redundancies cloud providers
@JFirn86Q
19 күн бұрын
Ah yes, I always ensure my production code has a Buy High, Sell Low program somewhere in it. All the best ones do.
@miguelmartins3864
8 ай бұрын
Excellent video - very informative! I enjoyed the blend of finance and software. Given how intertwined they are these days, there's likely many more topics to explore!
@DanielBoctor
8 ай бұрын
THANK YOU MIGUEL! I completely agree as well ❤
@oldmanwilikers1252
5 ай бұрын
Great video, I can’t understand the quotes though the audio is too cooked through my speakers.
@mikethespike7579
5 ай бұрын
One of my former professors once told us that computers are just hard working idiots. They will readily wipe you and your assets off the face of the earth if just one line of code tells them to.
@richardgilson3512
5 ай бұрын
Old server number 8 is the hero we need ;)
@brucelee-hl8zn
20 күн бұрын
Funny thing is I am an IT engineer and just before this incident i asked them during a job interview for a VP position that I would 100% secure all the deployment of all their new projects. I just asked to be paid 1 million $ per minute and they all laughed to my face. Who is laughing now?
@hgbugalou
5 ай бұрын
No kill switch? Why can't you tell the dude at the datacenter to start pulling network cables? What am I missing here? Analysis paralysis?
@Palemagpie
20 сағат бұрын
To quote Benjamin. "How....how could you lose so much fukin money?"
@taralalram
5 ай бұрын
As someone who's worked in IT for 40 years from machine code programmer to head of engineering, this is definitely the CEO's fault. Testing takes time and yes men are too afraid to speak truth to power, instead ignoring all the advice from engineering. No one died here, it didn't work out so well for the people on the space shuttle.
@hari1408
5 ай бұрын
i don't feel stupid anymore after watching this
@JoeSmith-cy9wj
5 ай бұрын
In the end, the blame is on the CEO. Mistakes happen. The problem was greed, preventing proper procedure in a rush to grab more money. The entire stock exchange is there so a bucket of leeches can suck the life blood out of people who actually provide labor and goods, in favor of those too lazy, stupid, or entitled to work for a living. Betting on other's fortunes should be outlawed.
@benshiotsu8553
5 ай бұрын
real men test in production
@gFamWeb
6 ай бұрын
2:00 "two and a half seconds" did you mean minutes?
@DanielBoctor
5 ай бұрын
oops, good catch. should have been minutes. good catch! updated the pinned comment.
@TheJensss
Ай бұрын
Why did they bot cut the power to the servers, floor or the building or cut the fiber or something? This could have been stoped in minutes if someone has just taken the decision to take the measures needed to get the servers offline
@danielhobbyist
5 ай бұрын
Why did they not just turn the servers off? I do not understand.
@sebastiankeeton8839
5 ай бұрын
Goldman saw the real opportunity 😂
@ooplesoft
4 ай бұрын
Excellent video! What a great summary dude well done.
Пікірлер: 846