Man your videos have such valuable information, really helps with getting the most bang for the buck, you rock!
@diacritic8508
Жыл бұрын
I wanted to add that even though the Proxmox installer will enforce selecting same size disks when creating a mirrored ZFS boot drive, you can still use disks with different sizes and have a mirrored ZFS boot drive. To do that you initially need to select zfs (RAID0) and select only one of your boot drives (the smaller one of the two) in the Proxmox installer ie. create stripe boot pool with a single drive. After completing your Proxmox installation you can create a partition on your second disk that matches the size of the first disk and also change the type of your boot zfs pool from stripe (RAID0) to mirror (RAID1) by adding the newly created and size matched partition on your second drive to the pool. This can be especially useful if you don't have any m.2 slots, because you use older hardware predating m.2 and you don't want to spend money on 2 small SSDs even if they are quite cheap, you can get used 150-300 GB hard drives for literally nothing (
@YouTubeGlobalAdminstrator
9 ай бұрын
You can disable cluster services if not used, they are VERY active with logging.
@chromerims
Жыл бұрын
6:27 --- 10TB writes per year (or 30GB /day) in a PVE cluster . . . insightful information. Jolly superb video 👍 Your mannerisms evoke the "ExplainingComputers" YT channel (perhaps the non-British version). I enjoy your content covering production deployments with hyper-scaled / converged / private cloud / composable (whatever ppl deign to label it next). P.s. Whereas RAID discussions are as old as time immemorial, I understand that recovering a mirror in RAID-1 is not trivial. Kind regards, neighbours and friends and happy turkeys 2022.
@andre-le-bone-aparte
Жыл бұрын
Just found your channel. Excellent content and a solid review! - Another sub for you sir.
@utweak
Жыл бұрын
Thank you for putting this together. I spent hours going down the ssd wear rabbit hole and figuring out what to put on which drive - could have saved that time with yr video. yr video helped clarify a ton
@gordslater
Жыл бұрын
when BTRFS comes out of beta in late 2049 it should make a great boot drive :)
@theundertaker5963
Жыл бұрын
Quality videos as always! I will be referencing this video to make people realize just how write intensive Proxmox really is. You have no idea how many production servers I have seen around running on a single SanDisk flash drive as the boot drive! Personally I try to go with two SSDs in Raid 1 where possible and as a hard prerequisite in the production environment, regardless of their operation size. That being said, since Raid is not a form of backup, I was wondering if there is a way to make actual backups of your boot drive in Proxmox or not?
@chefvoyardee7681
Жыл бұрын
Great content. You're killing it dude!
@JathanLane
Ай бұрын
Your a Gaddamn Legend. Thank you sir!
@curmudgeoniii9762
Жыл бұрын
All you vids are just great ... in depth which few do... thank you very much.....
@haroldwren1660
3 ай бұрын
Thank you for the very thorough video.
@franciscolastra
Жыл бұрын
Great video. Many thanks! The fact that you share you measurements on written data is really helpful. Some questions that are now bouncing in my head.... If you just use PVE as stand alone (no cluster), does the amount of written data per day come down? How much of it is logging? And... how can we move log files to other drives/server?
@esotericjahanism5251
5 ай бұрын
super informative video! I just picked up 2 Intel optane p1600x 58gb drives for $30 each. Those little bastards are perfect for server boot drivers, they have amazing latency and IOPS and their write endurance is off the charts! Got a few more of the 118gb p1600x to help accelerate my ZFS workflow
@sorin.n
Жыл бұрын
Can the logging be sent to a log server in order not to stress the boot drive? Or disabled for good?
@Liebemasheen
2 жыл бұрын
Over provisioning helps with wear endurance on boot drives. Personally, I prefer TLC data centre SSD's with high writes endurance and MTBF
@mangoes142
Жыл бұрын
Hi. Can you do a video on how you make ssd to run only the boot drive? And do you store VMs in it or in a different drive altogether?
@c0p0n
Жыл бұрын
Great, concise video. Very useful for my upcoming home build, thank you
@bencampbell5476
Жыл бұрын
The Dell R630s/R640s internal SD cards work great for this as well.
@janw.443
Жыл бұрын
This a great video. Learned a lot. Thank you!
@dpilcher
4 ай бұрын
Good info thanks for putting this together
@dominick253
10 ай бұрын
That's one thing that really messed me up starting with proxmox. Why does it split the boot drive!?! At least let me pick how it's split.
@danbrown586
Жыл бұрын
All four servers in my cluster boot from pairs of SATA SSDs in a ZFS mirror (what Proxmox calls RAID 1), with data via Ceph and NFS. A fifth standalone server runs a single NVMe SSD for boot and data.
@curmudgeoniii9762
Жыл бұрын
More great stuff ... really!! only one comment .. but maybe will not work due to U-tube.. put all PM in one video ... thanks for all your very detailed and good videos .. I win lottery ... you win.. ;)
@ElectronicsWizardry
Жыл бұрын
Would you want one super long video with all my Proxmox content? I could edit the videos together to create a long video like that if there was demand for it. I also have a playlist on my KZitem channel that has all my Proxmox videos in one place and the videos will autoplay like how one long video would.
@philipkeeler9997
26 күн бұрын
@@ElectronicsWizardry That you even considered this request shows the content and quality of your character. I like gentleman scholars!
@pavelzadorski
Жыл бұрын
Awesome videos, thank you! Is it feasible to boot off PXE for a PVE host, especially for a small PVE cluster? I'm intrigued by the idea of redirecting logging to a log PC and having configs stored on a PXE server with storage redundancy. This sounds like an interesting approach to explore! Just subscribed, by the way :)
@ElectronicsWizardry
Жыл бұрын
I haven't looked into doing PXE booting into Proxmox. I'll take a look at it and see if possible and practical, and may make a video about it. Thanks for the idea.
@mjohn50n
7 ай бұрын
Are there any speed-related issues when using flash drives as boot drives for Proxmox? I have a few mini PCs with 1TB NVMe drives but no options for disk expansion
@sonictech1000
Жыл бұрын
When booting a Linux box off a USB flash drive I've found that using the noatime flag in fstab makes a massive difference in longevity.
@ElectronicsWizardry
Жыл бұрын
I should try this after changing to noatime but it does seem to significantly lower writes and improves performance. I’m guessing a good amount of those writes were for logging access time.
@OpposableGums
7 ай бұрын
Three issues for your next Boot video: o How to set up booting with Root on RAIDz. o How to install ZFS Boot Menu (Nobody has a video on ZBM.) o How to recover if the wrong drive fails when root is on ZFS.
@AlL-fw2cy
Жыл бұрын
You recommend a 128GB SSD boot drive as well as a raid mirror. ZFS has got to tax those drives pretty hard. I'll probably just use a single SSD in ext4.
@ChrisHolzer
Жыл бұрын
I had quite a few SSDs suddenly 'die' over the years. RAID1 of 2 120GB SSDs is a smart move - you will sleep better.
@nalle475
2 жыл бұрын
I have pcie cards holding ssd’s. Used to run TrueNAS from a USB stick (usable for up to 2 years before they needed to be reformatted) but switched to pcie card with ssd.
@recalion
Жыл бұрын
For my future Home-Server I plan thw Solidigm SSD D3-S4510 240GB for boot drive. It is a SSD with 2M MTBF + 900TB TBW - and quite cheap for that. But a version with fewer GByte is not available.
@cbrady3_98
Жыл бұрын
Great video. I had a USB drive fail and I'm still trying to recover my VM information. I'm getting ready to do ZFS mirror Drive like your suggesting for a boot disk. You had mentioned that you can set up another disk for proxmox to do it s temporary writes to but I haven't seen where to set that up.
@ElectronicsWizardry
Жыл бұрын
You can turn a single boot drive into a mirror if your using ZFS on the boot drive and the new drive is the same size or bigger than the existing drive. There is no way of doing this in the gui. I plan on making a video on this in the future but the process is to copy the partition table, randomize the guid of the new drive, add the mirror in zfs, then set up the boot loader.
@igordasunddas3377
7 ай бұрын
A year later and I still face a similar question: should I have a boot drive that is an MD array (RAID 1, NVMe)? Or a simple boot drive but RAID 1 for the data drive? Or RAID 1 on ZFS? Also I have 2*1TB Samsung 980Pro, so perhaps I could use those for boot drive and data? Or is that discouraged? File storage is much simpler IMO. I'll probably run whatever I currently have (RAID5 with Ext4 on it).
@JonathanYankovich
Жыл бұрын
One suggestion I heard is to use a usb flash drive and turn off swap. Not sure if that solves issues with temp and logs thrashing the drive.
@ElectronicsWizardry
Жыл бұрын
I was using ZFS, and ZFS when setup as a boot drive in Proxmox doesn't have any swap configured by default. I'm still looking more into what may explain where all the writes are coming from.
@mbunds
2 жыл бұрын
Do you have any advice on naming conventions for single and clustered volumes?
@hpsfresh
2 жыл бұрын
Thank you for explanation. Can you check on any of your servers how does zfs mirror looks like? Main question: does both drives have boot record and does system can boot from any of two drives in case of failure? Practical test will be very appreciated :) thanks
@ElectronicsWizardry
2 жыл бұрын
The boot record is stored on boot drives with a ZFS mirrored setup. Grub and other boot loader files are stored on both drives, so both drives can be booted from. When replacing a drive, there is a guide on the Proxmox website to follow, as just replacing the drive in ZFS won't let the second drive be bootable.
@chriss377
9 ай бұрын
My servers won't boot to NVME but a few years ago I installed ubuntu onto a tiny ssd for /boot and everything else on the NVME. Wondering could I do the same with Proxmox and utilize three 118gb Optane NVMEs for the OS drives in my cluster and keep the SATAs just as the boot partition. In this configuration I would think my 1tb 970 Evo Data SSDs as boot would outlast the hardware.
@njsilverandblack9178
Жыл бұрын
Just found your channel and subscribed immediately. I am running Proxmox 7.3 in home-lab environment on a pc. How do I install PVE 7.3 on a 58GB Intel Optane SSD and use that drive for OS only?
@ElectronicsWizardry
Жыл бұрын
To install Proxmox on the Optane SSD, boot from the Proxmox installer ISO, then select the Optane SSD as the drive you will install on. If its easy I sometimes remove all the other drives so I can't accidentally install on a drive I didn't want to. Then once the OS is installed, add your other storage options, and don't store anything else on the local drives, and the local drive will be used for boot only.
@SuperLugol
Жыл бұрын
How about booting from a SSD connected with a sata to USB2 ? No endurance issues and still keep the max number of drives.
@ElectronicsWizardry
Жыл бұрын
Yea a usb 2 to sata to ssd would work well. Usb drives are generally pretty reliable these days and the performance should be plenty for a boot drive. My usb 2 drives in using as a boot drive for me server now is plenty of speed and since I have 2 in raid 1 I can easily replace them as needed.
@vincentgallo3351
Жыл бұрын
Which USB drives have high endurance for running mirrored boot ?
@ElectronicsWizardry
Жыл бұрын
I haven’t done a test of usb drives to have any hard data, but most of the usb sticks I have used are of fairly low quality. I’d generally recommend a sb ssd as they should be much more reliable and faster than most usb sticks.
@vincentgallo3351
Жыл бұрын
@@ElectronicsWizardry Thanks for getting back to me. Limited to 1 m2 and 1 sata drive in a small pc so I was thinking about usb sticks or a high endurance sd cards in usb adapter since mirror is impossible otherwise. What do you recommend ?
@h.b.5577
Жыл бұрын
@@vincentgallo3351 Can't say about ElectronicsWizardry's recommendations, but personally would recommend a Kingston KC3000, a bit overkill capacity wise, but they have great tbw and can be had for around 50 usd (at least locally for me). Have a Corsair Force M510 as boot drive in my server currently, but their a bit older and harder to find at this point and endurance wise (at least at lowest capacity) are a bit worse than the Kingston. Combine it with cheap m.2 enclosure and it should last basically the whole life of the server (quite possibly even outlive the server) in this use case.
@vincentgallo3351
Жыл бұрын
@@h.b.5577 THX!
@myto1326
Жыл бұрын
Great video, very useful for my installation ! Is there a difference between m.2 nvme at 2500mb/s (or more) in writing and ssd sata 3 up to 500mb/s max for boot proxmox ?
@ElectronicsWizardry
Жыл бұрын
For boot drive use a SATA SSD vs a NVMe SSD won't make any different performance wise. The extra performance won't matter for boot uses.
@ckirkyg
Жыл бұрын
The 30gb of writes per day is really the reason to avoid zfs.
@LubomirGeorgiev
Жыл бұрын
Dell BOSS Wohooo!
@mysteriumxarxes3990
Жыл бұрын
I dont know what a promox drive is, it just appeared in my recommended. Well thanks but I cant find any usefulness in it for me
@pg_usa
Жыл бұрын
Btrfs also has raid1 option...
@johnwashifi
2 жыл бұрын
Hello, could you make a tutorial on how to spin-down hdd in Proxmox? thanks in advance!
@ElectronicsWizardry
2 жыл бұрын
Thanks for the idea. Adding this to list of future videos to work on.
@jothain
8 ай бұрын
I'm embarassed to even ask. I checked out your script, but I'm bit confused as it's .sh but apparently runs on python? Hmm. How do you launch it?
@ElectronicsWizardry
8 ай бұрын
I'm far from the best programmer di there is sgood chance I didn't do that fully right. But there is a shebang at the top that tells the system to runcthr script on python3. You should be able to download the script and then make it executable with chmod +x diskWrites.sh, and the run the script with ./diskWrites.sh. Hope this helps.
@jothain
8 ай бұрын
@@ElectronicsWizardry Thanks, I forgot completely to make it executable. I don't understand why, but I get "./diskWrites.sh: line 1: payload:allShortcutsEnabled:false: command not found". I checked the script, as always even though I'm not programmer at all and don't understand why that env line doesn't work. Tried running it in Proxmox out of curiosity
@jothain
8 ай бұрын
@@ElectronicsWizardry Oh never mind. Me stupid ass downloaded the html format from github instead of raw. Reminder to myself "always check the local file". Man I feel stupid 🤦🏻 It works just fine..
@markstanchin1692
5 ай бұрын
Smicro sdom I wish there was 2 sockets so I could mirror
@neins
10 ай бұрын
I have 2 consumer SSDs 120gb and 256gb. Is it possible to make over provision onto 256gb drive and make its capacity 128gb to make more endurance. Is it overkill?
@ElectronicsWizardry
10 ай бұрын
Over-provisioning can only be done within a drive as it relys on knowledge of how much each physical block is used, and that isn't shared over the sata/nvme data. You can do something like a stripe, raid0, or jbod that would split the writes up, but I'd think its best to just keep the second drive as a replacement in case the primary one fails.
@neins
10 ай бұрын
@@ElectronicsWizardry my idea was that I would have z-raid1 with two same size drives and primarily point of view was the possibility of using leftover space for increasing TBW (on controller lever of the drive) I just have these two drives on hand and wanna make raid1 with them and don't wanna to trim and lose space on second drive
@ElectronicsWizardry
10 ай бұрын
@@neins Raidz1 is a single parity solution, and typically used for 3 or more drives. I think you want a mirror here so both drives have the same data, and if one drives dies you don't lose any data. You would get 128GB of space with this configuration.
@neins
10 ай бұрын
@@ElectronicsWizardry so on 256gb drive "redundant" space will be unused in my case?
@ElectronicsWizardry
10 ай бұрын
@@neins With a mirror setup there would be 2 128GB mirrored chunks, one on the 128GB SSD, and the other 128GB would be 1/2 of the second SSD. The empty space on the larger SSD would be used for overprovisioning. But with SSDs these days I wouldn't worry too much about having extra overprovisioning space and endurance isn't a big issue with most SSDs and use cases.
@chris-kr7gf
29 күн бұрын
Agree on the brand. Cheap chinese brands just fail constantly. OCZ, Teamgroup, pretty much garbage
Пікірлер: 79