Advanced configuration for server with many resources

Windows specific questions, problems.
Post Reply
Robertomcat

Advanced configuration for server with many resources

Post by Robertomcat »

Hello good afternoon!

I've been researching here in the forum and on github, but I have not been very clear about the configurations, because the experiences of other users are not the same as I have.

My query, apart from the advanced configuration, is about the excess of CPU consumption (In the file download), currently with the default configuration of qbittorrent (I have experienced this high CPU consumption in several versions).

Hardware:
Server= HPE ML110G9
Xeon E5-2603 v3 - 1600 MHz (6 cores, 6 threads)
RAM = 32768 MB DDR4
Card Raid Hardware = HPE P440 4GB RAM (you can combine the amount you want to read and write)
Integrated software raid = HPE b140i (Without cache memory)
Network card = HPE Ethernet 10Gb 2-port 561T Adapter (TCP / IP Offload)

Disk configuration:
The operating system is a Windows Server 2016-64 bits, no antivirus/third-party firewall (The router assigns a fixed IP address to the operating system), and is installed on the integrated software raid card b140i, on two SSDs Samsung 256 Pro in raid 1. There is also installed Qbittorrent 4.1.5 - 64 bits.
All files that download qbittorrent are stored on 8 x 6TB disks in raid 5, which manages the P440 hardware raid card, and the cache memory is entirely dedicated to writing. As additional data, as the amount of GB is large, the structure of the disks is 64k. I forgot, I currently have a 600 Mb symmetric fiber connection.

As for qbittorrent, it has 1400 active torrents, and sowing always oscillates between 20 and 50 torrents (There are almost never more than two leechers per torrent) and there is also the port that uses qbitt open in the router. I attached a few photos of the configuration:

Image

Image

Image

This is the current configuration of the server (if I forget anything else) in which qbittorrent uses the default configuration, and I would like to optimize it and adapt it to the hardware it currently has, and be able to reduce CPU usage.

Let's see if anyone can help me. Thank you!
Last edited by Robertomcat on Mon Jun 06, 2022 10:10 am, edited 2 times in total.
Switeck

Re: Advanced configuration for server with many resources

Post by Switeck »

Your qBitTorrent settings are probably ok.

The biggest limiting factor besides excessive CPU usage is the hard disk drives themselves. RAID 5 may make them faster to read (than if they were stand-alone) and the RAID controller's cache may make them somewhat faster writing...but there's still serious limitations.

File fragmentation on disk can be a problem even with crazy-fast storage:
https://qbforums.shiki.hu/index.php/top ... .html  and    https://qbforums.shiki.hu/index.php/top ... l#msg19159
Only around qBitTorrent v4.1.x started doing coalesced writes for big chunks of data at once (roughly 1 piece of a torrent).

Misaligned multi-file torrents don't help and it's extremely hard on SSDs in a striped RAID array: (RAID1 is mirrored, so should be safe)
https://qbforums.shiki.hu/index.php/top ... l#msg12725

https://en.wikipedia.org/wiki/Standard_ ... els#RAID_5
For RAID 5: "Since parity calculation is performed on the full stripe, small changes to the array experience write amplification:"

This is why I recommend downloading to a scratch/temp drive then auto-copying the torrent to its final storage drive/s -- it cuts down on file fragmentation immensely. A problem with that is large torrents take a LONG time to copy.
A decent workaround is fast+full file allocation (using a no-0-out method) which is possible on certain drive formats. (Probably not available under Linux though!)
Robertomcat

Re: Advanced configuration for server with many resources

Post by Robertomcat »

[quote="Switeck"]
Your qBitTorrent settings are probably ok.

The biggest limiting factor besides excessive CPU usage is the hard disk drives themselves. RAID 5 may make them faster to read (than if they were stand-alone) and the RAID controller's cache may make them somewhat faster writing...but there's still serious limitations.

File fragmentation on disk can be a problem even with crazy-fast storage:
https://qbforums.shiki.hu/index.php/top ... .html  and    https://qbforums.shiki.hu/index.php/top ... l#msg19159
Only around qBitTorrent v4.1.x started doing coalesced writes for big chunks of data at once (roughly 1 piece of a torrent).

Misaligned multi-file torrents don't help and it's extremely hard on SSDs in a striped RAID array: (RAID1 is mirrored, so should be safe)
https://qbforums.shiki.hu/index.php/top ... l#msg12725

https://en.wikipedia.org/wiki/Standard_ ... els#RAID_5
For RAID 5: "Since parity calculation is performed on the full stripe, small changes to the array experience write amplification:"

This is why I recommend downloading to a scratch/temp drive then auto-copying the torrent to its final storage drive/s -- it cuts down on file fragmentation immensely. A problem with that is large torrents take a LONG time to copy.
A decent workaround is fast+full file allocation (using a no-0-out method) which is possible on certain drive formats. (Probably not available under Linux though!)
[/quote]

Oh, some very interesting links! I will read them carefully.
As for my download system, I always download the files one at a time, and I always make the download be sequential (I think it can help reduce defragmentation)

Another topic that he tells me is the RAID system. This hardware controller uses only two disks to be able to parity all the other disks when there is new data to write, therefore it reduces the readings that can be made in general.

On the use of the download in a temporary folder, and then pass the file automatically to the destination folder, I had no idea that it could help reduce the fragmentation of the files, and it is something that I will seriously incorporate into the server. For example, a hard drive connected through a USB, and dedicate it to the temporary folder.

On the CPU consumption, there is no advanced configuration so that qbittorrent does not consume 30/35% (Many times the "System" process runs at the same time, and between the two they consume 60% of CPU) constant when Is the download of any file made?

Well right now, in the storage of the server there are about 25 TB, and they have not been downloaded through the use of the temporary folder. Windows Server tells me that RAID 5 has a fragmentation of 40%, but if I start to defragment, the files move them to the end of the disk and there comes a time when it can not fragment because there is no room for the new files that are waiting to defrag, also happens with another type of software to defragment.

P.D: I do not know if Windows server can falsify the fragmentation data of the raid 5, and 40% of fragmentation is not real.
Switeck

Re: Advanced configuration for server with many resources

Post by Switeck »

The RAID controller probably makes the group of HDDs seem like a single bigger HDD to Windows, file-system wise.
As a result, Windows may be unable to see how/where sectors/data blocks are stored on multiple HDDs.

So... I wouldn't be surprised if Windows incorrectly estimates fragmentation on RAID arrays...but I'd expect the real fragmentation to be even worse than the estimates.
Windows might have more access to real fragmentation levels if Windows itself was running a software RAID instead going through a RAID controller's hardware RAID.

(Trivia: SSDs mask their internal data layers so what's stored on each flash chip isn't directly reveled to Windows as well.)
Robertomcat

Re: Advanced configuration for server with many resources

Post by Robertomcat »

[quote="Switeck"]
The RAID controller probably makes the group of HDDs seem like a single bigger HDD to Windows, file-system wise.
As a result, Windows may be unable to see how/where sectors/data blocks are stored on multiple HDDs.

So... I wouldn't be surprised if Windows incorrectly estimates fragmentation on RAID arrays...but I'd expect the real fragmentation to be even worse than the estimates.
Windows might have more access to real fragmentation levels if Windows itself was running a software RAID instead going through a RAID controller's hardware RAID.

(Trivia: SSDs mask their internal data layers so what's stored on each flash chip isn't directly reveled to Windows as well.)
[/quote]

Any solution to be able to eliminate the fragmentation?

I have a backup in the cloud of all the data, if I delete all data from RAID 5 and I download them again with the system to use a temporary folder (first pass the data or a USB hard disk) and then copy them to RAID 5, could I avoid that fragmentation?

I have been reading the links you have indicated to me before, and there are people who use the automatic cache, and other people who disable the cache of the S.O. What is the configuration that I could use to have an efficient reading and writing job?

Thank you for your answers!
Switeck

Re: Advanced configuration for server with many resources

Post by Switeck »

You'd need to clear enough of the RAID HDDs so files can be written to them unfragmented. I don't know if that means totally formatting those drives or just deleting 20+ GB.

Redownloading all the files is almost pointless -- better to copy part of them to a single HDD (or some other reliable drive) then copy them back to the RAID, as that should remove the fragmentation.

qBT's cache size in your case (symmetric 300 mbit/sec internet) can be 1000 MB or larger if you're running the 64bit version of qBT. I need to test again to see if the cache behavior options can still cause slowdowns...especially "Send upload piece suggestions" (which did limit speeds in the past).
Coalesce reads & writes definitely needs to be check.

Enable OS cache causes some people's computers to use all available ram and doesn't on others.

Might need to at least double the size in KiB of the watermarks. Minimum needs a raise more than max, because if that empties speeds drop immensely. Full gigabit/second connections need that more than your connection!
Robertomcat

Re: Advanced configuration for server with many resources

Post by Robertomcat »

[quote="Switeck"]
You'd need to clear enough of the RAID HDDs so files can be written to them unfragmented. I don't know if that means totally formatting those drives or just deleting 20+ GB.

Redownloading all the files is almost pointless -- better to copy part of them to a single HDD (or some other reliable drive) then copy them back to the RAID, as that should remove the fragmentation.

qBT's cache size in your case (symmetric 300 mbit/sec internet) can be 1000 MB or larger if you're running the 64bit version of qBT. I need to test again to see if the cache behavior options can still cause slowdowns...especially "Send upload piece suggestions" (which did limit speeds in the past).
Coalesce reads & writes definitely needs to be check.

Enable OS cache causes some people's computers to use all available ram and doesn't on others.

Might need to at least double the size in KiB of the watermarks. Minimum needs a raise more than max, because if that empties speeds drop immensely. Full gigabit/second connections need that more than your connection!
[/quote]
It's not 20GB, it's 20TB  ;D.

In terms of bandwidth, they are 600 Mb/s symmetrical.

As for the cache memory, I have put it in automatic, and it doesn't use more than 1,7GB. And the cache memory of the operating system, disabled.
Switeck

Re: Advanced configuration for server with many resources

Post by Switeck »

I'm aware your total storage on the RAID is 20+ TB.
Deleting 20+ GB was in the hopes that would create enough of a continuous hole on the HDDs so files copied there won't be fragmented.

So you've got 0.6 gbit/sec symmetric instead of 0.3 gbit/sec symmeteric. Maybe 70-75 MiB/second in qBitTorrent.
Robertomcat

Re: Advanced configuration for server with many resources

Post by Robertomcat »

[quote="Switeck"]
I'm aware your total storage on the RAID is 20+ TB.
Deleting 20+ GB was in the hopes that would create enough of a continuous hole on the HDDs so files copied there won't be fragmented.

So you've got 0.6 gbit/sec symmetric instead of 0.3 gbit/sec symmeteric. Maybe 70-75 MiB/second in qBitTorrent.
[/quote]
I'm sorry, I didn't understand you about the 20GB before. The RAID currently has 11 TB free, and since yesterday I have the Windows defragmenter activated, this is going to take its time, since they are many terabytes.

Yes, qBT works between 65 and 70MiB when it consumes all the bandwidth.
Robertomcat

Re: Advanced configuration for server with many resources

Post by Robertomcat »

[quote="Switeck"]
Redownloading all the files is almost pointless -- better to copy part of them to a single HDD (or some other reliable drive) then copy them back to the RAID, as that should remove the fragmentation.
[/quote]

Hello again.

I put the downloads in a temporary folder inside a USB 3.0 hard drive, with an HDD. When the download starts, there is no problem until the end, but when you automatically pass the file from the temporary folder to the destination folder, the upload speeds of all torrents drops to zero. What's this about?

Then I've been testing the temporary folder in the same destination where all the other files are, and there is no problem, but then when you move the file from the temporary folder to the destination folder, you just move it, so it doesn't defragment.
Last edited by Robertomcat on Wed Jan 16, 2019 4:08 pm, edited 1 time in total.
Switeck

Re: Advanced configuration for server with many resources

Post by Switeck »

qBitTorrent's disk I/O is single-threaded, or at least libtorrent was till very recently.
That's part of the reason upload speeds drop to 0 when a torrent is being moved.
The other part is the HDDs are unable to handle the massed writes and reads when that happens.

A properly-designed cache would let uploads run at least briefly without needing constant disk access.
But it seems that's not working well either...
Robertomcat

Re: Advanced configuration for server with many resources

Post by Robertomcat »

Ok, thanks!!!!
Switeck

Re: Advanced configuration for server with many resources

Post by Switeck »

Search for posts by Fusk on the subject at this forum, he's probably the most experienced with qBitTorrent at-or-near gigabit/sec speeds.
Robertomcat

Re: Advanced configuration for server with many resources

Post by Robertomcat »

[quote="Switeck"]
Search for posts by Fusk on the subject at this forum, he's probably the most experienced with qBitTorrent at-or-near gigabit/sec speeds.
[/quote]
I found this link, which is a collection of links that you made this past year, to see if I can find the best configuration. Thank you!
https://qbforums.shiki.hu/index.php/top ... l#msg27951
Post Reply