qBittorrent and disk performance

[Pentium100]

Hi, I am a long time uTorrent user, but decided to try qBittorrent because uTorrent was too slow. However, I have pretty much the same problem with qBT.

I have a very fast internet connection (up to 500mbps) but could not seed faster than ~20MB/s. However, I only had a single hard drive, and Windows Performance monitor showed that the drive was the bottleneck (~10% Idle time). So, I bought a newer server (with 2xXeon E5540) and created a RAID1E array out of 4 WD Red 1TB 2.5" drives (with 4 drives it is essentially RAID10). uTorrent managed to reach 30MB/s under ideal conditions (the array is empty except the single 22GB torrent). I decided to try qBT. It reaches 20MB/s. Windows again shows that the array is the bottleneck, but I noticed something strange - the disk queue length never goes above 1. And the request size is always ~64KB (one cluster, with uTorrent it was 128KB). So, qBT essentially tries to read one cluster, waits for the read to complete and then reads another. While this may not be so bad on a single HDD, it essentially removes the advantage of RAID - being able to read from multiple drives.

Is there a way of making qBT issue multiple read requests to the drive so that RAID works faster?

[ciaobaby]

If you want better performance, JUST GET RID OF Windows , THAT'S the biggest bottleneck not your drives.

Dual quad core Xeons, Linux Mint 16 or 17 with LXDE if you need a pointy-clicky UI  and qBT 3.1.9.2, ... No problem.

[Pentium100]

I would prefer to use Windows as Peerblock works there and also I want to use the server for other stuff in addition to serving torrents (dual quad core Xeon would be too much for just torrents). Also, I have not found good graphical remote control software for Linux - ssh is text-only and VNC for me is slow and unreliable (for Windows I use Radmin which works great, even over cell phone connection).

So, yea, while I am not completely opposed to Linux (I have a couple of Linux VMs and my router is Linux - all Debian), I want to try to keep torrents on Windows (though for some reason 2008 R2 does not want to work on this server, but 2012 R2 does, though I dislike 2012).

[sledgehammer_999]

peerblock on windows and pgl(previously moblock) on linux: http://moblock-deb.sourceforge.net/

As for your issue, I think I remember the libtorrent author saying that this is actually a limitation of the currecnt(0.16.x) and next(1.0.0) libtorrent. It said that the I/O thread is made to read in 64KB chunks.
However, he mentioned that next-next libtorrent version is configurable on this aspect. Now this version resides in the "libtorrent-aio" svn branch.
I'll see if I can provide a test build for you based on libtorrent-aio, but don't hold your breath.

Anyhow, this problem isn't controllable/fixable by qbt.

[fusk]

@pentium100
I know your feeling man. I my self have 800+ mbit and it just kills any harddrive almost instantly with too many i/o operations. I have 2 sshd's in raid0 and 15/20 mb/s is max.
However, that is on multifiled torrents, on singlefiled torrents 50 mb/s is not a problem.
Need a big ass raid to keep up on multifiled torrents, i'd probably have to add 1 or 2 more sshd's to get there, and i'm not really willing to do that. But 4x1TB/8GB would be nice.

[Pentium100]

@fusk
The problem is that the access is not optimized for fast connections. The key to high performance (out of any modern drive) is to issue multiple requests at once. The drive (any SATA drive with NCQ or SCSI/SAS drive with TCQ) can then rearrange them in the order that would be fastest. If you have striped RAID (basically anything other than JBOD or RAID1), then the controller will distribute the requests to multiple drives to serve at once.

Here's an example. Let's say I need blocks 1, 5000, 2, 40, 3, 41, 5001, 4 from the drive. The way most torrent software does this (uT, qbt...) is to issue a single request and wait for response (it's single threaded). So, the drive would read block 1, then get a request for block 5000, then for block 2 and so on. This is slow. If all requests came in at once, then the drive would rearrange them to be 1,2,3,4,40,41,5000,5001 and would read them much faster (because it would need to do 3 seeks instead of 7). This is for single drive. With my RAID1E, multiple drives would read different blocks at once and complete the operation even faster.

Same applies to SSD, it's just that SSDs have so low latency that they can be used in single request mode and provide decent performance. Increasing the queue would increease the IOPS though.

[fusk]

I understand, i'm just not technically strong enough to answer with anything else than, "What that guy just said"

[sledgehammer_999]

Small update: building libtorrent_aio currently fails. Waiting arvid to fix it.

PS: I think libtorrent_aio works more or less the way Pentium100 said 2 posts above.
PS2: when you max out your disks is one of your cores at 100% too?

[Pentium100]

When the disks are maxed out, the most CPU usage on one core is about 20%. The way qbt (and uT) works now is that it reduces the performance of RAID to that of a single drive (with a single drive, I can seed about as fast as with a 4 drive RAID1E or 5).

EDIT: I read about libtorrent-aio and yes, it looks like what I need. Hoping it is fixed soon so that qbt can use it (and in the mean time, I ordered a better RAID controller). Libtorrent-aio even can use an SSD as cache (which would be nice if it turns out that RAID is still too slow).

[fusk]

Maybe it'll even benefit me even tho i'm using a much smaller raid set up, but with a much faster connection.

[sledgehammer_999]

I don't know if you understand what svn(subversion) is, but even if libtorrent_aio is fixed qbittorrent will not use it for a long time. Why?
The current stable version is the 0.16.x series.
The next major stable version is the upcoming 1.0.x series (arvid has already released RC tarballs).
And after that, the next major stable version will be the libtorrent_aio branch (currently named 1.1.x).

If you look at the past, arvid releases major versions once every 2 years at least. eg 0.16.0 was released 2012-04-11 and 0.15.0 was released 2010-03-21. So I'd expect libtorrent_aio sometime in 2016.
However, once 1.0.0 gets released, libtorrent_aio will be moved to trunk and be more stable. This could mean that me/someone could do unofficial qbt builds based on it.

[ciaobaby]

Always nice to see that some BitTorrent users do understand drive controller queuing. I did try explaining it to the uT 'community' with respect to the perennial "disk overload" problems and how users could actually help themselves avoid it. Suffice to say that it mostly 'fell on stony ground'.

With SSDs it would be wise to avoid them as the download locations, because the wear levelling algorithms in the controllers does add to the queue latency on write cycles, and once the drive gets somewhat 'older' this can be quite substantial. Obviously 'read' cycles do not suffer from this so SSDs are far better suited for 'WORM' (Write Once, Read Many) use than they are for Write-Read-Overwrite storage. Using a 'journalling' filing system (NTFS, Ext3, Ext4 etc.) on an SSD also adds to the write latency.

All in all, SSDs, or any other "Flash" memory are  'better' for seeding jobs, and provided you keep the EMD (ElectroMechanical Drive) platters spinning the 'seek' time of a write cycle can match or even exceed that of an SSD. Keeping them spinning does not decrease the 'life-span', as it is the constant spin-up - spin-down that accelerates wear rates.

[Pentium100]

Write performance is not that big an issue for me as I do not download a lot, however, read performance is. I am getting a better RAID controller (the one I have now turns out to be host based and using a borrowed real controller I got slightly better performance with RAID5), but really do not want to buy a SSD - they are really expensive (if high capacity) and if I get a small one, then I would have to do "cache management manually", though I will try to see if my new RAID controller (Adaptec 5805Z) can use a SSD as cache.

[sledgehammer_999]

This is something a little bit older. I don't how this user does it but he can achieve 75MiB/s with a single HDD. Look at this issue report: https://code.google.com/p/libtorrent/is ... ail?id=552

[fusk]

Could that be because of single filed torrent?