qBittorrent and disk performance

[ciaobaby]

BitTorrent protocol clients don't know or care about 'files', whether there is one file or one hundred files in the payload,  it still just a collection of equally sized blocks of binary data. It is the operating system DFS that deals with 'files'

[fusk]

I was thinking that with a single filed torrent i can get 60+ mb/s and i've uploaded with 100+ mb/s.
With a multi filed torrent it doesn't go higher than around 20/22 mb/s, which is caused by raid0 or single drive overload (think it's too many i/o's). So my thought was that if the raid wasn't utilized properly like OP talks about, then a single filed torrent might be why sledgehammers example is getting 75 mb/s to a single hdd.

I might be completely wrong tho.

[Pentium100]

It could be that either the torrent is small or the guy has an insane amount of RAM. I am sure if my server had 32GB of RAM, the 22GB torrent would seed at max speed. Alternatively, it could be that downloading (writing to disk) is faster and more optimized than reading (which would make sense as the writes would be a couple of megabytes (entire piece) at a time instead of 64KB).

For example, checking the torrent goes at around 95MB/s, the avg queue is 2 and avg read size is 128KB (not much, but better than 64KB). This is for single drive, so, not bad.

Now this is weird: I redownloaded the same torrent (to test download speed, which was ~20MB/s with avg write requests being in the MB range) and now it uploads at 48MB/s. Either the cache efficiency now is really high or something else is going on. Previously I was seeding the exact same torrent, just that I copied the downloaded files from another server onto a newly formatted drive. Now I created a new folder and downloaded a copy of the files there and it seeds faster. Just weird. Still, small requests and synchronous IO with multiple torrents would mean slow. Now I'm going to try the exact same thing with uT.

[ciaobaby]

It could be that either the torrent is small or the guy has an insane amount of RAM. I am sure if my server had 32GB of RAM, the 22GB torrent would seed at max speed.

Umm? No!

The payload byte size or file structure has absolutely nothing at all to do with upload speeds (or download for that matter).
Assuming that your client has no limiting in place;


Upload speed is limited by how many peers are connected and how fast they request and download pieces.

Download speed is limited by how many connected seeds & peers are actively sending pieces to your client.

[fusk]

I have 16gb ram, x64 version of qbit doesn't use more than 1.9gb if cache is set to auto, & auto has been recommended by both these guys. So, i'm not sure 32gb would do much difference.

p.s. Am i the only one getting 403 error almost every time posting ?

[Pentium100]

[quote="ciaobaby"]
Upload speed is limited by how many peers are connected and how fast they request and download pieces.[/quote]
What if the files are on a slow hard drive that cannot read faster than 2MB/s? Or (hypothetically) a tape, that could read really fast, but had really long seek times? Then the drive would be the bottleneck. If the server had enough RAM to fit the entire torrent into RAM, then upload speed would be limited by the network and the speed of downloaders. Windows also cache file access, so even if qbt does not use a big cache, windows caches the rest.

I managed to repeat my previous 50MB/s with uT - yea, it seems like the cache sometimes becomes very efficient - uploads at 50MB/s, while disk reads at 20MB/s. Windows has about 12GB of the 22GB torrent in RAM. If I had a smaller torrent (or more RAM) then it would fit entirely into RAM and there would be almost no disk activity.

I guess I could replicate that either using SSD for active torrents or (looks like my new RAID controller supports it) using SSD as cache.

[ciaobaby]

Or (hypothetically) a tape, that could read really fast

What are you thinking of using? An ICL 9200?

2MB/s?? you are kidding, right? I don't think any one with half a clue would be using floppy disks or class 2 'Flash' memory and running torrent payloads off it.

So mean-while back in the real world.

Windows has about 12GB of the 22GB torrent in RAM

Nope, can I suggest that you read up on the BitTorrent specification, because it doesn't work like that and neither does Windows caching.

The reason you may see a difference between single file payloads and multiple file payloads is that Windows reserves cache space for every FILENAME (in the payload) that is 'in use' and bitTorrent does NOT cache the entire payload it caches the 16kiB BLOCKS that are currently active for the pieces during data transfer

[Pentium100]

2MB/s?? you are kidding, right?

When I make examples, I usually tend to exaggerate. I have a hard drive (1GB) that reads at about 2MB/s, so I used this as an example that the disk can be a bottleneck. So is a disk that can do random reads of 64KB (queue length=1) at 20MB/s.

Windows has about 12GB of the 22GB torrent in RAM

The reason you may see a difference between single file payloads and multiple file payloads is that Windows reserves cache space for every FILENAME (in the payload) that is 'in use' and bitTorrent does NOT cache the entire payload it caches the 16kiB BLOCKS that are currently active for the pieces during data transfer[/quote]
Still, Windows does cache disk reads, as long as there is available memory. The amount of "free memory" in my server right now is 80MB and all programs take up about 3GB, the rest is the cache for the torrent. How do I know it? Well, for one, RAMMap says that the three files in the torrent are memory mapped (in case of uT 2.2.1) and take up ~12GB. And also if I close uT, the amount of free memory goes right back up to 12GB.

Also, the torrent is being seeded at 50MB/s, while the disk reads (Windows Performance monitor says so) at ~20MB/s.

I do not claim to know how uT or qbt uses their allocated 1GB cache.

[newborn]

@anyone who can answer

have been using Qbt for some time and love it...despite a few bugs...
BUT ....something bothers me: is it POSSIBLE to choose the download directory according to each category
for ex.
category films --- D:
category software ----E:
category manga------ C:
etc...

if not possible....could it be an interesting feature for future updates of Qbt

[sledgehammer_999]

currently no. there is a feature request on the bug tracker but I don't have an ETA on that.

[hyperion1is]

@newborn
I don't know about your different partition preference but, there an option in qBt "Append label to the save path" and your downloads will be saved in such a folder. All you have to do is to make labels like the categories you presented above: films, software, manga etc. I use this feature and it works (generally).
Only that I use the same partition for my saves. I have the path: E:\Download and it follows with \Film; \Software...
This is what I use, I don't know if it helps you until the above feature is included.

[newborn]

thanx hyperion: i did see that option...but is is a little of a hassle; as it creates FOLDERS in my old folders....and messes my structures ; and like you said it does work but seems to, only if the Whole drive is the same
i use my pc to stream To My multimedia device...so not ideal; but thanx

@sledg:  i think you know what i mean, if this option is coming, i am in no rush...would be great to choose paths directly in options menu

[Pentium100]

I managed to solve my original problem by using hardware to compensate for inefficient software

I bought a 120GB SSD and configured it as cache for the array. At first uploads are slow, but eventually all most accessed blocks get copied to SSD and i can saturate my connection.

[fusk]

I'm using sshd's as that was a cheaper way of doing exactly that. Each 1tb drive has 8gb ssd cache for a total of 16gb. Which shows in upload speed as my upload is significantly faster than my download speed.
But i have no control over it, as it's controlled by the drives firmware that has it's own algorithm that decides which files go into the cache.

[Pentium100]

[quote="fusk"]
I'm using sshd's as that was a cheaper way of doing exactly that. Each 1tb drive has 8gb ssd cache for a total of 16gb. Which shows in upload speed as my upload is significantly faster than my download speed.
But i have no control over it, as it's controlled by the drives firmware that has it's own algorithm that decides which files go into the cache.
[/quote]

Similar for me - as the cache software (VeloSSD or PrimoCache - still haven't decided which one to use) decides what goes to the cache. That's why I bough a larger SSD - it should be enough to hold all my active torrents and have some space left.

Now I am seeding at 56MB/s and my hard drives are idle...