Hello everyone,
I've been trying out qBittorrent lately, but noticed an abnormally after short usage.
Basically, the All-time upload in "View-Statistics" shows 6.08 GiB.
Whereas the sum of uploaded data of all torrents is only around 3 GiB.
See the attached screenshot for detail.
(Notice that All-time download is about 3 GiB larger than per-task sum as well.)
I can confirm that I have only ever added the four tasks as shown in the screenshot. Nothing is filtered out of view.
The only thing I can think of is protocol overhead. But I cannot imagine it being this large.
What could be the culprit? Thanks!
qBittorrent 4.1.5
Windows 7 32-bit
All-time upload does not match per-torrent sum by a huge margin
-
SL9
All-time upload does not match per-torrent sum by a huge margin
- Attachments
-
- 2019-06-22_230619.png (59.15 KiB) Viewed 3367 times
-
- qbt_stat.png (12.84 KiB) Viewed 3367 times
Last edited by SL9 on Sat Jun 22, 2019 3:08 pm, edited 1 time in total.
-
Switeck
Re: All-time upload does not match per-torrent sum by a huge margin
Protocol overhead really can get silly if you're on torrents with 50+ connected peers/seeds at once.
Peers/seeds regularly send HAVE messages to each other for every piece of the torrent they have, as well as PEX messages that often repeat the same known list of ip:port values of other seeds/peers.
uTP has about 5-20% packet loss on top of that, likely due to bugs in libtorrent's utp code.
Peers disconnecting and reconnecting with encrypted handshakes add a little overhead as well.
While overheads are unlikely to double the base amount uploaded to peers, it's possible for overheads to do so at worst-case scenario.
Worst-case scenario may also be due to overloads on download or upload bandwidth sides. Packet loss is guaranteed if qBT is trying to upload at unlimited speed to lots of peers at once on anything other than ridiculous-fast (1+ gbit/sec UL) fiber lines.
Peers/seeds regularly send HAVE messages to each other for every piece of the torrent they have, as well as PEX messages that often repeat the same known list of ip:port values of other seeds/peers.
uTP has about 5-20% packet loss on top of that, likely due to bugs in libtorrent's utp code.
Peers disconnecting and reconnecting with encrypted handshakes add a little overhead as well.
While overheads are unlikely to double the base amount uploaded to peers, it's possible for overheads to do so at worst-case scenario.
Worst-case scenario may also be due to overloads on download or upload bandwidth sides. Packet loss is guaranteed if qBT is trying to upload at unlimited speed to lots of peers at once on anything other than ridiculous-fast (1+ gbit/sec UL) fiber lines.
-
SL9
Re: All-time upload does not match per-torrent sum by a huge margin
[quote="Switeck"]
Protocol overhead really can get silly if ..... While overheads are unlikely to double the base amount uploaded to peers, it's possible for overheads to do so at worst-case scenario.[/quote]
Thank you very much for the detailed response. Now I have the big picture in mind. I'll continue using this particular copy of QBT to see if I can find some pattern. Would this delta be more related to more connected peers? uTP? Fast transfer rate? I'll test and see.
[quote="Switeck"]
Worst-case scenario may also be due to overloads on download or upload bandwidth sides. Packet loss is guaranteed if qBT is trying to upload at unlimited speed to lots of peers at once on anything other than ridiculous-fast (1+ gbit/sec UL) fiber lines.
[/quote]
I am struggling to keep my ratio at some PT site so I cancelled the upload limit. But since I am "hopelessly firewalled" by my ISP I am not getting much connection for seeding. Usually one or two from time to time. For leeching, usually I am connected to 10-15 peers each with a speed of 5-10 KiB/s.
Also it's interesting to note that the "Session upload & download" stat seems close to actual torrent data transfer. So it seems that "All-time UL/DL" stat is using a different code path or logic counting the possible big overhead, whereas the session stats are not.
Protocol overhead really can get silly if ..... While overheads are unlikely to double the base amount uploaded to peers, it's possible for overheads to do so at worst-case scenario.[/quote]
Thank you very much for the detailed response. Now I have the big picture in mind. I'll continue using this particular copy of QBT to see if I can find some pattern. Would this delta be more related to more connected peers? uTP? Fast transfer rate? I'll test and see.
[quote="Switeck"]
Worst-case scenario may also be due to overloads on download or upload bandwidth sides. Packet loss is guaranteed if qBT is trying to upload at unlimited speed to lots of peers at once on anything other than ridiculous-fast (1+ gbit/sec UL) fiber lines.
[/quote]
I am struggling to keep my ratio at some PT site so I cancelled the upload limit. But since I am "hopelessly firewalled" by my ISP I am not getting much connection for seeding. Usually one or two from time to time. For leeching, usually I am connected to 10-15 peers each with a speed of 5-10 KiB/s.
Also it's interesting to note that the "Session upload & download" stat seems close to actual torrent data transfer. So it seems that "All-time UL/DL" stat is using a different code path or logic counting the possible big overhead, whereas the session stats are not.
Last edited by SL9 on Sun Jun 23, 2019 9:21 am, edited 1 time in total.
-
SL9
Re: All-time upload does not match per-torrent sum by a huge margin
Since I've started this thread it's been another 18 hours of running, the margin has widened by a little bit, now it's at:
10.46 GiB "All-time upload" vs 7.03 GiB "Actual upload". Delta: 3.24 GiB -> 3.45 GiB (0.21 GiB)
Total session upload: 3.65 GiB
90.29 GiB "All-time download" vs 86.93 GiB "Actual download". Delta: 3.16 GiB -> 3.36 GiB (0.20 GiB)
Total session download: 1.96 GiB
So the "overhead" of download and upload is pretty close.
However, the big picture of download and upload is very different. Download is capped at 32 KiB/s with ~10 peers regularly connected, while upload is mostly idle with zero peers, connected to 1 or 2 peers from time to time and creating short bursts of traffic.
Upload tranferred more data, download connected to more peers, they ended up with roughly the same amount of "overhead" I am not able to come to a conclusion the prevailing factor.
10.46 GiB "All-time upload" vs 7.03 GiB "Actual upload". Delta: 3.24 GiB -> 3.45 GiB (0.21 GiB)
Total session upload: 3.65 GiB
90.29 GiB "All-time download" vs 86.93 GiB "Actual download". Delta: 3.16 GiB -> 3.36 GiB (0.20 GiB)
Total session download: 1.96 GiB
So the "overhead" of download and upload is pretty close.
However, the big picture of download and upload is very different. Download is capped at 32 KiB/s with ~10 peers regularly connected, while upload is mostly idle with zero peers, connected to 1 or 2 peers from time to time and creating short bursts of traffic.
Upload tranferred more data, download connected to more peers, they ended up with roughly the same amount of "overhead" I am not able to come to a conclusion the prevailing factor.
-
Switeck
Re: All-time upload does not match per-torrent sum by a huge margin
"But since I am "hopelessly firewalled" by my ISP I am not getting much connection for seeding."
For that, you could get a port forwarded VPN or rent a seedbox.
"Upload tranferred more data, download connected to more peers, they ended up with roughly the same amount of "overhead" I am not able to come to a conclusion the prevailing factor."
A lot of things are far more 2-way traffic than most people realize.
Being firewalled means having to make a lot more outgoing connections (and likely encrypted handshakes) to get the same results.
For that, you could get a port forwarded VPN or rent a seedbox.
"Upload tranferred more data, download connected to more peers, they ended up with roughly the same amount of "overhead" I am not able to come to a conclusion the prevailing factor."
A lot of things are far more 2-way traffic than most people realize.
Being firewalled means having to make a lot more outgoing connections (and likely encrypted handshakes) to get the same results.
-
fusk
Re: All-time upload does not match per-torrent sum by a huge margin
If you want to keep track of download and upload you should have a look at networx. It's awesome for that, i have a track record back to 2009.
-
SL9
Re: All-time upload does not match per-torrent sum by a huge margin
[quote="Switeck"]
A lot of things are far more 2-way traffic than most people realize.
Being firewalled means having to make a lot more outgoing connections (and likely encrypted handshakes) to get the same results.
[/quote]
Thanks for the follow-up. Recently I noticed that transport overhead can be separately displayed in the speed graph, and indeed it's quite significant. During a download session with 500KiB/s speed, overhead speed was ~20 KiB/s.
I also found that in uTorrent overhead speed graph can be further divided into "ack", "header", "connect/close", "retransmission", which seems interesting.
A lot of things are far more 2-way traffic than most people realize.
Being firewalled means having to make a lot more outgoing connections (and likely encrypted handshakes) to get the same results.
[/quote]
Thanks for the follow-up. Recently I noticed that transport overhead can be separately displayed in the speed graph, and indeed it's quite significant. During a download session with 500KiB/s speed, overhead speed was ~20 KiB/s.
I also found that in uTorrent overhead speed graph can be further divided into "ack", "header", "connect/close", "retransmission", which seems interesting.
-
SL9
Re: All-time upload does not match per-torrent sum by a huge margin
[quote="fusk"]
If you want to keep track of download and upload you should have a look at networx. It's awesome for that, i have a track record back to 2009.
[/quote]
Thanks for the advice. I just got aquired of networx back in May, and indeed it's an incredible tool, logged ~1.3TiB of transfer since then. Also I managed to persuade the developer to add more precise timespan control (hourly) in usage statistics view for the latest version.
If you want to keep track of download and upload you should have a look at networx. It's awesome for that, i have a track record back to 2009.
[/quote]
Thanks for the advice. I just got aquired of networx back in May, and indeed it's an incredible tool, logged ~1.3TiB of transfer since then. Also I managed to persuade the developer to add more precise timespan control (hourly) in usage statistics view for the latest version.