Qbt slower than rtorrent - thoughts on why?

[mhertz]

Deluge(and hence qbt, as both use libtorrent for backend) has always been mentioned as faster than rtorrent, and hence, many have both running, deluge for snatch 'n grab(quick downloads) and then rtorrent for long-term seeding. When looking at the download speed reported in rtorrent, and comparing that to qbt, then indeed rtorrent is slower in getting up to full speed than qbt, but this is an UI bug it seems, because i've timed it with a stopwatch, and every case rtorrent is about 10-20% faster than qbt/deluge(i've only tested files of max 1 minuttes download time granted, and around 500 mb). I've tried messing with cache settings and connections, but to no avail.

Could someone please give me there thoughts on why rtorrent is quicker than qbt? Also, if anybody got any tips to make the two comparable in speed?  libtorrent-rasterbar is more developed and up-to-date than rtorrent's libtorrent, and it's often stated that rtorrent is very conservative compared to other clients and where libtorrent-rasterbar is classified as a pretty aggressive "client", so it makes no sense to me these things i'm finding. Qbittorrent is slightly faster than deluge often, even though both have same backend, so i'm guessing this is python slowing things down for deluge. Also, qbt is set to contact all trackers at once, where I know rtorrent only uses one tracker and just uses another from other tier every 30 secs if x amount of peers where obtained before and/or max 3 tries. Again, I don't get it. I made sure to enable encryption, hash-checks, dht and Utp in rtorrent to maintain same level, but no matter what, rtorrent is fastest.

Thanks in advance, and sorry for somewhat strange question, but i'm a little strange and care about such semantics I thought I used the fastest torrent-downloader on *nix, but found otherwise. I cannot switch to rtorrent(I use qbt-nox together with shell-scripts/webUI), as I need socks5 proxy support which isn't available in rtorrent(socks5 proxy obviously disabled during testing ).

Edit: Obviously swarms vary, so direct comparison is difficult and practically impossible(unless simulation where controlling all peers yourself), but i've tested this about 50 times now, on/off over many months, so have a pretty good indication of the performance capabilities of each client. Btw, I have a tested 168/34'ish mbit cable connection and ext4 formatted SSD, intel core I5 CPU, 4GB ram, and running arch linux and using latest stable of each software - same result with default settings and "tweaked".

[Switeck]

There probably is no single magic bullet fix for libtorrent that will make qBT a lot faster, but there's a lot of work needed to be done.
There's lots of weird low-level bugs in libtorrent that may/may not show up depending on settings changes in qBT:
https://github.com/arvidn/libtorrent/issues/2454
https://github.com/arvidn/libtorrent/issues/2719
https://github.com/arvidn/libtorrent/issues/2845
https://github.com/qbittorrent/qBittorrent/issues/7745
https://github.com/qbittorrent/qBittorrent/pull/7974
https://github.com/qbittorrent/qBittorrent/issues/8433
https://github.com/qbittorrent/qBittorrent/issues/7155
https://github.com/qbittorrent/qBittorrent/issues/7104
https://github.com/qbittorrent/qBittorrent/issues/7881

Disabling uTP and using only TCP is the biggest I know of, but that won't help on torrent swarms where most other peers+seeds expect uTP connections:
index.php/topic,3982.msg24170.html#msg24170
https://github.com/arvidn/libtorrent/issues/2752
"The utp_stream.cpp code is in pretty serious need of cleaning up and modernisation as well. Perhaps such effort would tease out this issue."

Instead of disabling uTP totally, another option is setting prefer TCP to TRUE.
prefer_tcp set to false also causes local peers to slow to almost nothing when internet uTP peers are connected even when LAN speeds are set to unlimited.
Send upload piece suggestions cripples upload speed for some reason. I'm guessing it's an super/initial seeding type of feature to be used with low upload max?
Encryption Mode set to Prefer or Require encryption also resulted in upload speeds being reduced despite ~10% LESS cpu usage WITH encryption used.
Apply rate limit to transport overhead can also cause drastic speed decreases, maybe even worse than setting upload speed max slightly too low.

qBT is a long ways away from hitting 1 gbit/sec speeds for many people:
index.php/topic,4237.msg22062.html#msg22062
index.php/topic,4716.0.html
index.php/topic,5343.0.html
index.php/topic,5509.0.html

Having said that, I've done "EXTREME Speed Tests of various BitTorrent Apps" under special conditions and gotten vastly different results:
index.php/topic,3956.0.html
The "catch" being, I used a ramdrive for storage, 127.0.0.1 local loopback as the network connection between the BT clients, and a semi-fast cpu that's not doing much else.
And of course...with uTP disabled!
qBitTorrent was arguably the fastest there, however I did not test rtorrent as I'm not running Linux.

[mhertz]

Wov, thanks a bunch Switeck, that's some reply you've made there for me to digg into and analyze for sure, thanks man for the effort! Btw, I remember your name alot from googling things like this for years, like on the utorrent forum and here etc. You're a great resource and I thank you very much not only for this reply but for all your past efforts, posts and testings! You helped me here previously also, a year or two back, if I remember correct - a question regarding memory usage between qbt and deluge I'd wish rtorrent where included in your tests, but of course if on windows only then obviously not a candidate - It just was weird to me, because rtorrent is used ALOT on seedboxes and has always had a rep of beeing lightweight and conservative and good for seeding thousands of torrent, but NEVER been said to have any edge whatsoever on DL speeds, so this baffled me. I used to use utorrent for years back when I used windows, in the non-bloated/adds versions/times, and there's a version available for *nix too, called utorrent server, with a webui, and the speeds there is comparabel to qbt. 

Again, appreciated for sure, and i'll slowly digg in

Edit: Damn, I think that rtorrent doesn't even support uTP! It supports UDP through UDP trackers(and DHT), so was sure it also supported uTP i.e. UDP connections instead of  TCP. I have some testings to do now, thanks to Switeck!

[mhertz]

Indeed, the sole issue where uTP, and in the case of rtorrent, lack of uTP support(even though it supports UDP???), so when disabling uTP in qBT, then the speed is the same. I'm happy about this, as I really couldn't make any sense out of it. Thank you so much Switeck for finally clearing this issue op for me! It wasen't an actual problem, but more of an "intellectual issue", which I wanted clarification upon, and now gotten! The other links i've gotten still to me is useful to learn more about the semantics Thanks again!

[Switeck]

uTP is still important, because some/many peers+seeds may be using uTP in preference over TCP.
It's especially important to have uTP, DHT, and PEX enabled on public torrents that have few seeds+peers, because together those will help qBitTorrent find+connect to almost all of them.

uTP can hole-punch through NAT routers that aren't port forwarded, so people on it can get incoming connections. Normally, when someone is firewalled they cannot get incoming connections but thanks to uTP/UDP tricks they sometimes can. (PhD-worthy papers have been written on the subject.) With more-and-more people behind routers they can't control, especially on ISPs that only give out IPv4 LAN ip addresses to their customers, this is of growing importance...unless everyone quickly jumps over to IPv6.

uTP if fixed has potential to be better than TCP in terms of efficiency due to UDP packets having less overhead than TCP packets of the same size. So for the same number of packets sent and the same amount of bandwidth used, uTP could result in 1-5% higher upload and download speeds.

[mhertz]

Thanks alot again Switeck for the insight! I always just thought that uTP would be fastest, as UDP instead of TCP, and as you say, UDP is without the TCP overhead, so SHOULD be faster, though not currently/always it seems. Actually, i'm not even disabling uTP, as also thought about the "compatibility" issue, and rather wan't maximum "compatibility", than faster speed sometimes. Also, as I said, it wasen't an issue for me, but just buggled my brain, and I where after a clarification simply, of why rtorrent, known as conservative, would beat qBT every single time.

Indeed, uTP has benefits, one of the ones you state(NAT hole-punching) I've discovered before - well I don't know if it specifically was NAT hole-punching, but that I could get incoming connections through socks5 proxy - that isn't possible with torrents because of specific semantics of the protocol(developed primarily with passive FTP in mind, I believe - Arvid tried to implement it once, but removed it cause it didn't work). Whenever I got an incoming connection through the socks5 proxy, then it was always with uTP and gotten from PEX or DHT. I mailed Arvid, lead-dev of libtorrent, some years ago about this, and he stated it was because of uTP and that both DHT and PEX would preserve your UDP port and correctly map it to the one bound on the proxy. Many think that no incoming connections means no uploads, but as you know the bittorrent protocol is asymmetric, and supports uploading also through outgoing connections, but of course incoming connections helps obviously.

Thanks again!

Edit: Just read up on uTP and the unofficial UDP hole-punching protocol-extension(BEP10) - This is cool stuff! So UDP hole-punching is supported between qBt and clients supporting uTP and BEP10, if of course having uTP enabled on both ends.

[Switeck]

That's why I recommend setting prefer TCP to TRUE. It prevents uTP speed regulations from crippling TCP peers/seeds. You'll need sane DL+UL speed limits or TCP and/or uTP will cause mega-lag on busy torrents.

Now uTP-based UDP hole-punching has its limits...if there's only 1 peer and 1 seed on a torrent and both are firewalled, there's no other peers/seeds to "introduce" them to each other and no connection will succeed. Ironically, even a firewalled peer/seed could introduce other firewalled peers/seeds to each other but that scenario is probably unlikely. There always needs to be at least 1 unfirewalled peer/seed to act as an initiator of it all.

[mhertz]

Thanks mate! Yes, I have mixed mode set to prefer TCP, or rather, i've left it as per default This, and the other settings as per default, gives me 10-20% slower speed than rtorrent(all TCP), but I don't mind, as long as I know the explanation for it, and that I can change it if wanted, + as said, I prefer the added compatibility of it. Currently, I haven't bothered messing with DL/UP constrictions, because I only upload a little, and never max out my upload or even close. As said, I use a socks5 proxy(when not testing stuff ), and there only outgoing connections work(and a few uTP incoming ones), and even though I can upload through outgoing connections, I only upload little and sometimes not at all, which is a little strange, but my speeds are fine regardless and I max out my connection on popular torrents with 20 MB/sec on my 150/30 mbit cable connection, so not complaining(with proxy enabled, around 17MB/sec). I usually don't use my computer when downloading, so not bothered if the TCP connections hog the system. I don't download much at all actually, and am more intellectually interested in it honestly Yeah, I don't expect the UDP hole-punching to work miracles e.g. between 2 firewall'ed peers, but it's a nice addition for sure to have available - I didn't knew about this NAT traversal thing beforehand Thanks again

-Martin.