Hello. I have qBittorrent 3.3.16 running with about 600 torrents. I use private trackers and have DHT and PEX disabled. I noticed the qBittorrent speed graph shows about 10 kilobytes/sec upload and 5 kbytes/sec download when no torrents are active. I thought this was weird, so I use Wireshark to sniff the connection.
First thing I find is the "bittorrent" part of packets is illegible. I remembered I have encryption option set to "Require Encryption." Changing this setting to "disabled" immediately dropped my network traffic to 2.5 kbyte/s upload and 1.2 kbyte/s down. With encryption disabled, I could understand the packets in Wireshark. Does encryption add this much overhead to bittorrent?
With encryption disabled, I could now inspect the packets. I found that my computer is initiating almost all of the bittorrent traffic. My computer starts a connection to a remote computer, sends a Bittorrent "Handshake" of 68 bytes, which contains info_hash and my peer_id. The remote computer then closes the connection with a RST flag.
It seems to be a lot of traffic. With 39 seconds captured there were 403 Bittorrent packets, about 10 packets/second. Could anyone help me to understand this behaviour? Is it normal to send so many handshakes on torrent files that are already completed, but with no actual data transfer?
If I ran my qBittorrent client 24/7 with 10k/5kbs traffic that would amount to 26/13 Gigabytes per month traffic. That's all overhead and no actual data transfer. Is this normal? Thanks.
Bittorrent traffic when QBT is idle.
-
Switeck
Re: Bittorrent traffic when QBT is idle.
"Does encryption add this much overhead to bittorrent?"
Yes. Encapsulating that inside of a proxy and/or VPN adds even more.
"I found that my computer is initiating almost all of the bittorrent traffic. My computer starts a connection to a remote computer, sends a Bittorrent "Handshake" of 68 bytes"
This is the result of a high half-open limit in qBT and the hard-coded outgoing connection rate in qBT.
"That's all overhead and no actual data transfer. Is this normal?"
qBT is retrying dead ips and seed ips on torrents you're seeding as fast as every minute, assuming you've set the half open limit high enough to do that.
"Is it normal to send so many handshakes on torrent files that are already completed, but with no actual data transfer?"
It isn't helped by trackers that hand out seed ips to other seeds as though they have any reason to connect to each other. A torrent can have 100 seeds and 2 peers, and the tracker will hand out 47-50 seed ips to each seed but may not hand out the peer ips at all!
Higher quality trackers only hand out peers ips to seeds and peers+seeds ips to other peers and also reduces or removes old ips that haven't responded in 1-3 handshake intervals to reduce this problem, but private trackers seldom use those because most of them are public trackers only.
Yes. Encapsulating that inside of a proxy and/or VPN adds even more.
"I found that my computer is initiating almost all of the bittorrent traffic. My computer starts a connection to a remote computer, sends a Bittorrent "Handshake" of 68 bytes"
This is the result of a high half-open limit in qBT and the hard-coded outgoing connection rate in qBT.
"That's all overhead and no actual data transfer. Is this normal?"
qBT is retrying dead ips and seed ips on torrents you're seeding as fast as every minute, assuming you've set the half open limit high enough to do that.
"Is it normal to send so many handshakes on torrent files that are already completed, but with no actual data transfer?"
It isn't helped by trackers that hand out seed ips to other seeds as though they have any reason to connect to each other. A torrent can have 100 seeds and 2 peers, and the tracker will hand out 47-50 seed ips to each seed but may not hand out the peer ips at all!
Higher quality trackers only hand out peers ips to seeds and peers+seeds ips to other peers and also reduces or removes old ips that haven't responded in 1-3 handshake intervals to reduce this problem, but private trackers seldom use those because most of them are public trackers only.
-
tekno
Re: Bittorrent traffic when QBT is idle.
[quote="Switeck"]
"I found that my computer is initiating almost all of the bittorrent traffic. My computer starts a connection to a remote computer, sends a Bittorrent "Handshake" of 68 bytes"
This is the result of a high half-open limit in qBT and the hard-coded outgoing connection rate in qBT.
[/quote]
I found there is a setting for half open connections in qBT. Tools->Options->Advanced->Maximum number of half-open connections. In my case it is set to 20. I believe this is the default. Is this a high number?
[quote="Switeck"]
"That's all overhead and no actual data transfer. Is this normal?"
qBT is retrying dead ips and seed ips on torrents you're seeding as fast as every minute, assuming you've set the half open limit high enough to do that.
[/quote]
I used Wireshark's filter to display Bittorrent packets from just 1 remote IP address. Indeed, it does send handshakes out to a peer/seed every 60 seconds. But, why? If there isn't any actual data transfer, why handshake? Just a lot of wasted bandwidth. A handshake should only happen when a peer wants to download. If I have 100% of a torrent, it shouldn't even try handshaking to a seed. Seeds already have 100% of a torrent.
[quote="Switeck"]
It isn't helped by trackers that hand out seed ips to other seeds as though they have any reason to connect to each other. A torrent can have 100 seeds and 2 peers, and the tracker will hand out 47-50 seed ips to each seed but may not hand out the peer ips at all!
Higher quality trackers only hand out peers ips to seeds and peers+seeds ips to other peers and also reduces or removes old ips that haven't responded in 1-3 handshake intervals to reduce this problem, but private trackers seldom use those because most of them are public trackers only.
[/quote]
I think I understand what you are saying. Is it the tracker's fault for telling my qBT that there are seeds/peers that need connecting?
Would your example of a tracker sending 50 seed IPs but no peer IPs help explain why it is hard for people with relatively slow connections to seed on a private tracker?
Even still, shouldn't qBT be smart enough to refrain from contacting other seeds about a torrent if itself is already seeding that torrent?
qBittorrent does keep a tally of how many seeds and peers there are for each torrent.
I didn't understand the last line about private trackers being public trackers.
Thanks for your help.
"I found that my computer is initiating almost all of the bittorrent traffic. My computer starts a connection to a remote computer, sends a Bittorrent "Handshake" of 68 bytes"
This is the result of a high half-open limit in qBT and the hard-coded outgoing connection rate in qBT.
[/quote]
I found there is a setting for half open connections in qBT. Tools->Options->Advanced->Maximum number of half-open connections. In my case it is set to 20. I believe this is the default. Is this a high number?
[quote="Switeck"]
"That's all overhead and no actual data transfer. Is this normal?"
qBT is retrying dead ips and seed ips on torrents you're seeding as fast as every minute, assuming you've set the half open limit high enough to do that.
[/quote]
I used Wireshark's filter to display Bittorrent packets from just 1 remote IP address. Indeed, it does send handshakes out to a peer/seed every 60 seconds. But, why? If there isn't any actual data transfer, why handshake? Just a lot of wasted bandwidth. A handshake should only happen when a peer wants to download. If I have 100% of a torrent, it shouldn't even try handshaking to a seed. Seeds already have 100% of a torrent.
[quote="Switeck"]
It isn't helped by trackers that hand out seed ips to other seeds as though they have any reason to connect to each other. A torrent can have 100 seeds and 2 peers, and the tracker will hand out 47-50 seed ips to each seed but may not hand out the peer ips at all!
Higher quality trackers only hand out peers ips to seeds and peers+seeds ips to other peers and also reduces or removes old ips that haven't responded in 1-3 handshake intervals to reduce this problem, but private trackers seldom use those because most of them are public trackers only.
[/quote]
I think I understand what you are saying. Is it the tracker's fault for telling my qBT that there are seeds/peers that need connecting?
Would your example of a tracker sending 50 seed IPs but no peer IPs help explain why it is hard for people with relatively slow connections to seed on a private tracker?
Even still, shouldn't qBT be smart enough to refrain from contacting other seeds about a torrent if itself is already seeding that torrent?
qBittorrent does keep a tally of how many seeds and peers there are for each torrent.
I didn't understand the last line about private trackers being public trackers.
Thanks for your help.
-
Switeck
Re: Bittorrent traffic when QBT is idle.
"I found there is a setting for half open connections in qBT. Tools->Options->Advanced->Maximum number of half-open connections. In my case it is set to 20. I believe this is the default. Is this a high number?"
That's 20 half open connections at a time, pretty much ALL the time while running lots of torrents that each have 10+ seeds+peers on them. The moment a handshake starts with an outgoing connection it is no longer half-open and another half-open connection is started in its place to maintain the 20-at-once limit. The upper limit is probably about 20 outgoing handshakes per second, not counting any additional incoming connections you might get.
I read elsewhere that...
Encryption has zero overhead after the encryption-handshake (which is on average 774 bytes + overhead for three extra TCP or uTP/UDP packets)
...I don't know if that's true.
"Is it the tracker's fault for telling my qBT that there are seeds/peers that need connecting?"
Considering on private trackers, that's the ONLY way for peers and seeds to find each other...I'd say yes.
"Would your example of a tracker sending 50 seed IPs but no peer IPs help explain why it is hard for people with relatively slow connections to seed on a private tracker?"
One of a couple reasons.
The other is download times vs tracker update times. If someone can download a torrent in 10 minutes but only updates every 30 minutes, then the tracker reports they're still a peer LONG after they've finished the torrent. If a new peer joins the torrent swarm and does its 1st tracker announce 2 minutes after you did your last one, then you're 28 minutes away from your NEXT tracker update to get that peer's ip. If it doesn't make an outgoing connection to you, by the time your BitTorrent client attempts its ip...it's already a seed as well.
If any peer or seed is firewalled, (which can happen due to VPNs and proxies as well as software firewalls) they're nearly unreachable for other peers and will likely have an extremely hard time maintaining a decent ratio.
There's also a huge amount of contention on busy torrents -- lots of seeds want to upload but peers can only download so fast:
index.php/topic,2911.msg14171.html#msg14171
"Even still, shouldn't qBT be smart enough to refrain from contacting other seeds about a torrent if itself is already seeding that torrent?"
Yes, qBT should be smart enough to only retry seed ips ONCE every time the torrents are started...but it probably doesn't.
There should also be an ever-increasing cooldown timer for failed ips. Those should only be retried maybe once an hour to once a day if they've never succeeded even after 5+ tries.
"qBittorrent does keep a tally of how many seeds and peers there are for each torrent."
Which is generally wrong.
Some peers/seeds may be on quick-changing ip addresses, so get counted multiple times. Some peers/seeds have multiple ip addresses at once -- such as IPv4 and IPv6. Some peers may stop the torrent shortly after they become a seed. qBT may be counting peers/seeds "seen" only once over a week ago...or trackers may incorrectly report ips as peers and qBT sees those ips as seeds but "forgets" to remove them from the peers list.
Knowledge of the exact number of peers and seeds on a torrent is generally delayed by minutes due to time between tracker updates or if some peers/seeds refuse to connect to you (because they're firewalled or already have max connections). Public torrents using Peer Exchange (PEX) can get a quicker snapshot of how many peers+seeds are on the torrent, but it is still delayed and imperfect information. To instantly keep track of all the peers/seeds on a torrent would take infinite bandwidth and 0 latency.
Even qBT's settings can greatly change their "view" of a torrent swarm:
index.php/topic,2911.msg14150.html#msg14150
(and following comments)
"I didn't understand the last line about private trackers being public trackers."
Private tracker websites can't run public tracker software that has the logic to avoid seeds getting seed ips. They need private tracker software which tends to lack that.
That's 20 half open connections at a time, pretty much ALL the time while running lots of torrents that each have 10+ seeds+peers on them. The moment a handshake starts with an outgoing connection it is no longer half-open and another half-open connection is started in its place to maintain the 20-at-once limit. The upper limit is probably about 20 outgoing handshakes per second, not counting any additional incoming connections you might get.
I read elsewhere that...
Encryption has zero overhead after the encryption-handshake (which is on average 774 bytes + overhead for three extra TCP or uTP/UDP packets)
...I don't know if that's true.
"Is it the tracker's fault for telling my qBT that there are seeds/peers that need connecting?"
Considering on private trackers, that's the ONLY way for peers and seeds to find each other...I'd say yes.
"Would your example of a tracker sending 50 seed IPs but no peer IPs help explain why it is hard for people with relatively slow connections to seed on a private tracker?"
One of a couple reasons.
The other is download times vs tracker update times. If someone can download a torrent in 10 minutes but only updates every 30 minutes, then the tracker reports they're still a peer LONG after they've finished the torrent. If a new peer joins the torrent swarm and does its 1st tracker announce 2 minutes after you did your last one, then you're 28 minutes away from your NEXT tracker update to get that peer's ip. If it doesn't make an outgoing connection to you, by the time your BitTorrent client attempts its ip...it's already a seed as well.
If any peer or seed is firewalled, (which can happen due to VPNs and proxies as well as software firewalls) they're nearly unreachable for other peers and will likely have an extremely hard time maintaining a decent ratio.
There's also a huge amount of contention on busy torrents -- lots of seeds want to upload but peers can only download so fast:
index.php/topic,2911.msg14171.html#msg14171
"Even still, shouldn't qBT be smart enough to refrain from contacting other seeds about a torrent if itself is already seeding that torrent?"
Yes, qBT should be smart enough to only retry seed ips ONCE every time the torrents are started...but it probably doesn't.
There should also be an ever-increasing cooldown timer for failed ips. Those should only be retried maybe once an hour to once a day if they've never succeeded even after 5+ tries.
"qBittorrent does keep a tally of how many seeds and peers there are for each torrent."
Which is generally wrong.
Some peers/seeds may be on quick-changing ip addresses, so get counted multiple times. Some peers/seeds have multiple ip addresses at once -- such as IPv4 and IPv6. Some peers may stop the torrent shortly after they become a seed. qBT may be counting peers/seeds "seen" only once over a week ago...or trackers may incorrectly report ips as peers and qBT sees those ips as seeds but "forgets" to remove them from the peers list.
Knowledge of the exact number of peers and seeds on a torrent is generally delayed by minutes due to time between tracker updates or if some peers/seeds refuse to connect to you (because they're firewalled or already have max connections). Public torrents using Peer Exchange (PEX) can get a quicker snapshot of how many peers+seeds are on the torrent, but it is still delayed and imperfect information. To instantly keep track of all the peers/seeds on a torrent would take infinite bandwidth and 0 latency.
Even qBT's settings can greatly change their "view" of a torrent swarm:
index.php/topic,2911.msg14150.html#msg14150
(and following comments)
"I didn't understand the last line about private trackers being public trackers."
Private tracker websites can't run public tracker software that has the logic to avoid seeds getting seed ips. They need private tracker software which tends to lack that.
-
tekno
Re: Bittorrent traffic when QBT is idle.
Thanks for the detailed reply. Your post gives some insight into the realities of how bittorrent works.
[quote="Switeck"]
"Is it the tracker's fault for telling my qBT that there are seeds/peers that need connecting?"
Considering on private trackers, that's the ONLY way for peers and seeds to find each other...I'd say yes.
[/quote]
I should have been more detailed with my question. I know trackers are how peers/seed find each other. But, what I was trying to ask is: is it the tracker's action that is causing my qBT to repeatedly initiate handshake connections to other seeds/peers when no data transfer is needed? What is the point in repeatedly handshaking with a seed/peer that does not request data?
Doesn't a peer decide to initiate connections and request data when the peer needs it? Peers should get a list of seed IPs from the tracker.
Like I wrote in my earlier post. 5 kb/s doesn't sound like much data transfer, but it adds up to 13 GB per month. Lots of handshaking but without even transferring any actual data.
[quote="Switeck"]
Private tracker websites can't run public tracker software that has the logic to avoid seeds getting seed ips. They need private tracker software which tends to lack that.
[/quote]
OK, so the situation could be improved if the private tracker software had better logic?
[quote="Switeck"]
"Is it the tracker's fault for telling my qBT that there are seeds/peers that need connecting?"
Considering on private trackers, that's the ONLY way for peers and seeds to find each other...I'd say yes.
[/quote]
I should have been more detailed with my question. I know trackers are how peers/seed find each other. But, what I was trying to ask is: is it the tracker's action that is causing my qBT to repeatedly initiate handshake connections to other seeds/peers when no data transfer is needed? What is the point in repeatedly handshaking with a seed/peer that does not request data?
Doesn't a peer decide to initiate connections and request data when the peer needs it? Peers should get a list of seed IPs from the tracker.
Like I wrote in my earlier post. 5 kb/s doesn't sound like much data transfer, but it adds up to 13 GB per month. Lots of handshaking but without even transferring any actual data.
[quote="Switeck"]
Private tracker websites can't run public tracker software that has the logic to avoid seeds getting seed ips. They need private tracker software which tends to lack that.
[/quote]
OK, so the situation could be improved if the private tracker software had better logic?
-
Switeck
Re: Bittorrent traffic when QBT is idle.
"Doesn't a peer decide to initiate connections and request data when the peer needs it?"
No, peers make connections almost in contradiction to need.
"Peers should get a list of seed IPs from the tracker."
No, peers should get a list of peer and seed ips from the tracker.
Once a peer is connected to a bunch of other peers and a few seeds, the other peers should be where that peer does the majority of its downloading -- only using the seeds for pieces the other peers appear to be missing. Currently, peers often demand 1st and last pieces from seeds as well as other peers with no consideration of rarity. Sequential downloading is an even more severe form of the same problem.
"What is the point in repeatedly handshaking with a seed/peer that does not request data?"
So long as a BitTorrent client isn't at max connections either globally or per torrent (for each torrent), it will continue to make outgoing connection attempts so long as its unconnected ip cache isn't "exhausted". That means retrying the same ips over-and-over again. There seems to be little-to-no memory if it's tried those ips before, except to avoid retrying the same ip within 1 minute of the previous time because some BT clients would auto-ban them for hostile activity.
Even if you put in sensible max connection limits, others don't...so you'll still see a lot of incoming connections just because they can.
You're not the only one who wants a change from that insanity.
"OK, so the situation could be improved if the private tracker software had better logic?"
Sadly, this would only slightly improve efficiency. This would just let seeds spam connection attempts to peer ips earlier.
No, peers make connections almost in contradiction to need.
"Peers should get a list of seed IPs from the tracker."
No, peers should get a list of peer and seed ips from the tracker.
Once a peer is connected to a bunch of other peers and a few seeds, the other peers should be where that peer does the majority of its downloading -- only using the seeds for pieces the other peers appear to be missing. Currently, peers often demand 1st and last pieces from seeds as well as other peers with no consideration of rarity. Sequential downloading is an even more severe form of the same problem.
"What is the point in repeatedly handshaking with a seed/peer that does not request data?"
So long as a BitTorrent client isn't at max connections either globally or per torrent (for each torrent), it will continue to make outgoing connection attempts so long as its unconnected ip cache isn't "exhausted". That means retrying the same ips over-and-over again. There seems to be little-to-no memory if it's tried those ips before, except to avoid retrying the same ip within 1 minute of the previous time because some BT clients would auto-ban them for hostile activity.
Even if you put in sensible max connection limits, others don't...so you'll still see a lot of incoming connections just because they can.
You're not the only one who wants a change from that insanity.
"OK, so the situation could be improved if the private tracker software had better logic?"
Sadly, this would only slightly improve efficiency. This would just let seeds spam connection attempts to peer ips earlier.