Upload speed drops
Upload speed drops
Is that normal? Every ~18 minutes upload speed drops and immediately reaches the normal level. That happens when OS cache is enabled. When OS cache is disabled rarely happens but still happens. Also when the drops happens there is a not typical hdd noise. I tried various combinations of large/small cache + short/long cache expiry but no results. With utorrent 2.2.1 there is no such behavior.
Last edited by qbi on Thu Apr 14, 2016 9:36 pm, edited 1 time in total.
Re: Upload speed drops
It might be, but how can we tell?Every ~18 minutes upload speed drops and immediately reaches the normal level.
Input! .... .... Need Input!
And what 'noise' is there instead?Also when the drops happens there is a not typical hdd noise.
Re: Upload speed drops
6 hours Look how regularly are the drops.ciaobaby wrote:It might be, but how can we tell?Every ~18 minutes upload speed drops and immediately reaches the normal level.
Input! .... .... Need Input!
30 minutes When the drop happens the speed graphic looks like "1". After some time happens another drop - the first drop in the graphic is already on place "2". Every next drop happens when the previous drop is in point "2". When the drops starts I'll make screenshot on 30 minutes graphic.
When drops happened I checked the "Statistics" and "Read cache overload" goes up to ~50%. I'm only browsing meantime. The hdd read activity goes up to 50%. Also my torrents are on hdds only for torrents. I have ssd for OS.
When you read from hdd the noise is constant without much change in the noise level. When drop happens the initial noise is very aggressive/high and then lowering.ciaobaby wrote:And what 'noise' is there instead?Also when the drops happens there is a not typical hdd noise.
P.S. I'm using qBittorrent 3.3.4/Libtorrent 1.0.9.0/Boost 1.60.0. With 3.3.3 the situation was the same.
30 minutes screenshot and6 hours
Last edited by qbi on Fri Apr 15, 2016 2:31 am, edited 1 time in total.
Re: Upload speed drops
If that was happening with a drive of mine, I would be ordering a brand new one pretty sharpish.When drop happens the initial noise is very aggressive/high and then lowering.
The two are not necessarily connected because there are many applications and services running on your machine that could interrupt or 'steal' network capacity"Read cache overload" goes up to ~50%. I'm only browsing meantime
With what libtorrent version?With 3.3.3 the situation was the same.
because it is libtorrent that handles all the I/O
Re: Upload speed drops
The hdd smart is excellent. That behavior starts when I changed utorrent with qBittorrent.ciaobaby wrote:If that was happening with a drive of mine, I would be ordering a brand new one pretty sharpish.When drop happens the initial noise is very aggressive/high and then lowering.
Why while OS cache is enabled happens almost always and while OS cache is disabled - very rarely happens?The two are not necessarily connected because there are many applications and services running on your machine that could interrupt or 'steal' network capacity"Read cache overload" goes up to ~50%. I'm only browsing meantime
With default libtorrent for 3.3.3 and with 1.0.9.With what libtorrent version?With 3.3.3 the situation was the same.
because it is libtorrent that handles all the I/O
P.S. I'm using 5 hdds only for torrenting without RAID.
Last edited by qbi on Fri Apr 15, 2016 12:50 pm, edited 1 time in total.
Re: Upload speed drops
[quote="qbi"]
Why while OS cache is enabled happens almost always and while OS cache is disabled - very rarely happens?
[/quote]
OS cache in windows is pretty bad, windows likes to preload stuff which leads to bloat.
I'd guess you have a large default cache size set in qbittorent, try lowering it or just not using OS cache and allowing qbittorent to manage itself.
As for reasons why your upload could periodically drop, could be simple packet loss, peers disconnecting from you as theres a better seeder, or your modem/router choking.
As you've said it only happens when you're browsing the internet its possible you have QOS rules setup in your router to prioritise port 80/8080/443/tcp traffic and its doing its job by choking your torrent upload to perform a handshake/lookup for a fraction of a second causing qbittorent to report a drop.
/e: Could also always be simple HDD failure as ciao has mentioned.
If my HDD was making that kind of noise i'd be looking into replacing it asap.
SMART isnt considered reliable in business environments btw.
Why while OS cache is enabled happens almost always and while OS cache is disabled - very rarely happens?
[/quote]
OS cache in windows is pretty bad, windows likes to preload stuff which leads to bloat.
I'd guess you have a large default cache size set in qbittorent, try lowering it or just not using OS cache and allowing qbittorent to manage itself.
As for reasons why your upload could periodically drop, could be simple packet loss, peers disconnecting from you as theres a better seeder, or your modem/router choking.
As you've said it only happens when you're browsing the internet its possible you have QOS rules setup in your router to prioritise port 80/8080/443/tcp traffic and its doing its job by choking your torrent upload to perform a handshake/lookup for a fraction of a second causing qbittorent to report a drop.
/e: Could also always be simple HDD failure as ciao has mentioned.
If my HDD was making that kind of noise i'd be looking into replacing it asap.
SMART isnt considered reliable in business environments btw.
Last edited by KitKat on Fri Apr 15, 2016 1:18 pm, edited 1 time in total.
Re: Upload speed drops
I wrote in the first post that I have tried small and large (auto, from 20MB to 2GB) cache size + long and short (from 15s to 600s) cache expiry.KitKat wrote: I'd guess you have a large default cache size set in qbittorent, try lowering it or just not using OS cache and allowing qbittorent to manage itself.
How OS cache affects packet loss, router problems and peer disconnecting?As for reasons why your upload could periodically drop, could be simple packet loss, peers disconnecting from you as theres a better seeder, or your modem/router choking.
When I wrote that I'm only browsing I meant that I'm doing no intensive things like copy/write on hdd or downloading. The drops happens also when I'm watching movie.As you've said it only happens when you're browsing the internet its possible you have QOS rules setup in your router to prioritise port 80/8080/443/tcp traffic and its doing its job by choking your torrent upload to perform a handshake/lookup for a fraction of a second causing qbittorent to report a drop.
Re: Upload speed drops
[quote="qbi"]
[quote="KitKat"]
I'd guess you have a large default cache size set in qbittorent, try lowering it or just not using OS cache and allowing qbittorent to manage itself.[/quote]
I wrote in the first post that I have tried small and large (auto, from 20MB to 2GB) cache size + long and short (from 15s to 600s) cache expiry.
[/quote]
I appear to have skipped over that, sorry.
Could you try disabling OS cache entirely and seeing if qbittorents management is better than window's in your case?
The option is found under tools > options > advanced and is called Enable OS Caching, un-tick it and restart qbittorent.
[quote="qbi"]
[quote="KitKat"]
As for reasons why your upload could periodically drop, could be simple packet loss, peers disconnecting from you as theres a better seeder, or your modem/router choking.
[/quote]
How OS cache affects packet loss, router problems and peer disconnecting?
[/quote]
OS cache doesnt affect packet loss or router problems.
It affects hardware read/write which COULD be a problem if you're doing something intensive, i never mentioned those things in that context.
[quote="qbi"]
[quote="KitKat"]
As you've said it only happens when you're browsing the internet its possible you have QOS rules setup in your router to prioritise port 80/8080/443/tcp traffic and its doing its job by choking your torrent upload to perform a handshake/lookup for a fraction of a second causing qbittorent to report a drop.[/quote]
When I wrote that I'm only browsing I meant that I'm doing no intensive things like copy/write on hdd or downloading. The drops happens also when I'm watching movie.
[/quote]
Browsing would involve sending TCP Handshakes which could cause Quality Of Service (QOS) router rules to drop or delay the torrent upload traffic to ensure your connection succeeds. If you're using 100% of your upload in qbittorent then QOS packet delaying could (and likely will) cause qbittorent upload drops in statistics like that, if you're not using all of your upload in qbittorent this is most likely not the cause.
For the record watching a movie can be harddrive intensive depending on the video codec and video size.
Its cache intensive at the very least.
/e:
Check MMC's Log for Read/Write Interrupts when qbittorent shows the upload drop, if they exist then whatever the process causing them is at fault.
[quote="ciaobaby"]
With what libtorrent version?
[/quote]
Not to be an asshat but the vast majority of windows users will be using the "release" builds as compiling this(most things really) under windows is a fairly tedious procedure until you have it setup..
So this question answers itself alot of the time :X
[quote="KitKat"]
I'd guess you have a large default cache size set in qbittorent, try lowering it or just not using OS cache and allowing qbittorent to manage itself.[/quote]
I wrote in the first post that I have tried small and large (auto, from 20MB to 2GB) cache size + long and short (from 15s to 600s) cache expiry.
[/quote]
I appear to have skipped over that, sorry.
Could you try disabling OS cache entirely and seeing if qbittorents management is better than window's in your case?
The option is found under tools > options > advanced and is called Enable OS Caching, un-tick it and restart qbittorent.
[quote="qbi"]
[quote="KitKat"]
As for reasons why your upload could periodically drop, could be simple packet loss, peers disconnecting from you as theres a better seeder, or your modem/router choking.
[/quote]
How OS cache affects packet loss, router problems and peer disconnecting?
[/quote]
OS cache doesnt affect packet loss or router problems.
It affects hardware read/write which COULD be a problem if you're doing something intensive, i never mentioned those things in that context.
[quote="qbi"]
[quote="KitKat"]
As you've said it only happens when you're browsing the internet its possible you have QOS rules setup in your router to prioritise port 80/8080/443/tcp traffic and its doing its job by choking your torrent upload to perform a handshake/lookup for a fraction of a second causing qbittorent to report a drop.[/quote]
When I wrote that I'm only browsing I meant that I'm doing no intensive things like copy/write on hdd or downloading. The drops happens also when I'm watching movie.
[/quote]
Browsing would involve sending TCP Handshakes which could cause Quality Of Service (QOS) router rules to drop or delay the torrent upload traffic to ensure your connection succeeds. If you're using 100% of your upload in qbittorent then QOS packet delaying could (and likely will) cause qbittorent upload drops in statistics like that, if you're not using all of your upload in qbittorent this is most likely not the cause.
For the record watching a movie can be harddrive intensive depending on the video codec and video size.
Its cache intensive at the very least.
/e:
Check MMC's Log for Read/Write Interrupts when qbittorent shows the upload drop, if they exist then whatever the process causing them is at fault.
[quote="ciaobaby"]
With what libtorrent version?
[/quote]
Not to be an asshat but the vast majority of windows users will be using the "release" builds as compiling this(most things really) under windows is a fairly tedious procedure until you have it setup..
So this question answers itself alot of the time :X
Last edited by KitKat on Fri Apr 15, 2016 2:30 pm, edited 1 time in total.
Re: Upload speed drops
I mentioned in the first post that when I disable OS cache the drops happens very rarely.KitKat wrote:I appear to have skipped over that, sorry.qbi wrote:I wrote in the first post that I have tried small and large (auto, from 20MB to 2GB) cache size + long and short (from 15s to 600s) cache expiry.KitKat wrote: I'd guess you have a large default cache size set in qbittorent, try lowering it or just not using OS cache and allowing qbittorent to manage itself.
Could you try disabling OS cache entirely and seeing if qbittorents management is better than window's in your case?
The option is found under tools > options > advanced and is called Enable OS Caching, un-tick it and restart qbittorent.
I capped my upload speed to 90%. I have problems when global connections are > 140 (slow browsing but upload speed is fine) so I limited global connections to 130. Actually in utorrent my global connections was ~200 and I have problems with browsing but never with upload drops in the torrent client. I don't enable any QOS in the router or in OS. You can see that now the global connections are less than the limit but drops happens. Also you can see that the drops are huge - from 5-6MB/s to < 1MB/s - browsing can't cause so huge drops.Browsing would involve sending TCP Handshakes which could cause Quality Of Service (QOS) router rules to drop or delay the torrent upload traffic to ensure your connection succeeds. If you're using 100% of your upload in qbittorent then QOS packet delaying could (and likely will) cause qbittorent upload drops in statistics like that, if you're not using all of your upload in qbittorent this is most likely not the cause.qbi wrote:When I wrote that I'm only browsing I meant that I'm doing no intensive things like copy/write on hdd or downloading. The drops happens also when I'm watching movie.KitKat wrote: As you've said it only happens when you're browsing the internet its possible you have QOS rules setup in your router to prioritise port 80/8080/443/tcp traffic and its doing its job by choking your torrent upload to perform a handshake/lookup for a fraction of a second causing qbittorent to report a drop.
Also when I'm watching movie which is on other hdd there is drop and that noise. Actually I don't want disable OS cache because when I'm watching something or I creating a torrent the upload speed drops but drops not for moment and then goes to the normal level, drops for the time when I'm watching the movie or creating the torrent which is normal when there is no cache.For the record watching a movie can be harddrive intensive depending on the video codec and video size.
Its cache intensive at the very least.
Do you mean the "Log" from "View" menu in the client?Check MMC's Log for Read/Write Interrupts when qbittorent shows the upload drop, if they exist then whatever the process causing them is at fault.
I'm compiling it with mxe. Under Windows was pain in the ass.ciaobaby wrote:Not to be an asshat but the vast majority of windows users will be using the "release" builds as compiling this(most things really) under windows is a fairly tedious procedure until you have it setup..With what libtorrent version?
So this question answers itself alot of the time :X
Re: Upload speed drops
You should probably ask Micro$haft for why they haven't fixed their crappy caching system at some point during the last twenty years, (Ever since NT4 was released in 1996)Why while OS cache is enabled happens almost always and while OS cache is disabled - very rarely happens?
Don't rely on S.M.A.R.T. getting it right, it is not that reliable.The hdd smart is excellent.
Re: Upload speed drops
Thank you for the answers, boys. Sadly but I'll back to utorrent to see if that problem will appear there.
When OS cache is disabled the buffer size in Statistics is almost the same size like when OS cache is enabled. I'm thinking that when I disable OS cache shouldn't exists buffers and every write/read is directly from/to the hdd without any buffers. How exactly works that buffers and OS cache. In Deluge there is no option "OS cache".
When OS cache is disabled the buffer size in Statistics is almost the same size like when OS cache is enabled. I'm thinking that when I disable OS cache shouldn't exists buffers and every write/read is directly from/to the hdd without any buffers. How exactly works that buffers and OS cache. In Deluge there is no option "OS cache".
Re: Upload speed drops
As on would expect, the required "size" does not change, only the mechanism of handling the cached items changes.the buffer size in Statistics is almost the same size like when OS cache is enabled
The memory cache is there for holding the 16 kB blocks of data as they arrive from remote peers and are actively being sent to other peers.How exactly works that buffers and OS cache.
The option of using either the Operating System or libtorrent to handle the cache was added to qbittorrent so that users where there usual running payloads cause problems with Windows caching.. In Deluge there is no option "OS cache".
Not everybody needs to use libtorrent caching, but when caching causes a problem because of the type of payloads that are running, users have the choice, the usual cause is high activity, multi-file payloads on Windows operating systems, so if you NEVER run that type of torrent you will probably never come across the problem at all.
Re: Upload speed drops
So when I disable OS cache actually I enable libtorrent cache?
What I have to change in the libtorrent cache settings so the speed remains in the same level (like when OS cache is enabled) when I'm watching movie or creating torrent?
The libtorrent cache size and cache expiry are ignored by the same settings in qBittorrent, right?
I tried settings for high_performance_seed:
set.cache_size = 32768 * 2;
set.use_read_cache = true;
set.cache_buffer_chunk_size = 128;
set.read_cache_line_size = 32;
set.write_cache_line_size = 32;
set.low_prio_disk = false;
Also I tried cache_line 64, buffer_chunk 254/64, cache_size 65536*2/, 32768 but the speed always drops with OS disabled when I'm playing movie or creating torrents.
What I have to change in the libtorrent cache settings so the speed remains in the same level (like when OS cache is enabled) when I'm watching movie or creating torrent?
The libtorrent cache size and cache expiry are ignored by the same settings in qBittorrent, right?
I tried settings for high_performance_seed:
set.cache_size = 32768 * 2;
set.use_read_cache = true;
set.cache_buffer_chunk_size = 128;
set.read_cache_line_size = 32;
set.write_cache_line_size = 32;
set.low_prio_disk = false;
Also I tried cache_line 64, buffer_chunk 254/64, cache_size 65536*2/, 32768 but the speed always drops with OS disabled when I'm playing movie or creating torrents.
Re: Upload speed drops
Neither of those are remotely connected to the qbittorrent cache which is a separate entity to the system caching, yes it uses the same hardware (RAM) but the two do not interact especially once the OS cache is bypassed.when I'm watching movie or creating torrent?
"Speeds" dropping when you are 'creating' a torrent is nothing at all to do with caching. It is because libtorrent is single threaded, so when a processor intensive operation is being done such as hash-summing and committing pieces to permanent storage, creating the meta-data, running a force recheck, moving a payload etc. all other I/O stops until that task is complete, there are two or three threads around here about network transfer speeds dropping to zero during a payload move between file systems or drives.but the speed always drops with OS disabled when I'm playing movie or creating torrents.
While watching 'movies' will probably be due to a similar issue, where the media player application has a higher thread priority than qbittorrent/libtorrent, this will possibly be more noticeable if your CPU uses "Hyper-Threading" to simulate 'multiple cores'.
Re: Upload speed drops
Thanks for the explanation.
Why there is a min (15s)/max (600s) cache expiry limit in qBittorrent?
When creating torrent with utorrent with checked "Disable Windows caching of disk reads and writes" there is no speed drop. Is this mean that utorrent protocol is multithreaded?"Speeds" dropping when you are 'creating' a torrent is nothing at all to do with caching. It is because libtorrent is single threaded, so when a processor intensive operation is being done such as hash-summing and committing pieces to permanent storage, creating the meta-data, running a force recheck, moving a payload etc. all other I/O stops until that task is complete, there are two or three threads around here about network transfer speeds dropping to zero during a payload move between file systems or drives.but the speed always drops with OS disabled when I'm playing movie or creating torrents.
I checked the media player application priority - yes, the default priority is "above normal".While watching 'movies' will probably be due to a similar issue, where the media player application has a higher thread priority than qbittorrent/libtorrent, this will possibly be more noticeable if your CPU uses "Hyper-Threading" to simulate 'multiple cores'.
Why there is a min (15s)/max (600s) cache expiry limit in qBittorrent?