Total byte control !

[ciaobaby]

And have you reduced the cache setting to something sensible?

[nico910]

http://img11.hostingpics.net/pics/172617Capture.png

I tried with only 256k Pieces size and got same results than big pieces, qbt use 1.4GB max when i set limit to 1024MB

[sledgehammer_999]

This shouldn't be happening.
I updated the bug report with your remarks nico910. Please check that I mentioned them correctly.
The bug most probably exists in 64bits too. But those apps can address so much memory that you won't hit the "crashing limit".

[nico910]

not correct, when i set limit to 1000Mb qbt no longer crash ! qbt use max 1400Mb

1000MB limit -> qbt use ~1400Mb max
1400MB limit -> qbt use >1800MB and crash

that's why i said there is a calculation problem.

i tried first to seed only big pieces, then to seed only 256k pieces and got the same results, so pieces size are not the problem ! when i set limit to 1000MB qbt never use more than ~1400MB

With 64 bits qbt seems limited to 2048 MB he won't use more memory, i set cache expiracy to 600s and i'm seeding between 10mb/s and 20mb/s and qbt is always using ~2030 MB this value don't move.
EDIT : Nevermind ! the cache limit set on "auto" is limited to 2048 MB but i can set the cache limit manually to 4096MB max then qbt use ~5200MB without crash

[quote="Switeck"]
Actually, last I looked setting the cache to 1500 MB can cause a crash because qBT uses about 1.32 times as much ram as the cache size. Each increase in the cache of 100 MB increases ram use by ~132 MB.
[/quote]

this idea is very interesting, i'm not sure if the value is correct and if he can prove it but we are sure there is something wrong about limits.

~5285/4096 = 1.29...
~1400/1024 = 1.36...

the coeff is not perfect but we can observe a relation betwen the cache limit and the real memory usage.

[ciaobaby]

i tried first to seed only big pieces, then to seed only 256k pieces and got the same results, so pieces size are not the problem ! when i set limit to 1000MB qbt never use more than ~1400MB

Piece size and cache size are nothing directly to do with each other!

The cache is used for the 16kB blocks which is the size of the data block that BitTorrent clients transfer between peers, you absolutely do NOT EVER need huge amounts of memory reserved as cache sizes as cache space is ONLY required for the number of 16k blocks that are being transferred at any particular point in time.

As stated previously, the more memory you reserve for cache space, the more likely it is that you are causing the crashing. 256k - 512k is sufficient memory reserved for cache use for the average use case on a machine where qBT is 'competing' with other applications for resources. If the machine is dedicated to a single BitTorrent client that is only seeding then going up to 8 - 10 MB may be warranted, but there is NEVER a point where reserving a gibibyte plus of memory could be considered 'useful'.

Contrary to the YouTube 'expert' claims It will not significantly 'speed up' download and if you allow local electro-mechanical drives to 'sleep' or 'spin down' you may well be reducing drive life and slowing the client while it waits for the drive to start up. Just as a "by the way", this is pretty much invariably the cause of the uTorrent "disc overload" status messages.
If you use a network drive mapped to a drive letter or  any other 'external' drive for downloading to, you could be reducing the efficiency of the download. While ever your client can keep the drive platters spinning at full speed and the heads in motion, you do reduce wear and tear on the drives and improve 'seek time', so writing 'little and often' is better than coalescing blocks of data, waiting for the drive to respond (several milliseconds are almost an eternity to computers) then over loading the drive queue with large amounts of data.

[nico910]

ciaobaby maybe you are right for certain case, but we have to solve the qbt crashs, there is a software solution to apply somewhere and we need to find it, sledgehammer_999 said "qbt puts an upper limit of 1800MB to cache if it is compiled for 32bit ", now we can say this limit is not efficient, we need to apply a better calculation.
With 4096MB limit qbt use ~5200MB max ! i don't say i will use all this memory, it is just for testing and to prepare the future.
Switeck and i would know what do you think about this enigmatic coeff ? there is a mathematical explanation somewhere...

[ciaobaby]

now we can say this limit is not efficient, we need to apply a better calculation.

Not while you are running a 32 bit application, keeping well, WELL below Gibibyte cache sizes is much, much safer.

With 4096MB limit qbt use ~5200MB max ! i don't say i will use all this memory, it is just for testing and to prepare the future

No it won't and, ... No you are not.

A huge cache is inefficient and wasteful of computer resources, no matter how much memory your machine has available there is no need to 'reserve' large chunks of it for an application that has no real use for it. The case that it might use it is immaterial, it does the client no real use .

Sure, if you are running a dedicated machine with thousands of very active seeding jobs, with tens of thousands of peer connections transferring many hundreds of 16k blocks then a 20 - 50 MiB cache could, only could be qualified as beneficial, because it will probably save a round trip to the drive in order to retrieve the frequently requested blocks (not pieces).

My comments here on cache size are for the average user. Someone running a few dozen tasks and ostensibly downloading more than they are seeding. on a machine that is also in use for other tasks, whatever they may be.

[sledgehammer_999]

I put 1800MB as limit because in 2000GB it would certainly crash. The cache*1.32 is some overhead that isn't documented and maybe it isn't actually an overhead but a mem leak of libtorrent...

[Switeck]

There's a side-issue of qBT doing reads and writes in tiny blocks:
http://qbforums.shiki.hu/index.php/topi ... l#msg15869

What may be happening to cause the excessive memory usage is qBT is reserving ram memory (virtual memory more than likely) in discrete chunks but only using about 60-70% of each chunk.

If qBT's cache stayed persistent much longer and was close to the size of the torrent/s active, then it could slowly fill up if lots of peers are downloading/uploading...acting almost like a ramdrive in theory.

[ciaobaby]

was close to the size of the torrent/s active,

Please ... ... The cache does not need to be anywhere even close to the size of the active jobs, by saying that you are demonstrating that you do not understand what the cache is used for and how it is used.

By giving it far more than it needs you are simply allowing it to leave blocks in the cache that no longer NEED to be cached and remain there until a restart [or a crash]  simply because it has 'lost track' of them. Keeping the cache manageable size means the client ( libtorrent )  has to maintain the cached blocks so is NOT going 'leak' the memory it no longer actually needs.

[sledgehammer_999]

Please stop talking about if the cache NEEDS to be high or low.
The issue here is that it crashes and we need to find out why.

[ciaobaby]

But it does matter, the crux of the issue WILL be found by testing at different cache sizes, and surely it makes more sense to have users who understand the relationship between caching and general client management, if you want a purely technical discussion on the hows and whys, the forum should have a separate area for developers and more 'technical' users rather than overloading the general threads with discussions that the average user reading the forum will not even understand, while bogging down discussions that are technical with interjections from people who just want to see their name 'in print'.

If this is going to be a real 'support' forum then it has to support ALL levels of users not just for development issues. Certainly the problems raised need to be looked at on a more technical level, but either segregate it here or confine technical aspects to the relatively poor discussion facilities GitHub. Maybe even bring 'bug' reporting here (with a custom post form) and only add them to GitHub when it has been determined that it REALLY is a 'bug'. That keeps GitHub as a developer resource and the forum as the user support and educational resource.

Keep support issues, user education and technical discussions separate otherwise ALL the qBT resources become a mish-mash of disorganised snippets of information. I know this is the "Linux way" to have a bunch of 'privateers' doing what they want, how they want, when they want and only  really communicating when they absolutely have to.
Organised doesn't have to be viewed as profanity or blasphemy.

[nico910]

sledgehammer_999 do you have the knowledge to modify libtorrent ? could you test different alloc functions ? arvid sounds busy, i think we have to solve this ourself, i'm ready to test if you want, with cache expiry set to 10min i can reproduce the crash very fast if you don't have windows ? else if you have windows you can test to set cache on 200MB and you will have the same problem...

In worst case, if you can't find a solution, you can limit qbt for Windows to 1200MB and you can be sure qbt will no longer crash, cause 1200*1.4 = 1680. It is very sad but at least no more crash will occur. of course i hope someone has enought C language knowledge to find an explenation. but i doubt...

[sledgehammer_999]

ping me again the weekend (send me an email at sledgehammer999 (at) qbittorrent (dot) org). I'll try to figure out how to use tcmalloc as arvid suggested.

[Switeck]

I use multiple BitTorrent clients on the same computer to test caching...and a ~5 GB ramdrive for the storage location. Since it's not using the network or internet connection, it's limited by the speed of the Bittorrent programs and Windows itself. The ramdrive is considerably faster -- hitting >1 GB/sec even for very fragmented data. I have to be careful to have room in ram for large caches as well, but I've got just enough ram (8 GB) to do that.

My tests of qBT cache crashes happens when qBT reaches or exceeds about 1950 MB of ram usage, which isn't surprising considering that's rather close to the "normal" 2 GB (2048 MB) limit for 32bit apps.