Page 1 of 1

downloading a 400g torrent where I have some files on hdd

Posted: Sun Jul 03, 2016 5:40 pm
by immo2012
is there any way I can scann a dir for a running torrent to add full or maybe part of files

in this case I have about 100g complete files on external dir

Re: downloading a 400g torrent where I have some files on hdd

Posted: Sun Jul 03, 2016 8:44 pm
by Switeck
If you can move existing files that perfectly match files in the torrent into the correct folder AND correct filenames expected by the 400 GB torrent file, you can then do a force recheck in qBT to get it to identify the files/parts of the torrent it already has.
This can be a LOT of work if more than a handful of files are involved!

Re: downloading a 400g torrent where I have some files on hdd

Posted: Sun Jul 03, 2016 10:59 pm
by immo2012
thats my point. Doing that by hand usually makes no sense as it takes too much time and who knows if its working then.

So my point is why not add a function where you can select a dir where the files there are crosschecked if they can "speedup" the torrent transfer

in mame this happens quite  often where you download a full set of chds where you have already 80% on your disk

maybe add a function which checks for existing files and then skip the file on the torrent?

e.g. the mame software chd list ist 1834GB and I want only download files which are new or updated in that  torrent

Re: downloading a 400g torrent where I have some files on hdd

Posted: Mon Jul 04, 2016 5:41 pm
by Switeck
It wouldn't work very well because the BitTorrent protocol treats multi-file torrents as a continuous series of bytes and hashes by piece only, not by file.
Files could be offset inside the multi-file torrent with no reference where a piece starts within them...so no piece hashes would match except perhaps with the first file (minus its last partial piece) and then only if the piece size were the same.
Even rearranging the file order means creating a new torrent that won't match the old one hash-wise, despite containing the exact same files!

Multiple small files could be part of a single piece, making this even more impossible.

Re: downloading a 400g torrent where I have some files on hdd

Posted: Mon Jul 04, 2016 5:50 pm
by immo2012
I am not a torrent expert but in qb I can deselct files within a torrent which will be skipped then
so why not use an existing dir to automatic skip files which are existing with same byte size
by hand this is possible but a pain alot of work

Re: downloading a 400g torrent where I have some files on hdd

Posted: Mon Jul 04, 2016 7:25 pm
by Switeck
Even when you skip files, BitTorrent protocol often forces you to download any parts of those files that are in a piece shared with another file/s.
So the end of the file you want may share a piece with the beginning of a file you don't want...qBitTorrent will be forced to download that whole piece and as a result get at least part of the file you don't want.
If you don't keep these end pieces (by deleting the partial files you didn't want), you cannot share all of the files you did want with others on the same torrent.
It's a mess, and all due to how BitTorrent works.

Re: downloading a 400g torrent where I have some files on hdd

Posted: Wed Jul 06, 2016 12:35 am
by immo2012
So I installed qb on my notebook , downloaded the torrent , copied all 220g of "old" files

now its doing the recheck where my main pc is still downloading the torrent (from scratch)

I am surprised hoch much after checking looks valid

keep you updated how it works out

recheck speed is limited to 40-50mb/s?

Re: downloading a 400g torrent where I have some files on hdd

Posted: Wed Jul 06, 2016 6:11 am
by Switeck
[quote="immo2012"]recheck speed is limited to 40-50mb/s?[/quote]Yes, partially due to being limited to 1 (virtual?) core of a cpu and partially so other programs are not severely impacted. And that's just "simple" hashing, not trying to match every piece size or starting location that would be needed to auto-detect files and match them to unfinished torrents.
With better programming and more aggressive hashing, the speed might be increased to 100+ MB/sec...limited of course by how fast the drive and your cpu are.

Even when I was using a ramdrive, qBitTorrent couldn't seem to download faster than about 130-150 MB/sec.
I was "downloading" from the ramdrive TO the same ramdrive using 2 different BitTorrent programs at once to see when bottlenecks showed up.
From what I could tell, the cpu speed (or rather the individual virtual core speed) was the main limiting factor -- the ramdrive itself is capable of speeds up to 40 times faster! (~6 GB/sec!)

Re: downloading a 400g torrent where I have some files on hdd

Posted: Wed Jul 06, 2016 6:16 am
by immo2012
here now my update

local peer detection was not working so I had to manual add the source with local ip

despite both pc are on a giganot network speed was super slow max 50mbit

both torrents was about 400gb where one was 15% and other was 60% so there was plenty of data possible to local transfer

maybe I need a special setting dunno about that but local transfer looks not working in gigabit network ages

Re: downloading a 400g torrent where I have some files on hdd

Posted: Wed Jul 06, 2016 1:31 pm
by Switeck
"max 50 mbit"

Do you mean 50 MegaBYTES/second here? Or ~6 MegaBYTES/second (50 mbit/sec)?
Even gigabit/sec should only reach about 100-120 MegaBYTES/second under perfect conditions.

"maybe I need a special setting"

I had to disable uTP in my ramdrive tests, or speeds maxed around 30-50 MegaBYTES/second.
Might also help if your network is using jumbo frames, especially 9000 byte packet size or larger.

Re: downloading a 400g torrent where I have some files on hdd

Posted: Tue Jul 12, 2016 1:15 pm
by gspbeetle
Bittorrent in theory should not care it is downloaded or simply copied, qB can detect that change and move to re-check automatically.
I am doing this automatically since I am also running deluge in parallel (for seeding only), just in case any of the client crashed and interrupting the seeding to the hive.
But for some reason, the 32-bit version of qB will most likely crash, only the 64-bit version will survive this massive re-checks.

I am not sure is this a feature as well (most likely not intended), but when I migrate qB progress to another machine by copying everything in the BT_backup folder only (%Userprofile%\AppData\Local\qBittorrent\BT_backup), it will not recheck the files and go seeding immediately; given that the file path is the same of cause.
But even the file path doesn't match, it will just simply show as "Errored" without any issues.

Re: downloading a 400g torrent where I have some files on hdd

Posted: Mon Jul 18, 2016 9:42 am
by Switeck
gspbeetle, that is a feature.
qBT checks when it starts up that files exist where it expects them and maybe even checks that their filesize matches, but it does not bother to recheck their contents...so it would happily seed garbage if the files were corrupted by copying/moving.

Rechecking files is limited to <100 MB/sec with little other activity going on the same computer.
To constantly check files would require nearly infinite cpu, ram, and drive speeds, so a lot of compromises are made to be workable.