Drive wasn't mounted & I/O Error (not found) brought 100's GBs of torrents to 0%

[sledgehammer_999]

btw do you have any small torrent(=<300MB) that exhibited this so I can test locally? link please.

[AsaRossoff]

I have the webui disabled fwiw.  Checked the options, and https is disabled on that options page to....
---
Also, I managed to recheck all my torrents, including the prior remaining 26, and even the one that I said wouldn't recheck.  It really seemed that way with multiple attempts in the last couple days, but I finally rechecked it last, just now, after everything else, and it did recheck.  It took an unexpectedly long time to "get to" the first selected file (the files selected were the last few)... maybe this is normal,... it may generate hashes of all the nonexistent data, too; although it seems like that would be unnecessary. and even if desired, could be optimized.  It rechecked 100%.

I think I see what happened: it must have completely hash-checked the preceding file to the selected files, which possibly through a prior file move or something, I notice is no longer a sparse file, either.  It's debatable if this is desired behavior or not, as it is possible to download data before deselecting files, and thus there may have been data there, but on the other hand, most likely there would be little or no data for pieces not selected for download.. there was only a partial piece that actually had any benefit in reading from the end of that file.  As an aside on hashing optimizations, not sure if libtorrent has optimizations to detect the structure of sparse files, which is possible to do... although it wouldn't have helped me as this file had become non-sparse at some point.  Reading zero-filled parts of sparse files still goes quickly of course, as real reads aren't involved, but repeatedly hashing all those zeroes would still be unnecessary work.


My qBT is at peace ^_^

I really do appreciate all the work that goes into qBittorrent, all the contributors, work on supporting it, Christopher Dumez's work in creating it, Sledgehammer999 -- for taking charge of the project and doing the bulk of the work now...  I wish I had the skill set to contribute code-wise, but hopefully my feedback has been and is some benefit.  I know a bit about programming and the Windows API, but my experience is almost entirely limited to working in Visual Basic and its predecessors (as well as a couple of now dead languages, a longg time ago).  If I do study up I will be glad to contribute code as time permits.
Thanks!
Asa

[AsaRossoff]

[quote="sledgehammer_999"]
btw do you have any small torrent(=<300MB) that exhibited this so I can test locally? link please.
[/quote]

Sorry, I didn't catch your replies earlier.

I did not retain the log from the 3.1.9.1. startup,  but I can point to ones that remained a problem after the downgrade for me.  I can also backup my localappdata and appdata folders and reinstall 3.1.9.1 to repeat the experience from a pre-existing known clean set of checked torrents.

Please verify if that is safe for me to do if afterwards I restore my backup of %localappdata%\qBittorrent  and %appdata%\qBittorrent if you want me to do that.


In the meantime I will email you a few small torrent links that still errored after the downgrade, to [email protected].

[sledgehammer_999]

I don't know the exact folder path on your Win ver
But there are 2 "qBittorrent" folders you need to backup. One that contains the "BT_backup" folder and the other one contains .ini files (qbittorrent-resume.ini).
I want a torrent that shows 0% in 3.1.9.1 and 100% in 3.1.9 and is smaller or equal to 300MB. Yes send the .torrent file at that email address.
I just hope you kept a copy of 3.1.9.1 locally because I deleted it from sf.net.

[ciaobaby]

It's best if qBT doesn't have any situations where it loses track of what extension it left on the file to  begin with.

It would, however that would mean having to rewrite the entries in the meta data (.torrent) file, which would then leave the client unable to respond safely to meta data requests from peers loading the job from magnet links.

Also, I think uTorrent doesn't force rechecks (or didn't).  It paused a torrent without resetting its state, suggesting a recheck if I recall,

It does, and always has done, particularly when recovering from a 'dirty' shutdown, a problem that was far worse with 3.3.n versions

NO forced recheck is 'unnecessary', it means that the BitTorrent client  has determined that the state of the payload may be different to the 'last known good' state, so the integrity needs to be verified. The way this is determined is by comparing the current checksum of the "resume data" file or files with the checksum of the file when it was last saved, any differences mean the file may have been changed so the information such as what pieces have been 'fetched' may not match what is in the payload. Your computer switching in or out of "Daylight Saving Time" or a NTP server synch adjustment coinciding with a file checksum being created can be the catalyst for the behaviour. You have to remember that in the pure logic of computer processing there are no "grey areas", so it is either yes or no.

And if the computer says no, ...

[sledgehammer_999]

We can save arbitrary settings per torrent in our qbittorrent-resume.ini file and later in the .fastresume files(for when qbittorrent-resume.ini will be deprecated).

[offtopic]The idea behind fixing the "keep incomplete folders in:" and qbt extension problem is to have this extra code logic. Load the paths from the .torrent. Check them against the final disk. If they exist assume the torrent to be there and act accordingly. If they don't exist there act as if the torrent is incomplete and search the incomplete folder. If custom extension is enabled, check in the folder(temp or final) if the files exist with custom extensions or not. Then act accordingly. This involves setting up libtorrent to look for specific files(renamed with the extension) in specific folders(final or temp)[/offtopic]

[AsaRossoff]

I didn't mean to sidetrack things with the .!qb comments.

Regarding the 3.1.9.1 update, I've sent an email with what I could and some more details of what occurred during my tests as I did so.

[Starshine]

This has happened to me A LOT.
Imho this is the single most annoying thing with qbittorrent.

Sometimes, if it can't find the files in the locations where it thinks they should be, it starts RE-DOWNLOADING it.
I've accidentally downloaded tens of GB because of this behaviour.

It's gutting on trackers that have strict ratios and are hard to seed at --- where every GB is worth its weight in gold.

I almost stopped using qbittorrent because of this.
Lately it got better though, as of release 3.0.8 or so.

My workaround for preventing re-downloading if a disk is temporarily offline, or qbittorrent loses some setting, is to set the DEFAULT (which I never use) directory to a drive and folder that doesn't exist. So if qbittorrent takes it upon itself to start re-downloading something or loses path to download, then it will trip and have to stop, because the drive doesn't exist. It can't create a folder on a drive that doesn't exist.

But it's a clumsy workaround! It would be better if it simply stopped torrents that have already been downloaded, and let them sit until the user sees it and manually decides what to do. 

[Nili]

[quote="Starshine"]
This has happened to me A LOT.
Imho this is the single most annoying thing with qbittorrent.

Sometimes, if it can't find the files in the locations where it thinks they should be, it starts RE-DOWNLOADING it.
I've accidentally downloaded tens of GB because of this behaviour.

It's gutting on trackers that have strict ratios and are hard to seed at --- where every GB is worth its weight in gold.

I almost stopped using qbittorrent because of this.
Lately it got better though, as of release 3.0.8 or so.

My workaround for preventing re-downloading if a disk is temporarily offline, or qbittorrent loses some setting, is to set the DEFAULT (which I never use) directory to a drive and folder that doesn't exist. So if qbittorrent takes it upon itself to start re-downloading something or loses path to download, then it will trip and have to stop, because the drive doesn't exist. It can't create a folder on a drive that doesn't exist.

But it's a clumsy workaround! It would be better if it simply stopped torrents that have already been downloaded, and let them sit until the user sees it and manually decides what to do.
[/quote]

Most powerful Torrent Application that use libtorrent suffers from the same phenomenon. I mean Deluge, qBittorrent, rTorrent i have tested all of them since 2 years, they all have a common denominator. I want to congratulate qBittorrent devs for fix report to tracker when pausing, this has been a early problem libtorrent that Deluge and rTorrent hasn't fixed so far. Deluge dev Cas also said qBt have found a way to fix this libtorrent bug. But can't find the thread now to bring here. Congratulation, i remember 1 year ago qBt pausing 500 torrents and resume them shown on my profile tracker 1000 files pausing and resuming again 1500 and so on... this is great fix of pausing and anounce.

well..

If the driver is not mounted first not step with torrent, be sure to close correctly qBittorrent because if something closes worst ie crash, shutdown system without closing torrapplication first, accidentally daemon closed, unmount driver while torrent is open etc... will bring a lot of worse stuff for seeders settings, Only people who care to seed (and usually in private tracker) encounter these concerns. Logically imo the problem lie most of time on libtorrent because one of the programs I mentioned will react differently (if it would be their problem), yes since all three use the same component "libtorrent" Here's something wrong at least.

I'm sorry to say but we're not secure to seed 24/7 with those torrent applications, most of time we are us the fault (we make mistakes, poweroff pc without closing properly torrent application, powercut, not mounting drivers first, system stuck etc...) but also torrent devs should think how to avoid these causes How can become a kind of backup settings that in each type of problem to be saved last data settings etc... uTorrent may be disgusting but has arranged these things better, obviously from long experience though. I systematically test the three torr app mentioned above and will use them when I see they've created strong stability together with libtorrent.

Well my 2cents'... and sorry for bad English wrote

Regards,
Nili

[sledgehammer_999]

[quote="Nili"]
Most powerful Torrent Application that use libtorrent suffers from the same phenomenon. I mean Deluge, qBittorrent, rTorrent i have tested all of them since 2 years, they all have a common denominator. I want to congratulate qBittorrent devs for fix report to tracker when pausing, this has been a early problem libtorrent that Deluge and rTorrent hasn't fixed so far. Deluge dev Cas also said qBt have found a way to fix this libtorrent bug. But can't find the thread now to bring here. Congratulation, i remember 1 year ago qBt pausing 500 torrents and resume them shown on my profile tracker 1000 files pausing and resuming again 1500 and so on... this is great fix of pausing and anounce.
[/quote]

rtorrent uses libTorrent(rakshasa)
qbittorrent and deluge use libtorrent (rasterbar)

If you find the Deluge dev and/or thread you can tell him that I'll gladly guide him on what the qbt code does for pause/download. AFAIK he is a python guy and we use C++ so it is understandable if he can't figure out what we do different.

Also, I should mark each torrent that has a fastresume failure as errored out and somehow don't perform any operation on it until the user either restarts the client or explicitly tells me to force recheck. (third option will be in the future to have the code retest the fastresume file without restarting the client)

[Nili]

Very helpful sledgehammer_999
I also watch with interest your work and qBt in general and follows with pleasure... I am sure that in the future qBt will soon be superior.

Thank you for the kin words regarding Deluge concern.

Nili

[Starshine]

$%^&**&(*(&*(&*))

This happened again!
It took 5 hours, not joking, to re-set the torrents to their correct path.
There wasn't even a crash, I have no idea what happened.

Because the bug hasn't been seen for a while, I had stopped being vigilant and back up the three files containing settings.
So I was 3 months behind.

I'm doing a weekly backup from now on but this bug MUST be fixed, it's the single biggest problem with QBT, as it takes unbelievably long to fix.

I had a delta of 350 torrents between my last backup, and today. It took 5 hours to figure out which one should be labelled with which tracker, and where among my hard drives I had saved it.

5 valuable hours of weekend time, and this is probably the fourth or fifth time something similar happens.

[sledgehammer_999]

This reply will get you even more pissed but it isn't my intention to make you pissed.

5 valuable hours of weekend time, and this is probably the fourth or fifth time something similar happens.

Quoting from help->about->license

NO WARRANTY
11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

And what I mean to say is that is a known issue. Secondly this is a free program and noboby gets paid for making it. Thirdly, this is an open-source program and anybody is welcome to fix things. I am one man and there are limited things I can do in my free time.

So until this is fixed you:
a) devise a script that will automatically backup those files (probably diff them)
b) close qbt without touching any torrent, then attach the drive and then start qbt. It should recognize most if not all torrents then.
c) go to a paid solution if you REALLY REALLY need to have that kind of stability/maintenance.

[Starshine]

This reply will get you even more pissed but it isn't my intention to make you pissed.

5 valuable hours of weekend time, and this is probably the fourth or fifth time something similar happens.

Quoting from help->about->license

NO WARRANTY
11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

And what I mean to say is that is a known issue. Secondly this is a free program and noboby gets paid for making it. Thirdly, this is an open-source program and anybody is welcome to fix things. I am one man and there are limited things I can do in my free time.

So until this is fixed you:
a) devise a script that will automatically backup those files (probably diff them)
b) close qbt without touching any torrent, then attach the drive and then start qbt. It should recognize most if not all torrents then.
c) go to a paid solution if you REALLY REALLY need to have that kind of stability/maintenance.

Fair enough! I was just really irritated but  you are right that I was ranting on as if I had paid for it! 
I had thought it was fixed, so I had let my guard down, and stopped backing up the folder.
I'll write a script to automate it.

[Starshine]

As suggested by Sledgehammer above. Below is a backup script for QBT settings.

Recommended for people who seed large numbers of torrents, used multiple labels and multiple save paths to different disks.

If your paths, labels etc are messed up, just replace the corrupted files with your latest good backup. You need to change the paths in the script to the paths that are applicable to your machine:

The script first backs up the last backup, then creates a new. So you always have the second on next to second last configuration.

Copy and paste the below into a text file and save with the extension .bat
Put the bat file in the Startup folder in the Start Menu, or set up a task to run it daily or weekly.

Code: Select all

@echo off
:: variables
set drive=D:\Backup\qBittorrent
set backupcmd=xcopy /s /c /d /e /h /i /r /y
set drive2=D:\Backup\qbittorrent_PREVIOUS

echo ### Backing up qBittorrent Backup
%backupcmd% "D:\Backup\qBittorrent" "%drive2%\qbittorrent_OLD"

echo ### Backing up qBittorrent Root
%backupcmd% "C:\Users\USERNAME\Appdata\qbittorrent" "%drive%\qbittorrent_root"

echo ### Backing up qBittorrent Local
%backupcmd% "C:\Users\USERNAME\Appdata\Local\qbittorrent" "%drive%\qbittorrent_local"

echo ### Backing up qBittorrent Roaming
%backupcmd% "C:\Users\USERNAME\Appdata\Roaming\qbittorrent" "%drive%\qbittorrent_roaming"

echo Backup Complete!
@pause