Drive wasn't mounted & I/O Error (not found) brought 100's GBs of torrents to 0%

[AsaRossoff]

Hi all
qBittorrent 3.1.9, Windows 7

SCENARIO
My system crashed/rebooted (while seeding only for about one week runtime).

On restart, I restarted qBittorrent without checking that my external drives with torrent files on them were mounted.  For some reason, one drive (with the most torrents) hadn't mounted properly...

RESULT:

• Hundreds of GBs of torrents got I/O errors ('file not found' or similar wording)
• Those same torrents were all paused and set to 0% completed.

CONCERN:

• I would really like to avoid rechecking all those torrents, as that drive overheats with that kind of usage.  I would have to sit there hitting pause/start thousands of time during rechecking for eons to allow the drive to rest.

QUESTION:

• Is there any way to fix the resume info for those torrents so qBT knows I have the files without a full recheck?

P.S.
I have always planned to backup my %appdata\qBittorrent and %localappdata%\qBittorrent folder regularly to have recourse in situations like... But have not done so. I have a month-old backup of %appdata\qBittorrent but I think it's too old to be helpful even if that is where the problem lies.  If it lies in the .fastresume files under %appdatalocal%, I have no backups.

I might be able to do a binary-safe edit using Notepad++ to those files.  As tedious as 100 or so edits would be, if they would be simple, I think it will be much preferable to a full drive re-check on my system.


Thank you for any advice!
Asa

[sledgehammer_999]

Normally if you didn't touch those torrents at all, exited qbt, mounted the drive and then started qbt again it should work.
Did it work? If not, I might be able to help you editing the .fastresume files.

[AsaRossoff]

Thanks.. I did as you said.  As soon as it errored out, I exited qBittorrent, mounted the drive properly, and restarted... before posting my situation.

I did manually deal with a large number of the smaller of the torrents in the meantime (icepack on HD plus frequent pausing during recheck.  If I just pressed Start without selecting recheck, it attempted a download-over existing files.

I'm down to 26 torrents that still need dealing with.
This will be educational for me for the future, so I'd be glad to know.  I might schedule a task to backup the fastresume files (and/or other important state files).  I checked for volume shadow copies in windows on the off-off chance there were any, but alas no. (I suppose those folders are excluded by default, and it's not a full version-control/backup system anyway).

Yes, I could use your help or advice.
Thanks,
Asa

[sledgehammer_999]

Well, as it seems it's not easy to edit .fastresume via hand.

As it seems you can do this instead.
1. Locate the 26 .torrent files in BT_backup and copy them somewhere safe.
2. Delete them from transfer list without deleting the data
3. Use either File->Import Existing torrent... or just add them like new torrent
4. In their respective dialog windows choose the correct save path and click the option that says do not recheck or start seeding

Do this for one of those torrents and afterwards do a force recheck to be sure that it really points to the correct data.
NOTE: If you have enable "keep incomplete torrent in:" the method above will not work
NOTE2: If while seeding libtorrent finds a piece that has hash mismatch it will schedule a recheck for the whole torrent and download the missing data. This is a countermeasure for the step 4 options.

[AsaRossoff]

OK,
I'll give that a try.

I don't use "keep incomplete torrents in", but I do use the ".!qb" extension for incomplete files, and sometimes (I haven't reported it) qBT can get confused about whether a file should have the extension or not, causing problems locating the files and/or checking them.

I have a lot of issues come up where the import/add-no-hashcheck methods frequently fail for me, in part because I rename folders, and Skip Hashcheck doesn't always work in those cases (I did report that).  It doesn't work in some other cases as well, and I have frequent problems.

Do the torrent files qBT caches have the same data as the original torrents, or have they been modified?

I'm off for now and will give it a go tomorrow.

Thanks
Asa

[Nemo]

A small suggestion: I wouldn't use .!qB extension for incomplete files if you are a heavy downloader (like me). Not only with qBittorrent but also with other clients like uTorrent in the past I experienced several I/O errors cause of this feature. Somehow it can cause some files to be corrupted, or can't find the files anymore and starts redownloading again from zero while the files are there. Im leaving it unchecked since 6/7 months and haven't got any errors since then, maybe you can try that out too for yourself .

[quote="AsaRossoff"]
but I do use the ".!qb" extension for incomplete files, and sometimes (I haven't reported it) qBT can get confused about whether a file should have the extension or not, causing problems locating the files and/or checking them.
[/quote]

Exactly this.

[sledgehammer_999]

but I do use the ".!qb" extension for incomplete files

Then probably the method I described will not work.

in part because I rename folders

The same will happen, unless you add the torrent and rename the files/folders in the dialog before you close it.

Do the torrent files qBT caches have the same data as the original torrents, or have they been modified?

IIRC it saves the original .torrents without modifications. The only modification(propably) is with merged trackers.

[ciaobaby]

Or if you are using .!qb you can remove the added extension with the [url=http://"http://www.bulkrenameutility.co.uk/Main_Intro.php"]Bulk Rename Utility[/url] before you start trying to get them running again

[AsaRossoff]

Thanks both of you..
I was referring to issues that I have that I described in #1562, #1556, and #1586.

The .!qb doesn't usually cause problems, but does occasionally in a couple of different scenarios (I haven't reported yet due to insufficient data).  I can disable that option during my import, and since all of my torrents are complete, there are (or should be) no files with the extension right now.

In any case, thanks for the attention... I guess I can connect the drive to a USB 2.0 interface to slow it down as well; that may help with my hardware overheating issue for any large torrents that I can't prevent rechecking of.  Ultimately I should probably collect on the drive's lifetime warranty.  I wanted to give them time to develop stable hardware first as it was cutting edge when I bought it maybe 2 years ago (USB3 2 TB 2.5" HD).

I have a few new, more aggressive ideas to try to mitigate the bugtracker issues I mentioned should they come up (I do remember specifically not working around the issues with at least one of the 26 torrents I now have left, not long ago -- a large one).

Thanks again... I'll report back with me experience.
Asa

[sledgehammer_999]

[quote="AsaRossoff"]
The .!qb doesn't usually cause problems, but does occasionally in a couple of different scenarios (I haven't reported yet due to insufficient data).  I can disable that option during my import, and since all of my torrents are complete, there are (or should be) no files with the extension right now.[/quote]

I don't know what the qbt code actually does because I haven't looked at that portion. But when doing a recheck of unknown torrents (if they are complete or not) it might actually look for the files with the .!qb extension, not find them and then proceed to download them.

[ciaobaby]

Whenever I have needed to reload partial tasks from one machine to another, files that have the .!qb extension (on Windows), the incomplete files in the payload are not checked and will start downloading again until the extra extension is removed and the job rechecked.

[AsaRossoff]

I've not gotten to this yet,
But yeah, a check of a new torrent looks for the filename without .!qb... so it shouldn't be an issue with your suggestions.  For existing torrents, it seems to recall which files are .!qb and which are not (at least as qBittorrent intended to leave the files).  There are some odd cases where problems crop up for existing torrents, including when completed file renames oddly fail.. leaving qBT confused, and files not named as I'd like them.

When nothing goes wrong, it seems to act OK, although it is not hardy when something unexpected happens along the way, I think.  Anyway, I will eventually report on some of that when I have some clear cases/time.  One way things can go wrong seems to be some inconsistency in determining if a filepath is MAX_PATH length or not when this feature is used.  The download in that case goes great, files have the .!qb extension possibly just ever MAX_PATH length with the extension, but then they won't be renamed.  I think they even seed fine, but can't be rechecked.. or something.. and of course I want the files named properly for access.

I've found the feature about 99% stable, with some issues in unusual cases... I would prefer qBT to find either filename with or without .!qb that exists and latch onto it, rather than being convinced the file has one name or the other, and occasionally being wrong.  That alone would automatically recover from most problem cases.

Wish you a great day..
Thanks for qBittorrent and all your efforts!
Asa

[ciaobaby]

I would prefer qBT to find either filename with or without .!qb that exists and latch onto it, rather than being convinced the file has one name or the other, and occasionally being wrong.

The reason why most clients do not recheck files with an extension that the client has itself added is because they use the file name list from the meta info and to check for incomplete files would require additional logic in the code to check for the meta info name and if that is not found, perform the same checks with the added extension. This is obviously going to increase the time taken to perform the recheck and increase the potential for error.

[AsaRossoff]

It's best if qBT doesn't have any situations where it loses track of what extension it left on the file to  begin with.  In any case, the time needed to check for the existence of up to two files is much less than the time needed to recheck even one torrent, and uTorrent never had any issues with it's temporary file extension that I experienced.

Also, I think uTorrent doesn't force rechecks (or didn't).  It paused a torrent without resetting its state, suggesting a recheck if I recall, if there was an issue.  This is from a long ago memory, as I have not used uTorrent in a long time.  Any temporary issue was a non-issue in that case.  I never once had to unnecessarily recheck files with uTorrent.

Unnecessary rechecking of files is at least inconvenient, and I think should be avoided for the best user experience.


On a side note, I just installed the 3.1.9.1 update after an apparently clean exit in 3.1.9 (File> Exit, no error), I just had 152 additional torrents fail fast-resume on 3.1.9.1 startup

So, my HD is in much more hot trouble now.

The inexplicable error for all 152 was:
- Reason: {torrrent name} fast resume rejected: missing or invalid 'file sizes' entry
- Fast resume data was rejected for torrent {torrent named}, checking again...

A minority of torrents resumed properly.

edit: I considered backing up data files before upgrading, but thought a 3.1.9 > 3.1.9.1 update sounded safe... didn't backup...

edit2: After writing this reply, I exited 3.1.9.1 and downgraded to 3.1.9, on startup the 152 was reduced to 69, rechecks.  Many of the torrents I saw in the torrent list go to 100% state without rechecking during fastresume.  I assume the others were corrupted during the update, and some may have finished rechecking (and either been OK to v3.1.9 and 3.1.9.1 or not, not sure)... but in any case, the downgrade did improve the situation.

edit3: after the downgrade to 3.1.9, the smaller number of failed fast-resumes had a combination of errors -- they weren't all "files sizes" errors.  Most were "- Reason: {torrent name} fast resume rejected: mismatching file timestamp" errors.  A smaller number were file size errors.  The file size error torrents were all reported last in the Execution log (at the top), not sure what the significance of that is.  But they were in two blocks. of error messages if you follow me.

I'm taking the approach of rechecking all torrents after all.  With a USB 2.0 connection to slow the drive down, an ice pack, and plenty of patience.  There is one torrent that won't recheck.  It says "Checking" indefinitely while staying at 0%.  It was a partially downloaded torrent (not all files selected).  I will probably have to re-add that one.

I will also start backing up qBT's data files

Peace.
Asa

[sledgehammer_999]

Damn it, this seems like a problem of libtorrent itself. I'll update a new build in a few hours(going to sleep now). If you don't use https webui you should be ok for the moment.
I deleted the installer from sf.net so noone would autoupdate anymore.