Best behavior for when torrent files are offline/inaccessible - per situation

[AsaRossoff]

[quote="AsaRossoff"]
...I am interested in a discussion of best behavior for paused torrents as it relates to this scenario of intentional removal of file access; and even the scenario of a drive going offline due to an accidental issue, such as network error, OS issues (perhaps file handles can't be allocated due to something), temporary bus failure (e.g. USB/1394 as most likely), temporary drive failure (e.g. overheating); and even accidental user action such as moving or renaming of files (which they might remedy by moving them back or using Set Location or renames in qBT afterward).

All thoughts welcome.  What do people think should happen?  What about my scenario or the others I mentioned above?  What counter-examples do you have to argue for the current or similar behavior, etc.?

Thanks, always
[/quote]

You can click that quote for my full set of ideas so far on what behaviors might be desirable, and read the remainder of this post and the replies for my initial post describing the specific problem I encountered that raised the issue for me.

It would be great if we can brainstorm together.

---

Initial Subject: Files Sizes Mismatch error; torrents to 0% on _paused_ torrents in unavail. loc.

---

I squeezed most of what needs to be said into the subject heading.
qBT 3.1.9.2, Windows 7.

What transpired:
I had about 150 torrents on an external drive that I needed to disconnect from the computer temporarily (for a couple days).
About 100 out of the 150 or so torrents were seeding.
The remaining 50 or so were long paused, probably never active during the qBT session.
I paused the ~100 seeding torrents.
I waited for all file handles to that drive to be closed.
I used "safely remove hardware" to unmount the external drive (which succeeded once all file handles were closed).
I don't know how much time transpired between the removal of the drive and qBT generating errors.. I did not notice the errors until I went to reconnect the drive a couple days later...
After remounting the drive, I was going to resume seeding torrents...
Without looking closely, I resumed some of them, but then I noticed that they went into stalled or downloading status!
I paused those torrents again.
I found that all the torrents on that drive, whether I had previously been seeding them or not, and without regard to whether I had just tried to resume them or not, were marked as 0% complete.
I checked the log and found that there were log entries from a couple days ago) indicating "File sizes mismatch for torrent ..., pausing it." (although they were all already in a paused state for all the torrents on that drive.  All the errors were recorded in a span of about 30 seconds, in short order after one another.

Finally, just now, I did a clean shutdown of qBT and a restart.  No change.  The torrents still all want to download if I unpause them.  All still show 0%.


Now I will begin force-rechecking them all...

Questions:
Is this known behavior?
Why did qBT look for the paused torrents' files?
More importantly, why did it consider it a critical file integrity error when the torrents were paused for the files not to be there?

Sounds similar to Torrent recheck forced after error I/O (even for paused torrent) #1471 - although that bug describes a different workflow.

If desired, I will report this on the bugtracker.

Thanks!

---

edit: changed purpose of post to discuss qBT functionality ideas, added intro to my initial post.

[ciaobaby]

If you remove the drive while qBitTorrent is RUNNING it WILL go into 'failure mode'

http://qbforums.shiki.hu/index.php/topi ... l#msg11794

qBT( libtorrent ) will continue to check that the payload is present because pause means "Temporarily cease activity, but remain ready to start at any instant".

[AsaRossoff]

Thanks ciaobaby,
Your response and that thread was elucidating.  Essentially they seem to answer my first question, "Is this known behavior?": YES.

After reading that thread though, I'm not sure that the fact, as you state it, that
[quote="ciaobaby"]
If you remove the drive while qBitTorrent is RUNNING it WILL go into 'failure mode'
[/quote]is fully appreciated as a problem for users -- and I would think -- unexpected behavior, as it was for me.
It was stated in that thread that "pause" means "stop" by a trusted member of the community.

[quote="ciaobaby"]
qBT( libtorrent ) will continue to check that the payload is present because pause means "Temporarily cease activity, but remain ready to start at any instant".
[/quote]
Although it might call for a libtorrent-rasterbar change to maintain that ready state for all torrents yet not even bother checking the files on disk, I would argue that it is at best a waste of time to check for the existence of or other properties of files that will not be accessed until a peer is uploading or downloading to them -- particularly when torrents are in a paused state.  Unpaused, there might be value in regularly checking, but even then I believe that the error generated should be recoverable simply by restoration of existence of files and at most the additional step of hitting the start button (see my final comments in this post).  The current design's greatest strength might lie in maintaining network connections or information, and that could be done without file system access to the torrent content.  There might be a scalability problem with the current design anyway, for users who maintain thousands of torrents in the torrent list, only intending to occasionally seed a subset of them.

I believe from the bug report #1471 that I linked to, that sledgehammer_999 suggested (or I interpreted from a test suggstion he made) that one way (possibly the only way) to avoid having qBittorent not mangle download data for temporarily unavailable files is to only make them unavailable when qBittorrent is not running, then close and restart qBittorrent again when you make them available.

I believe that qBT users would best be served if qBT permitted the unavailability of paused torrents' files in a non-fatal way during runtime.  In fact, it would be nice if even unexpected unavailability of files that were not paused had a recovery mechanism other than force rechecking... e.g. maintain the fastresume data that was valid at last file access time and attempt that normal level of validation when the files are available again (either automatically if polled, or through user interaction -- probably hitting the start button).

If the current behavior is considered intentional design, and not just a calculated, if in my view harmful, side-effect, this would be a feature request, otherwise if the effect is undesired by our much appreciated maintainer and developer who bears most all of the maintenance workload, he could consider it a bug.

I will leave this thread open to discussion for best functionality ideas.

Ciao or whoever can:  maybe this thread should be moved to the Suggestions board.
(If Sledge prefers this be moved to the tracker, I can post there.)

... as it sounds non-platform specific and now I am interested in a discussion of best behavior for paused torrents as it relates to this scenario of intentional removal of file access; and even the scenario of a drive going offline due to an accidental issue, such as network error, OS issues (perhaps file handles can't be allocated due to something), temporary bus failure (e.g. USB/1394 as most likely), temporary drive failure (e.g. overheating); and even accidental user action such as moving or renaming of files (which they might remedy by moving them back or using Set Location or renames in qBT afterward).

All thoughts welcome.  What do people think should happen?  What about my scenario or the others I mentioned above?  What counter-examples do you have to argue for the current or similar behavior, etc.?

Thanks, always

[Nemo]

Ciao or whoever can: maybe this thread should be moved to the Suggestions board.
(If Sledge prefers this be moved to the tracker, I can post there.)

There you go .

[ciaobaby]

The problem is the potential for disaster when a drive is removed and subsequently reconnected:

For Windows this may mean that the drive letter could change, dependant upon the order any other drives being connected.

For Linux or Mac same for Linux or Mac, what if the drive mounts as sde1 instead of sdd1

What if this is a NTFS format and you changed the drive label externally and your system is set to mount by label.

There are so many ways that reattaching a drive could 'break' things, it means the ONLY truly safe way is what happens now, the jobs pause and are flagged as 'missing'. SURELY the end user HAS to take the responsibility for their own actions, as there are only so many safeguards that can be envisaged by developers.

If you decide to tip a fresh cup of tea (no milk) onto your genitalia it is NOT up to the cup to tell you that it's probably not a good idea, likewise the qBT client cannot tell the difference between a drive being physically removed, unplugged from it's PSU, some type of hardware failure or somebody taking wire cutters to the USB lead. The failure point of the payload being missing is the ONLY thing the client is aware of so the least risk [to the data] option is taken.

[Nili]

[quote="ciaobaby"]
For Linux or Mac same for Linux or Mac, what if the drive mounts as sde1 instead of sdd1
What if this is a NTFS format and you changed the drive label externally and your system is set to mount by label.
[/quote]

On Linux driver mounted differently by UUID

Code: Select all

nili@crunchbang:~$ blkid
/dev/sdb1: LABEL="dat2" UUID="e1e11df6-91c5-464e-a295-b48fa1ba274b" TYPE="ext4" 
/dev/sda5: UUID="738d90c1-6764-4499-92fe-b6ff476413e5" TYPE="swap" 
/dev/sda1: UUID="6823b33f-db7f-45e4-b3ee-4c00cbf26532" TYPE="ext4" 
/dev/sdc1: LABEL="I3" UUID="CA14AB7114AB5F67" TYPE="ntfs" 
/dev/sdd1: LABEL="VERBATIM" TYPE="vfat" 

Does not matter is sde sde1 or sda sda1 etc etc... UUID never change my udevil or udisk, udisks2 "disks mounting things" mount like UUID and gives a name free in list like sde1, sdc1, sdd1 etc... Torrent client that i use recognizes paths on the basis of UUID doesn't generates errors for path names because UUID is there to stay forever attached on system.

Linux based systems, Is not the same as Windows,If the path is not D:\ or F:\ generates error if the names does not match. I think qBittorrent or accurately libtorrent rasterbar have issue as stated above. But not only, there are some similar discussions of this character here or here a few months ago. This problem of 0% or removing HDD disk before or after is here since long time, Normal user that download no seeding can't notice, but user that seed over 1000 torrents undergoing numerous difficulties especially when you have a few external hdd for seed. I agree with @OP he is completely correct in what has been written, I intervened just that I saw a thing on my mind sound right.

The tests that I made earlier under Debian 7.6 based system, qBittorrent 1.97 /1.98 " installed v1.99 to tell the truth but didn't checked for test, I have also found such issue a month ago. libtorrent rasterbar team should be put on knowledge for this behavior.

Regards,
Nili

[ciaobaby]

And if the user does not mount by UUID? What you do is not what every body does, qBitTorrent has to cover every unknown combination of anything and everything.

What exactly would you expect to happen if the payload disappears? No BT client can seed from a payload that has simply gone away and what would be the use of downloading pieces that have no way to be placed in the correct location in the payload. You seem to be expecting the client to be almost clairvoyant and anticipate that the drive is about drop out

[AsaRossoff]

[quote="Nili"]
This problem of 0% or removing HDD disk before or after is here since long time, Normal user that download no seeding can't notice, but user that seed over 1000 torrents undergoing numerous difficulties especially when you have a few external hdd for seed. I agree with @OP he is completely correct in what has been written, I intervened just that I saw a thing on my mind sound right.
[/quote][quote="Nili"]
I have also found such issue a month ago.
[/quote]

Nili, Thanks for confirming that it is an issue for you, and your (unquoted) more technical comments.

I don't know who normal users are, but perhaps you are right that a majority of torrent users do not retain or seed many torrents, and are thus less likely to encounter this issue.  I still suspect that a significant number of users do a lot of seeding and retention of torrents.

What really surprised me was that qBT went into "failure mode" when I very carefully ensured all torrents were paused and all file handles were closed before making the files inaccessible to qBT.  But as I described, files can also become temporarily unavailable for many other reasons, and they are not all within user control.  Which got me to thinking, we might not want to limit ourselves to elegantly handling a temporary file access problem to only paused torrents anyway.

[quote="ciaobaby"]
The problem is the potential for disaster when a drive is removed and subsequently reconnected:

For Windows this may mean that the drive letter could change, dependant upon the order any other drives being connected.

For Linux or Mac same for Linux or Mac, what if the drive mounts as sde1 instead of sdd1

What if this is a NTFS format and you changed the drive label externally and your system is set to mount by label.
[/quote]

ciaobaby, thanks for your perspective.
From mine, I don't see a disaster:  What are the chances of a volume being mounted to the same path, with all the same files and file sizes and modification times (and whatever other data qBT checks), and yet they are actually different files?  That would be the closest thing to a disaster for qBT... but even that "disaster" would be remedied as soon as we communicated with a peer and the peer told qBT it was the wrong data, at which point qBT would perform a recheck anyway.

The issues you mention seem like user responsibilities or inevitabilities, when and if they apply.
I'm not suggesting making them qBittorrent's problem.


This might even be a relatively simple case to code for, whether it's in qBT or libtorrent.

I am not thinking of any kind of magical-type behavior (magic example which I would not implement: although it is possible to have the OS -- at least in the case of Windows -- notify qBT/libtorrent if the files are moved and what their new location is if on the same volume, I don't think qBT should do that... it should treat paths provided it as explicit in my view.)

[ciaobaby]

notify qBT/libtorrent if the files are moved and what their new location is if on the same volume,

You are missing one vital point, ... BitTorrent clients do not understand the concept of 'files', they understand a block of data of a specific size (in bytes) that is apportioned  into equally sized pieces, and this data starts at a particular segment on the disc platter(s) and occupies a number of segments that equals the payload size / segment size.
The 'files' do not matter in the slightest, They are simply an arbitrary division that the operating system 'decodes' and uses for humans to understand.

[AsaRossoff]

notify qBT/libtorrent if the files are moved and what their new location is if on the same volume,

You are missing one vital point, ... BitTorrent clients do not understand the concept of 'files', they understand a block of data of a specific size (in bytes) that is apportioned  into equally sized pieces, and this data starts at a particular segment on the disc platter(s) and occupies a number of segments that equals the payload size / segment size.
The 'files' do not matter in the slightest, They are simply an arbitrary division that the operating system 'decodes' and uses for humans to understand.

Hi ciaobaby, you quoted that from my paragraph about what I *don't* want qBT to do, so remember, I am explicitly excluding from my request what I noted in that paragraph and other non-obvious behavior.  I think I'm requesting a simple I/O error handling issue.

However, you did say that "BitTorrent clients do not understand the concept of 'files'".
Since my I/O issue requires a qBT/libtorrent to notice files either being locatable or not, I will address that argument.
As I see it, Bittorrent clients must and DO have a concept of files.  The files are even defined in the torrent metadata.  They just aren't necessarily aligned with the data that clients download and upload (blocks from full pieces) or with the data that is hashed (full pieces).  Clients of course know what and where files are, and their relationship to the pieces and blocks; after all, qBT and other clients store the data in files on disk.

[ciaobaby]

and other clients store the data in files on disk.

Nope, the operating system recognises the payload data block as 'files'. A 'file' may span several pieces, or a piece may contain one file or several 'files'

That's is why the ".unwanted" or "partfile" exist on the drive if you 'skip' files from the payload. They are the 'store' for the pieces of data in the files that you do not require, but ARE required for the 'files' you do want.  File names are in the meta data for the benefit of the human user NOT the BT client.

[AsaRossoff]

ciaobaby, I don't want to argue about this.  If you look in the .unwanted folder, tell me: are there files there?  I think we probably both understand this aspect of Bittorent the same way.  But how does the way data is transferred have anything to do with whether clients use files to store the data?! They do! And I think you know that as well as I do.

[ciaobaby]

Because the disc filing system has to call them something, try opening the file and you will find that it is ONLY the first few or the last few bytes that are actually there.

On BT clients that use the $Partfile.dat convention for unwanted pieces the is only a single 'file' that holds the unwanted pieces.

[AsaRossoff]

Ciaobaby, Right. I know.  Please explain how it is relevant to the topic or stop discussing it.  I think this discussion is completely off-topic.

[ciaobaby]

It's relevant to the point of you misunderstanding how it  actually works, and expecting the client to know that you actually want to keep the file that you flagged as  "do not download" then claiming that it is a fault in the client for deleting what you wanted.

About 3/4 of the 'bugs' that get 'reported' are actually people not understanding how the client "does things" so whenever there is an opportunity to explain it for others to read in the future it is worth taking.