[Mldonkey-bugs] [Bug #1722] file gets downloaded more than 100% - false chunks received

[nobody]

=================== BUG #1722: LATEST MODIFICATIONS ==================
http://savannah.nongnu.org/bugs/?func=detailbug&bug_id=1722&group_id=1409

Changes by: Anonymous user        Date: 2002-Nov-21 19:30

------------------ Additional Follow-up Comments ----------------------------
I could not find a specification of the ICH (Intelligent Corruption Handling) 
protocol extension, but it looks rather simple.
With standard edonkey protocol, you fetch chunks (9.5Mb) md4 hashes before 
downloading, then each time you complete a chunk, you use its reference hash to 
validate the chunk. If it doesn't match, the whole chunk is invalidated.
With ICH, when corruption is detected, you ask for md4 hashes of zones (180Kb) 
inside the chunk to another peer, and individually validate or invalidate 
zones, then only refetch corrupted zones...




=================== BUG #1722: FULL BUG SNAPSHOT ===================


Submitted by: Loial                     Project: mldonkey, a free e-Donkey 
client
Submitted on: 2002-Nov-15 09:14
Category:  Core                         Severity:  5 - Major                    
Bug Group:  None                        Resolution:  None                       
Assigned to:  None                      Status:  Open                           
Release:  2.00                          Release:                                
Platform Version:  Linux i386-i686      Binaries Origin:  Compiled From CVS     

Summary:  file gets downloaded more than 100% - false chunks received

Original Submission:  I'm having difficulty downloading some files correctly 
because I download more than 100% of the file.
This is what I think happens:
I download a file, 
e.g.ed2k://|file|Need_For_Speed_Hot_Pursuit_2.DEVIANCE.ShareReactor.bin|739678128|CEDEDC2F223B6FC7A9B0AC651A686FBD|/

but someone seems to have some incorrect chunks, because the downloaded size is 
bigger than the original file size. Pressing 'Verify chunks' removes the 
incorrect data en leaves me with a file that is e.g. 95% downloaded. This is 
good. However the incorrect chunks are downloaded again, so I again have a file 
that is not correct and I have to press 'Verify chunks' again. This keeps 
repeating itself.
The problem is, I think, that the incorrect chunk is downloaded from the same 
user(s) again. It would be better though if the username is remembered as 
having that incorrect part, so the part is downloaded from someone else.
Problem might be that during transmission of the original correct chunk the 
data got corrupted, resulting in the now incorrect chunk and then it would not 
be wise to remember that user as having an incorrect chunk. Or is this no 
problem, because it is already taken care of by the transmission protocol?
It can be said that there isn't a real problem here, because eventually the 
missing part will be downloaded from someone else and then the chunk will be 
correct and the file will be finished. However at the moment I am downloading 
at 100kb/s for half a day already and I still only have the same 90% of the 
file that I had this morning. This is ofcourse a big waste of network bandwith 
and should therefor be prevented.
I hope I could make myself clear :)
Goodluck and thanx

Follow-up Comments
*******************

-------------------------------------------------------
Date: 2002-Nov-21 19:30             By: None
I could not find a specification of the ICH (Intelligent Corruption Handling) 
protocol extension, but it looks rather simple.
With standard edonkey protocol, you fetch chunks (9.5Mb) md4 hashes before 
downloading, then each time you complete a chunk, you use its reference hash to 
validate the chunk. If it doesn't match, the whole chunk is invalidated.
With ICH, when corruption is detected, you ask for md4 hashes of zones (180Kb) 
inside the chunk to another peer, and individually validate or invalidate 
zones, then only refetch corrupted zones...


-------------------------------------------------------
Date: 2002-Nov-19 10:05             By: Loial
Yesterday the download finished. It took me quite a while to get the last few 
percent. I have no idea how much false chunks I've downloaded, although I have 
seen the downloaded percentage wrap around and display a negative number once 
or twice ;-)

I do not know what you mean with 'the eMule's ICH Protocol extension', so I 
can't comment on that.

-------------------------------------------------------
Date: 2002-Nov-17 10:17             By: None
I tried downloading this file (giving the whole ed2k URL wasn't very wise; next 
time, if you really need to, only give MD4 hash and size, it's enough)
Download went to 101.6% or so, and the file completed normally, without my 
intervention...


-------------------------------------------------------
Date: 2002-Nov-16 15:11             By: None
IP packets are checksummed, so corruption during transfer should be detected; 
But they're other sources of corruption (disk, memory,... of a peer)
The problem is that a given chunk doesn't usually come from just one source, 
but from several sources, so mldonkey would have to keep a list of involved 
sources for each chunk...
Another solution would be to implement the eMule's ICH protocol extension (or a 
similar extension).


CC list is empty


No files currently attached


For detailed info, follow this link:
http://savannah.nongnu.org/bugs/?func=detailbug&bug_id=1722&group_id=1409