FTK Imager fragment...
 
Notifications
Clear all

FTK Imager fragments hashing failure??

14 Posts
5 Users
0 Likes
1,321 Views
(@uprisingfuture)
Posts: 3
New Member
Topic starter
 

Hi,

I recently (2 days ago) made an Raw dd image of an USB disk with FTK Imager 3.4.2.2. I had set the fragment size on 1500 mb each.
http//www.imgshark.org/image/zZc

When the image was made, there were 10 fragments of 1500 mb each. It's an image of an USB with 16 GB and it was a physical copy. When I checked the hashes, these SHA1Sum hashes came out
.001 0D18AC3644CFD5B1209779D2EBC48ABC7CCF3369
.002 DAA2A4036D25246224FEEA64AF1CB16C0DFCFC3C
.003 EEAF5B8817267BE53612730323D2A5DD16E44E22
.004 5FF8785AB297900E3DE7B8A43F58B626E222D131
.005 5FF8785AB297900E3DE7B8A43F58B626E222D131

.006 D1060231AFAE248AE3637AF95DB00064087B6A26
.007 6B88401616531A34691FF457D6C347FB07BE5FC8
.008 6B88401616531A34691FF457D6C347FB07BE5FC8
.009 6B88401616531A34691FF457D6C347FB07BE5FC8

.010 C5F3910184F919648310D407B52A26012E7FB2F3

Could someone please tell me how it's possible that some fragments got the same hash value? I'm working on a school project right now, so fast response is highly appreciated!

Great Regards,
UprisingFuture (The Netherlands)

 
Posted : 26/05/2016 7:12 pm
jaclaz
(@jaclaz)
Posts: 5133
Illustrious Member
 

Hmmm. ?

Are you really-really sure that the USB key is 16 Gb?

Is it not that you bought it for a fantastically low price on (say) e-bay by a private seller?

It is not uncommon to find "fake" sticks that "wrap around" beyond a given address or however respond with gibberish to read requests.

BUT - before - check the contents of two of the images with a same hash.

IF (possible in theory on an only partially used stick) the contents of the segments are all 00's (or other "fixed pattern"), then it is of course normal that they have the same hash.

You can also check the 00 hash with this tool
http//www.edenprime.com/tools/epAllZeroHashCalculator.htm

A looong shot, but the segments 7,8 and 9 might well be the case above (and the 10 might be not the same size as the preceding three ones so even if all 00's it would have a different hash).

I am more perplexed about the segments 4 and 5 as it would be extremely rare to find two contiguous segments that have (say) first sector all 1's and all other sectors 0's as it would mean that by pure chance your 1500 Mb (the exact value in bytes is needed) is *somehow* modular with two files or with two filesystem/OS structure elements. 😯

It could be that segments 4 and 5 are filled with a pattern, let's say all F6's, and that 7, 8, 9 (and possibly 10) are filled with another one, let's say all 00's, but it would IMHO be a rare occurrence.

jaclaz

 
Posted : 26/05/2016 9:21 pm
(@athulin)
Posts: 1156
Noble Member
 

Could someone please tell me how it's possible that some fragments got the same hash value?

Nothing odd in that, is there? You may have made some private assumptions, under which it would be extremely odd to see that – but that just tells you that those assumptions are wrong in one way or another.

As Jaclaz has already suggested, one assumption surprisingly many people make is that the equipment works as it should. That includes both the storage equipment being imaged, and the equipment used to create the image.

Another is that the contents of the equipment somehow is 'unpredictable'. There are many situations it which device contents is highly predictable.

Document *your* assumptions, and devise methods for verifying if they are correct or not.

 
Posted : 26/05/2016 10:16 pm
(@mscotgrove)
Posts: 938
Prominent Member
 

Examine the areas of the memory chip with a Hex editor. Are they blank streams?

All the hash is saying is that the contents are identical, so have a look at the contents. This might answer your question.

 
Posted : 30/05/2016 3:14 pm
(@pxtxr)
Posts: 7
Active Member
 

5FF8785AB297900E3DE7B8A43F58B626E222D131 = SHA1(1500 * 1024 * 1024 * '\xFF')
6B88401616531A34691FF457D6C347FB07BE5FC8 = SHA1(1500 * 1024 * 1024 * '\x00')

So the ones with hash 5FF8785AB297900E3DE7B8A43F58B626E222D131 contains 1500 MB worth of FF and the ones with hash 6B88401616531A34691FF457D6C347FB07BE5FC8 contains 1500 MB worth of 00. Both are empty. Just in different ways.

 
Posted : 30/05/2016 3:24 pm
jaclaz
(@jaclaz)
Posts: 5133
Illustrious Member
 

5FF8785AB297900E3DE7B8A43F58B626E222D131 = SHA1(1500 * 1024 * 1024 * '\xFF')
6B88401616531A34691FF457D6C347FB07BE5FC8 = SHA1(1500 * 1024 * 1024 * '\x00')

So the ones with hash 5FF8785AB297900E3DE7B8A43F58B626E222D131 contains 1500 MB worth of FF and the ones with hash 6B88401616531A34691FF457D6C347FB07BE5FC8 contains 1500 MB worth of 00. Both are empty. Just in different ways.

Nice confirmation of my (educated ? ) guess , thanks ) .

Though the FF is a "strange" pattern (when we are talking of formatting tools), maybe it comes form some "overwriting" or from the manufacturer "mass production tool".

How did you (which tool/program did you use) calculate those hashes?

Or you did create some 1500 Mb test files to calculate it with "common" hashing tools?

A program capable to calculate the hashes of arbitrary length files filled with a given pattern may be useful for other chores.

The previously suggested tool gets a different SHA1 for 1572864000 bytes (for 00's), so maybe it is not as valid as I believed.

jaclaz

 
Posted : 30/05/2016 3:47 pm
(@pxtxr)
Posts: 7
Active Member
 

How did you (which tool/program did you use) calculate those hashes?

Simply wrote a small script that did it. It's a one-liner in python.

Though the FF is a "strange" pattern

Not really. He said USB which made me assume some sort of NAND backed storage. They're generally only able to transition bits from 1 -> 0 so assumed 0xff could be common empty stuff and 0x00 would either sectors the controller of the stick is hiding or that were filled with 0's by something external. Dumping the raw NAND would reveal more information about the controller and the FTL.

Don't know if any of these assumptions are actually true but at least they provide some plausible explanation. The other option is of course that there's very large or very many files on that memory stick filled with 0x00's or 0xff's.

 
Posted : 30/05/2016 3:58 pm
jaclaz
(@jaclaz)
Posts: 5133
Illustrious Member
 

I take it back. 😯
The epallzeroes does work fine (there must have been a glitch in the matrix when I first attempted it, now it gives a 6B88401616531A34691FF457D6C347FB07BE5FC8 alright ) ).

@Pxtxr
Yep, can you share the little python script/oneliner?
Does it work also for multibyte patterns?

About the FF I was trying to say that I know no FORMAT tool (OS tool) that would use that value to fill a drive, so it has either been done on purpose by running some specific tool or it is a remnant of the factory format (and a format command with a recent Windows OS has never been run on it or it has only been run with the /q switch).

jaclaz

 
Posted : 30/05/2016 4:10 pm
(@pxtxr)
Posts: 7
Active Member
 

jaclaz,

I did it very naively

hashlib.sha1(1500 * 1024 * 1024 * b"\x00").hexdigest()
It will load the entire buffer into memory and hash it from there. A better solution would be to use iterators and repeat() but you need a buffer interface.

hashlib.sha1(int((1500 * 1024 * 1024) / 4) * b"\x00\x01\x02\x03").hexdigest()
will give you arbitrary repeating patterns. In this case 00 01 02 03 …

About the FF I was trying to say that I know no FORMAT tool (OS tool) that would use that value to fill a drive

There's a few for mobile device file systems / FTLs that will do it by issuing an erase command which will reset a full erase block to 0xff usually. It all depends on how adapted to flash the file system is. Now a days it's mostly eMMC which does its own internal FTL and then hides the actual data as it's stored on flash. It's not only from factory that you get empty blocks filled with 0xff. How much of that shines through depends on the USB drive flash controller.

 
Posted : 30/05/2016 4:42 pm
jaclaz
(@jaclaz)
Posts: 5133
Illustrious Member
 

Thanks for the snippets ) I'll experiment with those.

It's not only from factory that you get empty blocks filled with 0xff. How much of that shines through depends on the USB drive flash controller.

We will need to agree to disagree on this. 😯

We are talking of a USB stick.

The image OP made was through FTK imager that images the "physicaldrive" through the OS, it has not any kind of "different" access than any other OS tools have.
I.e. it uses "standard" USB commands/protocols, doesn't use particular USB drivers and does not "talk" to the stick controller at a "lower" level than the OS does.

If it found FF's it means that the OS can see those same FF's, or more exactly any request for a given sector will result with FF's both in *any* OS issued command and in FTK imager as anyway it will go through the stick controller that will perform the *whatever* translation it is supposed to do.

*Any* normal format command in *any* "common" OS either ignores the current content of a disk-like device, writing only the file system structures OR (as it is the case since Vista if the /q switch is not specified) it will write 00's to it.

So the hypothesis (for me) are the following
1) the area's with FF's come from an initial "factory" formatting AND the stick was never reformatted with a recent Windows OS (without the /q switch) OR it was reformatted with one of those (but specifying the /q switch) OR it was reformatted with an older/other system (that wouldn't touch previous content)
2) the area's with FF's were written intentionally by a program/tool

The same reasoning does not apply to the 00's because a number of tools might write large files (or areas) with 00's .

There is a third hypothesis that could stand (though IMHO farfetched), if the device is partitioned, the FF's (that are in the first part) could well be part of the "factory" format, the original single partition has been shrunk and only the second one has been formatted (without the /q switch) but it is uncommon that a stick comes partitioned from factory (possibly with the exception of few "high speed" USB 3 models that are actually a USB to SATA bridge and have internally a miniaturized SATA SSD).

A fourth (still possible) hypothesis would be that it is actually a "fake" (or malfunctioning) stick that "produces" 00's beyond a given address (but still the FF's must come from one of the first two hypothesis).

Of course with the data available all of the above are still just guesses, for the sake of argument ) .

jaclaz

 
Posted : 30/05/2016 5:51 pm
Page 1 / 2
Share: