Millions of orphan files found with sleuthkit develop branch

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Millions of orphan files found with sleuthkit develop branch

Luís Filipe Nassif
We tested loaddb of both the released 4.1.3 version and the develop branch of sleuthkit on a NTFS image of a hard disk with a lot of bad blocks, many of them at the beginning of the disk.

The 4.1.3 version found ~400.000 allocated files more ~100.000 orphan files, about the same found by other forensic tools. The develop branch found the same ~400.000 allocated files more ~2.500.000 orphan files! Most of these millions of orphans have corrupted names or the name OrphanFile-xxxxxxx and have lengths ranging from 0 to 4.294.967.296 bytes. We think the recent changes to NTFS code are causing this large number of corrupted orphans to be added to the case. Maybe it should be investigated before the final 4.2 release.

Luis

------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
sleuthkit-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sleuthkit-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Millions of orphan files found with sleuthkit develop branch

Luís Filipe Nassif
Another information: the sum of the millions of file sizes resulted in 1,1 petabyte, while the image has only 250 GB.


2014-07-23 22:21 GMT-03:00 Luís Filipe Nassif <[hidden email]>:
We tested loaddb of both the released 4.1.3 version and the develop branch of sleuthkit on a NTFS image of a hard disk with a lot of bad blocks, many of them at the beginning of the disk.

The 4.1.3 version found ~400.000 allocated files more ~100.000 orphan files, about the same found by other forensic tools. The develop branch found the same ~400.000 allocated files more ~2.500.000 orphan files! Most of these millions of orphans have corrupted names or the name OrphanFile-xxxxxxx and have lengths ranging from 0 to 4.294.967.296 bytes. We think the recent changes to NTFS code are causing this large number of corrupted orphans to be added to the case. Maybe it should be investigated before the final 4.2 release.

Luis


------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
sleuthkit-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sleuthkit-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Millions of orphan files found with sleuthkit develop branch

Luís Filipe Nassif
This problem still happens with 4.2.0 branch. If I can help with some more information, please let me know.

Thanks
Luis

2014-07-24 9:21 GMT-03:00 Luís Filipe Nassif <[hidden email]>:
Another information: the sum of the millions of file sizes resulted in 1,1 petabyte, while the image has only 250 GB.


2014-07-23 22:21 GMT-03:00 Luís Filipe Nassif <[hidden email]>:
We tested loaddb of both the released 4.1.3 version and the develop branch of sleuthkit on a NTFS image of a hard disk with a lot of bad blocks, many of them at the beginning of the disk.

The 4.1.3 version found ~400.000 allocated files more ~100.000 orphan files, about the same found by other forensic tools. The develop branch found the same ~400.000 allocated files more ~2.500.000 orphan files! Most of these millions of orphans have corrupted names or the name OrphanFile-xxxxxxx and have lengths ranging from 0 to 4.294.967.296 bytes. We think the recent changes to NTFS code are causing this large number of corrupted orphans to be added to the case. Maybe it should be investigated before the final 4.2 release.

Luis



------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
sleuthkit-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sleuthkit-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Millions of orphan files found with sleuthkit develop branch

Luís Filipe Nassif
This error have happened again with a colleague's NTFS image, using the develop branch compiled about 1 month ago. Thousands of huge corrupted orphans were added by loaddb, which caused our processing application (and probably Autopsy too) to process indefinitely the evidence.

Any help will be appreciated.

Regards,
Luis Nassif

2014-09-30 21:00 GMT-03:00 Luís Filipe Nassif <[hidden email]>:
This problem still happens with 4.2.0 branch. If I can help with some more information, please let me know.

Thanks
Luis

2014-07-24 9:21 GMT-03:00 Luís Filipe Nassif <[hidden email]>:
Another information: the sum of the millions of file sizes resulted in 1,1 petabyte, while the image has only 250 GB.


2014-07-23 22:21 GMT-03:00 Luís Filipe Nassif <[hidden email]>:
We tested loaddb of both the released 4.1.3 version and the develop branch of sleuthkit on a NTFS image of a hard disk with a lot of bad blocks, many of them at the beginning of the disk.

The 4.1.3 version found ~400.000 allocated files more ~100.000 orphan files, about the same found by other forensic tools. The develop branch found the same ~400.000 allocated files more ~2.500.000 orphan files! Most of these millions of orphans have corrupted names or the name OrphanFile-xxxxxxx and have lengths ranging from 0 to 4.294.967.296 bytes. We think the recent changes to NTFS code are causing this large number of corrupted orphans to be added to the case. Maybe it should be investigated before the final 4.2 release.

Luis




------------------------------------------------------------------------------

_______________________________________________
sleuthkit-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sleuthkit-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Millions of orphan files found with sleuthkit develop branch

Stefan Petrea
Hi Luis,

Could the NTFS image you're looking at be trimmed down and provided as sample input to reproduce the problem ?

Best Regards,
Stefan

On Thu, Aug 13, 2015 at 8:05 PM, Luís Filipe Nassif <[hidden email]> wrote:
This error have happened again with a colleague's NTFS image, using the develop branch compiled about 1 month ago. Thousands of huge corrupted orphans were added by loaddb, which caused our processing application (and probably Autopsy too) to process indefinitely the evidence.

Any help will be appreciated.

Regards,
Luis Nassif


2014-09-30 21:00 GMT-03:00 Luís Filipe Nassif <[hidden email]>:
This problem still happens with 4.2.0 branch. If I can help with some more information, please let me know.

Thanks
Luis

2014-07-24 9:21 GMT-03:00 Luís Filipe Nassif <[hidden email]>:
Another information: the sum of the millions of file sizes resulted in 1,1 petabyte, while the image has only 250 GB.


2014-07-23 22:21 GMT-03:00 Luís Filipe Nassif <[hidden email]>:
We tested loaddb of both the released 4.1.3 version and the develop branch of sleuthkit on a NTFS image of a hard disk with a lot of bad blocks, many of them at the beginning of the disk.

The 4.1.3 version found ~400.000 allocated files more ~100.000 orphan files, about the same found by other forensic tools. The develop branch found the same ~400.000 allocated files more ~2.500.000 orphan files! Most of these millions of orphans have corrupted names or the name OrphanFile-xxxxxxx and have lengths ranging from 0 to 4.294.967.296 bytes. We think the recent changes to NTFS code are causing this large number of corrupted orphans to be added to the case. Maybe it should be investigated before the final 4.2 release.

Luis




------------------------------------------------------------------------------

_______________________________________________
sleuthkit-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sleuthkit-developers



------------------------------------------------------------------------------

_______________________________________________
sleuthkit-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sleuthkit-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Millions of orphan files found with sleuthkit develop branch

Luís Filipe Nassif
Sorry for the long delay. I do not have the image with me, I will ask my colleague if trimming the image is possible... We worked around the problem by filtering out orphans with logical size greater than 10 MB before sending them to the processing engine.

Thank you,
Luis 

2015-08-13 14:13 GMT-03:00 Stefan Petrea <[hidden email]>:
Hi Luis,

Could the NTFS image you're looking at be trimmed down and provided as sample input to reproduce the problem ?

Best Regards,
Stefan

On Thu, Aug 13, 2015 at 8:05 PM, Luís Filipe Nassif <[hidden email]> wrote:
This error have happened again with a colleague's NTFS image, using the develop branch compiled about 1 month ago. Thousands of huge corrupted orphans were added by loaddb, which caused our processing application (and probably Autopsy too) to process indefinitely the evidence.

Any help will be appreciated.

Regards,
Luis Nassif


2014-09-30 21:00 GMT-03:00 Luís Filipe Nassif <[hidden email]>:
This problem still happens with 4.2.0 branch. If I can help with some more information, please let me know.

Thanks
Luis

2014-07-24 9:21 GMT-03:00 Luís Filipe Nassif <[hidden email]>:
Another information: the sum of the millions of file sizes resulted in 1,1 petabyte, while the image has only 250 GB.


2014-07-23 22:21 GMT-03:00 Luís Filipe Nassif <[hidden email]>:
We tested loaddb of both the released 4.1.3 version and the develop branch of sleuthkit on a NTFS image of a hard disk with a lot of bad blocks, many of them at the beginning of the disk.

The 4.1.3 version found ~400.000 allocated files more ~100.000 orphan files, about the same found by other forensic tools. The develop branch found the same ~400.000 allocated files more ~2.500.000 orphan files! Most of these millions of orphans have corrupted names or the name OrphanFile-xxxxxxx and have lengths ranging from 0 to 4.294.967.296 bytes. We think the recent changes to NTFS code are causing this large number of corrupted orphans to be added to the case. Maybe it should be investigated before the final 4.2 release.

Luis




------------------------------------------------------------------------------

_______________________________________________
sleuthkit-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sleuthkit-developers




------------------------------------------------------------------------------

_______________________________________________
sleuthkit-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sleuthkit-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Millions of orphan files found with sleuthkit develop branch

Luís Filipe Nassif
Hi folks,

We are still having this issue with tsk-4.4. I have received one report yesterday and one today about the processing hanging with thousands of orphan files with ~4GB of size, which together result in 140TB of data in one image!

Fortunately I have access to the image of the other report. Looking at it, the orphans together resulted in ~500GB of data, they were recovered from a FAT32 file system of only ~32GB of size! That 32GB FAT32 FS was recovered from a 92,5 KB volume/partition! This 92,5KB partition recognized by TSK-4.4 is shown by FTKImager as an unpartitioned space, but I think this is another issue.

I have trimmed the image down from the original 320GB to 10GB (5GB in ewf format) without the user data. Because of that, the orphan files together now result in ~45GB (not the original 500GB). I will open a ticket at github and post the link to the trimmed image there for reproducing the problem.

PS: In the past I have received similar reports with thumbdrives. So looks like the issue is with FAT and not with NTFS like I originally reported below. Probably those ntfs images had fat32 partitions side by side. This explains the upper limit of 4GB of size of those orphans.

Regards,
Luis Nassif


2015-09-07 12:12 GMT-03:00 Luís Filipe Nassif <[hidden email]>:
Sorry for the long delay. I do not have the image with me, I will ask my colleague if trimming the image is possible... We worked around the problem by filtering out orphans with logical size greater than 10 MB before sending them to the processing engine.

Thank you,
Luis 

2015-08-13 14:13 GMT-03:00 Stefan Petrea <[hidden email]>:
Hi Luis,

Could the NTFS image you're looking at be trimmed down and provided as sample input to reproduce the problem ?

Best Regards,
Stefan

On Thu, Aug 13, 2015 at 8:05 PM, Luís Filipe Nassif <[hidden email]> wrote:
This error have happened again with a colleague's NTFS image, using the develop branch compiled about 1 month ago. Thousands of huge corrupted orphans were added by loaddb, which caused our processing application (and probably Autopsy too) to process indefinitely the evidence.

Any help will be appreciated.

Regards,
Luis Nassif


2014-09-30 21:00 GMT-03:00 Luís Filipe Nassif <[hidden email]>:
This problem still happens with 4.2.0 branch. If I can help with some more information, please let me know.

Thanks
Luis

2014-07-24 9:21 GMT-03:00 Luís Filipe Nassif <[hidden email]>:
Another information: the sum of the millions of file sizes resulted in 1,1 petabyte, while the image has only 250 GB.


2014-07-23 22:21 GMT-03:00 Luís Filipe Nassif <[hidden email]>:
We tested loaddb of both the released 4.1.3 version and the develop branch of sleuthkit on a NTFS image of a hard disk with a lot of bad blocks, many of them at the beginning of the disk.

The 4.1.3 version found ~400.000 allocated files more ~100.000 orphan files, about the same found by other forensic tools. The develop branch found the same ~400.000 allocated files more ~2.500.000 orphan files! Most of these millions of orphans have corrupted names or the name OrphanFile-xxxxxxx and have lengths ranging from 0 to 4.294.967.296 bytes. We think the recent changes to NTFS code are causing this large number of corrupted orphans to be added to the case. Maybe it should be investigated before the final 4.2 release.

Luis




------------------------------------------------------------------------------

_______________________________________________
sleuthkit-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sleuthkit-developers





------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
sleuthkit-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sleuthkit-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Millions of orphan files found with sleuthkit develop branch

Luís Filipe Nassif
Opened https://github.com/sleuthkit/sleuthkit/issues/821

I will provide any other info needed to help fix the issue, now that I have at hand an image to reproduce the problem.

Thanks,
Luis

2017-04-28 16:46 GMT-03:00 Luís Filipe Nassif <[hidden email]>:
Hi folks,

We are still having this issue with tsk-4.4. I have received one report yesterday and one today about the processing hanging with thousands of orphan files with ~4GB of size, which together result in 140TB of data in one image!

Fortunately I have access to the image of the other report. Looking at it, the orphans together resulted in ~500GB of data, they were recovered from a FAT32 file system of only ~32GB of size! That 32GB FAT32 FS was recovered from a 92,5 KB volume/partition! This 92,5KB partition recognized by TSK-4.4 is shown by FTKImager as an unpartitioned space, but I think this is another issue.

I have trimmed the image down from the original 320GB to 10GB (5GB in ewf format) without the user data. Because of that, the orphan files together now result in ~45GB (not the original 500GB). I will open a ticket at github and post the link to the trimmed image there for reproducing the problem.

PS: In the past I have received similar reports with thumbdrives. So looks like the issue is with FAT and not with NTFS like I originally reported below. Probably those ntfs images had fat32 partitions side by side. This explains the upper limit of 4GB of size of those orphans.

Regards,
Luis Nassif


2015-09-07 12:12 GMT-03:00 Luís Filipe Nassif <[hidden email]>:
Sorry for the long delay. I do not have the image with me, I will ask my colleague if trimming the image is possible... We worked around the problem by filtering out orphans with logical size greater than 10 MB before sending them to the processing engine.

Thank you,
Luis 

2015-08-13 14:13 GMT-03:00 Stefan Petrea <[hidden email]>:
Hi Luis,

Could the NTFS image you're looking at be trimmed down and provided as sample input to reproduce the problem ?

Best Regards,
Stefan

On Thu, Aug 13, 2015 at 8:05 PM, Luís Filipe Nassif <[hidden email]> wrote:
This error have happened again with a colleague's NTFS image, using the develop branch compiled about 1 month ago. Thousands of huge corrupted orphans were added by loaddb, which caused our processing application (and probably Autopsy too) to process indefinitely the evidence.

Any help will be appreciated.

Regards,
Luis Nassif


2014-09-30 21:00 GMT-03:00 Luís Filipe Nassif <[hidden email]>:
This problem still happens with 4.2.0 branch. If I can help with some more information, please let me know.

Thanks
Luis

2014-07-24 9:21 GMT-03:00 Luís Filipe Nassif <[hidden email]>:
Another information: the sum of the millions of file sizes resulted in 1,1 petabyte, while the image has only 250 GB.


2014-07-23 22:21 GMT-03:00 Luís Filipe Nassif <[hidden email]>:
We tested loaddb of both the released 4.1.3 version and the develop branch of sleuthkit on a NTFS image of a hard disk with a lot of bad blocks, many of them at the beginning of the disk.

The 4.1.3 version found ~400.000 allocated files more ~100.000 orphan files, about the same found by other forensic tools. The develop branch found the same ~400.000 allocated files more ~2.500.000 orphan files! Most of these millions of orphans have corrupted names or the name OrphanFile-xxxxxxx and have lengths ranging from 0 to 4.294.967.296 bytes. We think the recent changes to NTFS code are causing this large number of corrupted orphans to be added to the case. Maybe it should be investigated before the final 4.2 release.

Luis




------------------------------------------------------------------------------

_______________________________________________
sleuthkit-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sleuthkit-developers






------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
sleuthkit-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sleuthkit-developers
Loading...