Resizing a partition to the left an efficient way - without copying its all content

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Resizing a partition to the left an efficient way - without copying its all content

Sam Bul
There were several requests made on Gparted forum
(http://gparted-forum.surf4.info/viewtopic.php?id=14948) over time to
Resize partition without copying it's content. I was suggested to
request NTFSPROGS developers to consider updating the program to
address this long overdue issue.

Everyone who uses hard drives knows well a remarkable trend: their
reliability drops with drive size increase, even for Green slower
spinning models. Todays 2-3 Tb HDs are far less reliable than their
smaller counterparts just a few years ago. I'm not even talking about
10-20 years old models - they still spin today without any issues
while purchased 1-year ago HDs frequently fail one after another. And
replacement refurb drives often work no longer than one extra month.
It becomes a major task for each PC owner to use existing HD lifespan
resource more efficiently.

Situation also changed with energy consumption and efficiency trends
around the world . We must be responsible citizens of this planet, and
take good care about spending its resources in the most efficient way.
In view of these trends, current situation when increasing a partition
to the left results in unnecessary copying of all its content to the
left is no longer acceptable. For large drives with 2+ Tb of data such
ops significantly decrease their lifespan and waste plenty of precious
energy for no objective reason. Its especially true for lower quality
consumer drive models, despite they often hold massive video & movie
collections that frequently need expending.

Its a typical scenario, when a new large drive owner formats it
originally with several partitions only to find later, some of them
can be safely deleted to free more space for remaining ones. That's
where we all face an acute problem of how to avoid copying large movie
or file collections to the left at merging a previous and current
partitions. Those of us, who gone through multiple drive warranty
repairs understand very well what such ops usually entail.

I ask NTFSPROGS developers to add an algorithm that would increase a
partition to the left without copying all its content to the left. The
time is long overdue for the task. There is absolutely no need to move
huge data massive of constantly growing in size drive models, waste
precious energy and spend limited HD lifespan and significant user
time for literally no objective reason. A joint switchable algorithm
can be developed to resize partitions in any direction in a similar
fashion. It will not result in increased program maintenance time, if
done in a smart way. The differences in resizing a partition to the
left from its END and BEGINNING aren't that significant code wise.

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
Reply | Threaded
Open this post in threaded view
|

Re: Resizing a partition to the left an efficient way - without copying its all content

Anton Altaparmakov
Hi,

On 1 Jul 2011, at 19:25, Sam Bul wrote:
> There were several requests made on Gparted forum
> (http://gparted-forum.surf4.info/viewtopic.php?id=14948) over time to
> Resize partition without copying it's content. I was suggested to
> request NTFSPROGS developers to consider updating the program to
> address this long overdue issue.
[snip]
> It will not result in increased program maintenance time, if
> done in a smart way. The differences in resizing a partition to the
> left from its END and BEGINNING aren't that significant code wise.

Your tirade shows only one thing: you have no idea what you are talking about! /-;

Making larger at the end is a TRIVIAL operation.  The only thing that needs to happen is to update the boot sector with the new number size, extend the volume bitmap to cover the new size, and to extend the bad blocks file to cover the new size and that's it job done.  Can be done in a couple of seconds.

Moving the beginning of the partition to the left on the other hand is a MASSIVE operation.  You have to move the boot sector and its backup on older NTFS volumes for which you might need to move a file out of the way if the new location is not free.  You have to completely rewrite the volume bitmap so that you can insert the correct number of bits in front of it so actually you have to extend the bitmap then move all the data backwards.  Then you have to rewrite ALL metadata, i.e. the entire $MFT so that all cluster references are incremented by the number of clusters by which you have grown the partition, this involves reading all mft records, decompressing the mapping pairs array of each and every non-resident attribute, doing the sums in memory then compressing the updated runlist into mapping pairs arrays again which in turn might mean the new mapping pairs array does not fit in the space available so you then might have to create an attribute list attribute for the mft record and use that to split the no longer fitting attribute into multiple fragments and spread them over multiple mft records.  And guess what, you get one number wrong or even one bit wrong and your data has flown out of the window so it has to be done with great care and attention.

There is a very good reason why ntfsresize only resizes on the right as you put it, i.e. at the end of the partition: because it is easy and easy to get right.  Doing it on the left, i.e. at the beginning is significantly harder and even if implemented could easily take a long time.  On a several TiB volume the MFT could be tens if not hundreds of GiB and you would have to read, modify, write all of to create a valid volume without moving the actual data along...

Given you think it is a trivial job you are welcome to implement it yourself and send us a patch to include in ntfsresize.  (-;

Or alternatively you are welcome to contract Tuxera (http://www.tuxera.com/about-us/contact/) to implement it but it will cost you quite a lot as this is a significant piece of development work...

Best regards,

        Anton
--
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/


------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
Reply | Threaded
Open this post in threaded view
|

Re: Resizing a partition to the left an efficient way - without copying its all content

Sam Bul
In reply to this post by Sam Bul
Hi Anton,

I appreciate your spirit. The details of your reply show, you're well qualified developer capable of tackling this challenging task. I'd kindly ask you though to pay MORE attention to details of my proposal to avoid confusion and better understand its significance. In particular:

- its obvious from my proposal, it represents not my personal desire as a user (not programmer) of partitioning software, but rather demonstrates that NTFSResize is well behind current expectations and realities, and that needs to be fixed for the benefit of ALL its users (not to make me happy). Guess what - its not surprising, life goes on, things change around us, we change the way of using things, and all humans create must be continuously improved and updated to survive;

- note that I said "The differences in resizing a partition to the left from its END and BEGINNING aren't that significant code wise." Indeed, shrinking a partition from its end appears to be a different operation compare to extending it from the same end - would you agree? That's kind of detail, an experienced developer like yourself should have noted even at the highest spirit point. Otherwise one can expect for sure exactly what you wrote: "you get one number wrong or even one bit wrong and your data has flown out of the window so it has to be done with great care and attention." What if you get the whole idea wrong, not just one bit?

- and again you made up stuff, making yet another mistake along the way: "There is a very good reason why ntfsresize only resizes on the right as you put it". I did NOT say anything about resizing on the right for a simple reason that a partition frequently needs not only expending but also shrinking from left to right, and from right to left , and that is not less challenging than expending to the left without moving all data;

- as well, on large consumer drives large file massives often contain very limited number of very large files (like movies), so MFT is usually very small in size. That type of data is very good candidate for initial testing of the improved future functionality of NTFSResize.

Summarizing, with a little less spirituality and some more professional attention to detail, its quite feasible and long overdue (read the background) to fix this long outstanding issue. In fact, the way expending to the left currently done, looks today totally unprofessional, and not up to the current standard, even after ignoring numerous mistakes in your answer. So, lets have a cold look on how to bring NTFSResize functionality to today's realities.

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
Reply | Threaded
Open this post in threaded view
|

Re: Resizing a partition to the left an efficient way - without copying its all content

Anton Altaparmakov
Hi,

So what did you pay for ntfsresize?  Nothing?  Then you get what there is.  
You want more?  I told you already, you can hire Tuxera to do it for you
and it will cost you...

Alternatively, find someone willing to implement it for you for free.  
Good luck with that...

Until either of these things is done, you get to move the whole partition.  
Contrary to the malinformed FUD you are writing there is no problem with
that, it doesn't take that long, not a problem at all.  Any RAID array is
constantly reading the entire set to ensure the bits haven't rotted away.  
It does not reduce its lifetime or anything like that.  Please stop
writing things like that when you have no idea what you are talking
about...

Best regards,

        Anton
--
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
Reply | Threaded
Open this post in threaded view
|

Re: Resizing a partition to the left an efficient way - without copying its all content

Sergio Campamá-2
Summarizing, with a little less spirituality and some more professional attention to detail, its quite feasible and long overdue (read the background) to fix this long outstanding issue. In fact, the way expending to the left currently done, looks today totally unprofessional, and not up to the current standard, even after ignoring numerous mistakes in your answer. So, lets have a cold look on how to bring NTFSResize functionality to today's realities.

Wow Sam, that's a really nice way to convince someone to do your work for free. Maybe if you now say something about his mother he'll do it. 

If you need something done, either do it yourself or pay someone to do it. I think you're confusing open source philosophy with your personal team of coders. 

The fact that you mention so much about energy saving and being responsible to mother earth by reducing hard drive consumption makes me think that Computer Science is not your major (as we techies don't really care too much for energy saving). So maybe go to college (it's only four years), learn Computer Science and then come back and see if you think Anton's answers are wrong.

Regards,
---------------------------------------
Sergio Campamá
[hidden email]




On Jul 2, 2011, at 5:46 PM, Anton Altaparmakov wrote:

Hi,

So what did you pay for ntfsresize?  Nothing?  Then you get what there is.  
You want more?  I told you already, you can hire Tuxera to do it for you
and it will cost you...

Alternatively, find someone willing to implement it for you for free.  
Good luck with that...

Until either of these things is done, you get to move the whole partition.  
Contrary to the malinformed FUD you are writing there is no problem with
that, it doesn't take that long, not a problem at all.  Any RAID array is
constantly reading the entire set to ensure the bits haven't rotted away.  
It does not reduce its lifetime or anything like that.  Please stop
writing things like that when you have no idea what you are talking
about...

Best regards,

Anton
--
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev


------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
Reply | Threaded
Open this post in threaded view
|

Re: Resizing a partition to the left an efficient way - without copying its all content

Mark Day-2
In reply to this post by Anton Altaparmakov
On Jul 1, 2011, at 3:22 PM, Anton Altaparmakov wrote:

> Hi,
>
> On 1 Jul 2011, at 19:25, Sam Bul wrote:
>> There were several requests made on Gparted forum
>> (http://gparted-forum.surf4.info/viewtopic.php?id=14948) over time to
>> Resize partition without copying it's content. I was suggested to
>> request NTFSPROGS developers to consider updating the program to
>> address this long overdue issue.
> [snip]
>> It will not result in increased program maintenance time, if
>> done in a smart way. The differences in resizing a partition to the
>> left from its END and BEGINNING aren't that significant code wise.

Yes, they ARE significant and complex differences, even if you don't realize it.

> Your tirade shows only one thing: you have no idea what you are talking about! /-;
>
> Making larger at the end is a TRIVIAL operation.  The only thing that needs to happen is to update the boot sector with the new number size, extend the volume bitmap to cover the new size, and to extend the bad blocks file to cover the new size and that's it job done.  Can be done in a couple of seconds.
>
> Moving the beginning of the partition to the left on the other hand is a MASSIVE operation.  You have to move the boot sector and its backup on older NTFS volumes for which you might need to move a file out of the way if the new location is not free.  You have to completely rewrite the volume bitmap so that you can insert the correct number of bits in front of it so actually you have to extend the bitmap then move all the data backwards.  Then you have to rewrite ALL metadata, i.e. the entire $MFT so that all cluster references are incremented by the number of clusters by which you have grown the partition, this involves reading all mft records, decompressing the mapping pairs array of each and every non-resident attribute, doing the sums in memory then compressing the updated runlist into mapping pairs arrays again which in turn might mean the new mapping pairs array does not fit in the space available so you then might have to create an attribute list attribute for the mft record a
>
> There is a very good reason why ntfsresize only resizes on the right as you put it, i.e. at the end of the partition: because it is easy and easy to get right.  Doing it on the left, i.e. at the beginning is significantly harder and even if implemented could easily take a long time.  On a several TiB volume the MFT could be tens if not hundreds of GiB and you would have to read, modify, write all of to create a valid volume without moving the actual data along...

All of Anton's points above are correct.  I'll add two of my own.

A major concern for this kind of operation is the safety of the data, especially while you're in the middle of a resize.  If you were to crash, lose power, etc., during the resize, you wouldn't want to end up with a corrupt file system.  While you might not need to move most of the file contents, you have to touch the metadata for all non-empty files.  If the metadata gets damaged, you very often lose the ability to access the user data.

One way to address the possibility of failure during resize is to mark the volume such that it won't be mounted, and leave an indication of how far the resize has progressed so that you can continue it after a crash.  I personally consider that approach too risky.  (Consider what happens if there is a bug in the resize code that causes it to crash.  That bug would likely prevent it from continuing after the crash, too.)  Another possibility is to write brand new copies of the metadata with all of the updates, leaving the old copies untouched, so that the volume could be mounted and used after a resize failure.  This is certainly the approach I would favor.  But it requires free space.  Depending on how much you're resizing, there might not be adequate free space.

In practice, it's hard to ensure a drive has actually written the data you asked it to.  This is especially true of USB disks, which very often do not support a "flush track cache" or "synchronize cache" command properly.  Lots of drives claim to support it, but then don't actually implement it correctly.  This is another concern for in-place metadata updates.  But it is also a concern even if you write brand new copies of the metadata in new locations (since you could lose power after the boot sector has been written and points to the new metadata, you wrote the metadata to the drive, but the drive didn't actually write it to the media).  There are ways of trying to work around such problems, but they're inherently unreliable.

Resizing by moving the start of the volume, and not moving (most) file contents, could be done in some cases.  But it's decidedly non-trivial.  It's even harder to do it safely.

-Mark


------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev