Windows 7 Dynamic Disks and LDM

classic Classic list List threaded Threaded
23 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Windows 7 Dynamic Disks and LDM

Carl-Daniel Hailfinger-2
Hi,

I have trouble accessing Windows 7 Dynamic Disks under Linux 3.2. The
partitioning seems to be EFI-based with one big EFI partition apparently
containing all the volumes, but I can't see any block devices in Linux
for any of those volumes.
Is accessing those volumes supported, and if not, is there any
collection of documents I could use to implement support for that
myself? An extended Google search only turned up high-level overviews
and no real technical data about that disk management scheme.

Regards,
Carl-Daniel

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
Reply | Threaded
Open this post in threaded view
|

Re: Windows 7 Dynamic Disks and LDM

Anton Altaparmakov
Hi,

On 20 Jul 2012, at 01:17, Carl-Daniel Hailfinger wrote:
> I have trouble accessing Windows 7 Dynamic Disks under Linux 3.2. The
> partitioning seems to be EFI-based with one big EFI partition apparently
> containing all the volumes, but I can't see any block devices in Linux
> for any of those volumes.

First of all have you got LDM support compiled into your kernel?

To check on many Linux distros you can do:

        zgrep LDM /proc/config.gz

And if you see:

        CONFIG_LDM_PARTITION=y

As a result of that command you know it is enabled.

If you get:

        gzip: /proc/config.gz: No such file or directory

Then you need to find the kernel configuration for the running system which you can usually find in /lib/modules/kernel version/build/.config so to check that you would do:

        grep LDM /lib/modules/$(uname -r)/build/.config

Results should be same as above.  If you get no such file or directory again then you need to install the relevant package.  On Ubuntu that would be "linux-headers-$(uname -r)" so you need to install that and then repeat the above.

Anyway if CONFIG_LDM_PARTITION=y is not set then you need to enable it and recompile the kernel and then hopefully it will work.  If it is enabled and not working then can you please provide the output of:

        fdisk -l -u /dev/sda
        parted /dev/sda print

Assuming /dev/sda is the disk, otherwise substitute the actual disk.

> Is accessing those volumes supported, and if not, is there any
> collection of documents I could use to implement support for that
> myself? An extended Google search only turned up high-level overviews
> and no real technical data about that disk management scheme.

LDM is supported yes.  But so far the LDM driver only looks at the MSDOS partition table and if there is no LDM partition defined there it bails out.  Hence my request for fdisk and parted output to see what your disk actually looks like.

And yes there is documentation about LDM.  First the LDM driver itself (linux-3.2/block/partitions/ldm.[ch]) and then the actual documentation as well as some useful tools for dumping the LDM database:

        http://sourceforge.net/projects/linux-ntfs/files/

However as I said support is already in the kernel so it is just a matter of getting it to work I imagine rather than having to resort to the documentation and writing a new driver...

Best regards,

        Anton

> Regards,
> Carl-Daniel
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Linux-NTFS-Dev mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev

Best regards,

        Anton
--
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
Reply | Threaded
Open this post in threaded view
|

Re: Windows 7 Dynamic Disks and LDM

Carl-Daniel Hailfinger-2
Hi Anton,

thanks for the quick response.

Am 20.07.2012 10:09 schrieb Anton Altaparmakov:
> On 20 Jul 2012, at 01:17, Carl-Daniel Hailfinger wrote:
>> I have trouble accessing Windows 7 Dynamic Disks under Linux 3.2. The
>> partitioning seems to be EFI-based with one big EFI partition apparently
>> containing all the volumes, but I can't see any block devices in Linux
>> for any of those volumes.
> First of all have you got LDM support compiled into your kernel?

Yes, CONFIG_LDM_PARTITION=y.


> If it is enabled and not working then can you please provide the output of:
>
> fdisk -l -u /dev/sda
> parted /dev/sda print
>
> Assuming /dev/sda is the disk, otherwise substitute the actual disk.

Will do so on Monday.

 
>> Is accessing those volumes supported, and if not, is there any
>> collection of documents I could use to implement support for that
>> myself? An extended Google search only turned up high-level overviews
>> and no real technical data about that disk management scheme.
> LDM is supported yes.  But so far the LDM driver only looks at the MSDOS partition table and if there is no LDM partition defined there it bails out.  Hence my request for fdisk and parted output to see what your disk actually looks like.
>
> And yes there is documentation about LDM.  First the LDM driver itself (linux-3.2/block/partitions/ldm.[ch]) and then the actual documentation as well as some useful tools for dumping the LDM database:
>
> http://sourceforge.net/projects/linux-ntfs/files/

Thanks for the pointers. I had assumed that the sourceforge pages were
not relevant anymore (last update in 2009), and unfortunately
http://www.linux-ntfs.org/ redirects to tuxera.com where a search for
LDM does not yield any usable results.


> However as I said support is already in the kernel so it is just a matter of getting it to work I imagine rather than having to resort to the documentation and writing a new driver...

Great, thanks!

Regards,
Carl-Daniel

--
http://www.hailfinger.org/


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
Reply | Threaded
Open this post in threaded view
|

Re: Windows 7 Dynamic Disks and LDM

Anton Altaparmakov
Hi,

I created one to have a look myself so no need to send me anything.

It looks like this:

Sector zero is a standard protective MBR/MSDOS style partition table with just one partition of type 0xEE (protective) which covers whole disk.

Then there is a full GPT partition table (what Linux calls EFI though that is a misnomer) which contains three partitions.  The LDM database partition which is exactly 2048 sectors in size (sector size 512 bytes).  The UUID for that is 5808C8AA-7E8F-42E0-85D2-E1E90434CFB3.  That is followed by the microsoft reserved partition which we can ignore as it is just reserved space.  And that is followed by the ldm data partition which stores the actual volumes/volume fragments.  The UUID for that is AF9B60A0-1431-4F62-BC68-3311714A69AD.

Then there are a few other difference between LDM on MBR and GPT.

On MBR there are three copies of the PRIVHEADS at sectors 6 of the disk and sectors 1856 and 2047 relative to start of LDM database.

On GPT there are two copies of the PRIVHEADS at sectors 1856 and 2047 in the LDM database partition.

The LDM database is 2048 sectors (1MiB) thus the LDM database partition is the equivalent of the LDM database on MBR which also explains why we don't need the third PRIVHEAD - we already know where the LDM database is as we have the location of the LDM database partition from GPT.

The PRIVHEADs then point at the primary and secondary TOCBLOCKs.  This is again different to previous style where there were four copies of the TOCBLOCKs and they were at sectors 1, 2, 2045, and 2046 in the LDM database on MBR.

On GPT, we have only two TOCBLOCKs and they are as described in the PRIVHEADs namely at sectors 2 and 2045 in the LDM database partition.  (Note this may not actually be a difference between MBR and GPT but a difference between LDM version 2.11 and 2.12, not sure.  I don't have Windows 2000 / XP any more so can't double check but in any case I don't think they supported GPT disks anyway.)

I haven't checked much beyond that but what I have checked looks identical to what we had before so I think just adapting the partition recognition code in Linux with the above described differences should be enough to make it work.

Note we do not need to worry about the LDM data partition because the PRIVHEADs already specify where the LDM data is located on the disk.  As a sanity check we could verify that the two match in both location and size.

Looks like the right solution will be to hook into the efi.c code in the kernel and check the found GPT partition UUIDs and when a LDM database partition is found call into the LDM code and there we need to make some changes to allow for the above described differences and then it will probably start working...

Best regards,

        Anton
--
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
Reply | Threaded
Open this post in threaded view
|

Re: Windows 7 Dynamic Disks and LDM

Carl-Daniel Hailfinger-2
Hi Anton,

wow! Thanks for investigating this in detail.

Am 21.07.2012 03:25 schrieb Anton Altaparmakov:

> I created one to have a look myself so no need to send me anything.
>
> It looks like this:
>
> Sector zero is a standard protective MBR/MSDOS style partition table with just one partition of type 0xEE (protective) which covers whole disk.
>
> Then there is a full GPT partition table (what Linux calls EFI though that is a misnomer) which contains three partitions.  The LDM database partition which is exactly 2048 sectors in size (sector size 512 bytes).  The UUID for that is 5808C8AA-7E8F-42E0-85D2-E1E90434CFB3.  That is followed by the microsoft reserved partition which we can ignore as it is just reserved space.  And that is followed by the ldm data partition which stores the actual volumes/volume fragments.  The UUID for that is AF9B60A0-1431-4F62-BC68-3311714A69AD.
>
> Then there are a few other difference between LDM on MBR and GPT.
> [...]
> Looks like the right solution will be to hook into the efi.c code in the kernel and check the found GPT partition UUIDs and when a LDM database partition is found call into the LDM code and there we need to make some changes to allow for the above described differences and then it will probably start working...

I could try to hook up the LDM code to the GPT code. I have hacked on
the kernel partitioning code in the past, it would be fun to revisit
that. My experience with LDM internals is zero, though, so I'll probably
hit a roadblock eventually.

AFAICS the LDM partition code is called before the MSDOS/MBR partition
code, but after the GPT partition code (see
block/partitions/check.c:check_part[]). If I understood your suggestion
correctly to hook into the efi.c code, the detection of LDM on GPT would
happen in a way that's different from how LDM is detected on MSDOS/MBR.
Is that what you meant?

There is also the question of naming the devices... hiding all GPT
partitions is a bit inconvenient if someone wants to access the GPT BIOS
Boot Partition or the GPT EFI System Partition (e.g. for installing a
bootloader). Device names like hdXldmN (hdaldm1) instead of hdXN (hda1)
are really clumsy, but I don't have a better suggestion.

Regards,
Carl-Daniel

--
http://www.hailfinger.org/


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
Reply | Threaded
Open this post in threaded view
|

Re: Windows 7 Dynamic Disks and LDM

Anton Altaparmakov
Hi,

On 21 Jul 2012, at 03:24, Carl-Daniel Hailfinger wrote:
> Hi Anton,
>
> wow! Thanks for investigating this in detail.

You are welcome.

> Am 21.07.2012 03:25 schrieb Anton Altaparmakov:
>> I created one to have a look myself so no need to send me anything.
>>
>> It looks like this:
>>
>> Sector zero is a standard protective MBR/MSDOS style partition table with just one partition of type 0xEE (protective) which covers whole disk.
>>
>> Then there is a full GPT partition table (what Linux calls EFI though that is a misnomer) which contains three partitions.  The LDM database partition which is exactly 2048 sectors in size (sector size 512 bytes).  The UUID for that is 5808C8AA-7E8F-42E0-85D2-E1E90434CFB3.  That is followed by the microsoft reserved partition which we can ignore as it is just reserved space.  And that is followed by the ldm data partition which stores the actual volumes/volume fragments.  The UUID for that is AF9B60A0-1431-4F62-BC68-3311714A69AD.
>>
>> Then there are a few other difference between LDM on MBR and GPT.
>> [...]
>> Looks like the right solution will be to hook into the efi.c code in the kernel and check the found GPT partition UUIDs and when a LDM database partition is found call into the LDM code and there we need to make some changes to allow for the above described differences and then it will probably start working...
>
> I could try to hook up the LDM code to the GPT code. I have hacked on
> the kernel partitioning code in the past, it would be fun to revisit
> that. My experience with LDM internals is zero, though, so I'll probably
> hit a roadblock eventually.

Sounds good.  Feel free to direct questions at me.  I will send you an attachment off list with the latest ldm documentation (much more complete than the one on SF.net - I will update the one on SF.net with this one but I am just waiting to hear back from the person who updated it).

> AFAICS the LDM partition code is called before the MSDOS/MBR partition
> code, but after the GPT partition code (see
> block/partitions/check.c:check_part[]).

Correct.

> If I understood your suggestion correctly to hook into the efi.c code, the detection of LDM on GPT would happen in a way that's different from how LDM is detected on MSDOS/MBR.
> Is that what you meant?

Yes.  I would stick it inside block/partitions/efi.c::efi_partition() something like this:

#ifdef CONFIG_LDM_PARTITION
        u64 ldm_db_start, ldm_db_size, ldm_data_start, ldm_data_size;
        bool is_ldm = false;
        bool skip_ldm = false;
#endif /* CONFIG_LDM_PARTITION */

        [snip]

#ifdef CONFIG_LDM_PARTITION
retry:
#endif /* CONFIG_LDM_PARTITION */

        for (i = 0; i < le32_to_cpu(gpt->num_partition_entries) && i < state->limit-1; i++) {
                struct partition_meta_info *info;
                unsigned label_count = 0;
                unsigned label_max;
                u64 start = le64_to_cpu(ptes[i].starting_lba);
                u64 size = le64_to_cpu(ptes[i].ending_lba) -
                           le64_to_cpu(ptes[i].starting_lba) + 1ULL;

                if (!is_pte_valid(&ptes[i], last_lba(state->bdev)))
                        continue;

#ifdef CONFIG_LDM_PARTITION
                /*
                 * LDM can be both on MBR and GPT partitioning scheme.
                 *
                 * We deal with MBR based LDM later on but we need to deal with
                 * GPT based LDM now.
                 */
                if (!skip_ldm) {
               if (!i && !efi_guidcmp(ptes[i].partition_type_guid,
                                         PARTITION_LDM_DATABASE)) {
                                ldm_db_start = start;
                                ldm_db_size = size;
                                is_ldm = true;
                                continue;
                        } else if (is_ldm) {
                                int ret;

                                /*
                                 * We need to skip partitions until we find the LDM
                                 * data partition.
                                 */
                                if (efi_guidcmp(ptes[i].partition_type_guid,
                        PARTITION_LDM_DATA))
                                        continue;
                                ret = ldm_gpt_partition(state, ldm_db_start,
                                        ldm_db_size, start, size);
                                if (ret == 1) {
                                        kfree(ptes);
                                        kfree(gpt);
                                        return ret;
                                }
                                /*
                                 * ldm_gpt_partition() failed.  Might as well treat the disk as
                                 * non-LDM / normal GPT.
                                 */
                                is_ldm = false;
                                skip_ldm = true;
                                goto retry;
                        }
                }
#endif /* CONFIG_LDM_PARTITION */

                put_partition(state, i+1, start * ssz, size * ssz);

                [snip]
        }

#ifdef CONFIG_LDM_PARTITION
        /*
         * If we found an LDM database partition but no matching LDM data partition
         * there is something very odd going on.  Log a message and re-process the
         * partitions as non-LDM.
         */
        if (is_ldm) {
                printk(KERN_WARNING "LDM database partition found on GPT disk "
                                "but no matching LDM data partition found.  "
                                "Ignoring LDM database partition.\n");
                is_ldm = false;
                skip_ldm = true;
                goto retry;
        }
#endif /* CONFIG_LDM_PARTITION */

Then in efi.h put in constants for the PARTITION_LDM_DATABASE 5808C8AA-7E8F-42E0-85D2-E1E90434CFB3 and the PARTITION_LDM_DATA AF9B60A0-1431-4F62-BC68-3311714A69AD and of course add a function to ldm.c (and efi.h I guess!) something like that:

int ldm_gpt_partition(struct parsed_partitions *state, const u64 ldm_db_start, const u64 ldm_db_size, const u64 ldm_data_start, const u64 ldm_data_size)...   Then that function can do all the LDM specific bits and add the devices with put_partition(), etc.  The only reason I both with the ldm_data_start and ldm_data_size is so it can be cross checked with

Or something like that anyway.  This is a bit ifdef ugly but I can't see a neater way of doing it without having to re-read the GPT in the ldm driver itself (and obviously moving the ldm driver to before efi.c in check.c).

> There is also the question of naming the devices... hiding all GPT
> partitions is a bit inconvenient if someone wants to access the GPT BIOS
> Boot Partition or the GPT EFI System Partition (e.g. for installing a
> bootloader). Device names like hdXldmN (hdaldm1) instead of hdXN (hda1)
> are really clumsy, but I don't have a better suggestion.

Not really. I doubt you would ever have a GPT BIOS partition or a GPT EFI System partition on a Windows Dynamic Disk.  The only partitions are the LDM database partition, the Microsoft reserved partition, and the LDM data partition which covers the entire disk (except for the little bits occupied by the LDM database partition and MS reserved partition).  So I don't think this is an issue as there will be no other partitions that you would want to expose.

Best regards,

        Anton

> Regards,
> Carl-Daniel

--
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
Reply | Threaded
Open this post in threaded view
|

Re: Windows 7 Dynamic Disks and LDM

Anton Altaparmakov
Hi,

On 21 Jul 2012, at 09:30, Anton Altaparmakov wrote:
> On 21 Jul 2012, at 03:24, Carl-Daniel Hailfinger wrote:
>> There is also the question of naming the devices... hiding all GPT
>> partitions is a bit inconvenient if someone wants to access the GPT BIOS
>> Boot Partition or the GPT EFI System Partition (e.g. for installing a
>> bootloader). Device names like hdXldmN (hdaldm1) instead of hdXN (hda1)
>> are really clumsy, but I don't have a better suggestion.
>
> Not really. I doubt you would ever have a GPT BIOS partition or a GPT EFI System partition on a Windows Dynamic Disk.  The only partitions are the LDM database partition, the Microsoft reserved partition, and the LDM data partition which covers the entire disk (except for the little bits occupied by the LDM database partition and MS reserved partition).  So I don't think this is an issue as there will be no other partitions that you would want to expose.


Hm.  I was wrong.  (-;

Just googled this incredibly useful FAQ from Microsoft:

        http://msdn.microsoft.com/en-us/library/windows/hardware/gg463525.aspx

It explains bootable dynamic disks.  Fortunately it explains that the EFI system partition (ESP) must be the first partition and same for any other non-LDM partitions.  So my example code needs changing so that it allows "efi.c" normal code to work (i.e. call put_partition(), etc) UNTIL an LDM database partition is found at which point it should switch to LDM handling and if that fails it should retry with pure GPT but only starting at the LDM database partition as the others are already done.

And the LDM database partition detection needs to change to no longer insist on "i" being zero as that clearly doesn't need to be the case.

So I think there is no need to change the names.  We just have hdX1, 2, 3, 4, etc.  where 1 is the ESP, 2..N are other partitions, N+1 is the first LDM partition, etc.  I think that works well.  So it treats LDM partitions effectively like extended partitions on MBR.  What do you think?  I think it is neater than having to do hdXldmY...

Best regards,

        Anton
--
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
Reply | Threaded
Open this post in threaded view
|

Re: Windows 7 Dynamic Disks and LDM

Anton Altaparmakov
Hi,

One further thought.  If we go down the route of allowing access to other partitions followed by access to LDM partitions in the same namespace, just with higher numbers then perhaps we should also add the LDM database partition as an exposed device.  It would make it much easier for an LDM userspace tool to work with it if it could simply be pointed at the LDM database by specifying it is on /dev/hdaN or something...  What do you think?

Best regards,

        Anton

On 21 Jul 2012, at 09:47, Anton Altaparmakov wrote:

> Hi,
>
> On 21 Jul 2012, at 09:30, Anton Altaparmakov wrote:
>> On 21 Jul 2012, at 03:24, Carl-Daniel Hailfinger wrote:
>>> There is also the question of naming the devices... hiding all GPT
>>> partitions is a bit inconvenient if someone wants to access the GPT BIOS
>>> Boot Partition or the GPT EFI System Partition (e.g. for installing a
>>> bootloader). Device names like hdXldmN (hdaldm1) instead of hdXN (hda1)
>>> are really clumsy, but I don't have a better suggestion.
>>
>> Not really. I doubt you would ever have a GPT BIOS partition or a GPT EFI System partition on a Windows Dynamic Disk.  The only partitions are the LDM database partition, the Microsoft reserved partition, and the LDM data partition which covers the entire disk (except for the little bits occupied by the LDM database partition and MS reserved partition).  So I don't think this is an issue as there will be no other partitions that you would want to expose.
>
>
> Hm.  I was wrong.  (-;
>
> Just googled this incredibly useful FAQ from Microsoft:
>
> http://msdn.microsoft.com/en-us/library/windows/hardware/gg463525.aspx
>
> It explains bootable dynamic disks.  Fortunately it explains that the EFI system partition (ESP) must be the first partition and same for any other non-LDM partitions.  So my example code needs changing so that it allows "efi.c" normal code to work (i.e. call put_partition(), etc) UNTIL an LDM database partition is found at which point it should switch to LDM handling and if that fails it should retry with pure GPT but only starting at the LDM database partition as the others are already done.
>
> And the LDM database partition detection needs to change to no longer insist on "i" being zero as that clearly doesn't need to be the case.
>
> So I think there is no need to change the names.  We just have hdX1, 2, 3, 4, etc.  where 1 is the ESP, 2..N are other partitions, N+1 is the first LDM partition, etc.  I think that works well.  So it treats LDM partitions effectively like extended partitions on MBR.  What do you think?  I think it is neater than having to do hdXldmY...
>
> Best regards,
>
> Anton
> --
> Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
> Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
> Linux NTFS maintainer, http://www.linux-ntfs.org/
>

Best regards,

        Anton
--
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
Reply | Threaded
Open this post in threaded view
|

Re: Windows 7 Dynamic Disks and LDM

Carl-Daniel Hailfinger-2
Hi Anton,

Am 21.07.2012 22:07 schrieb Anton Altaparmakov:
> One further thought.  If we go down the route of allowing access to other partitions followed by access to LDM partitions in the same namespace, just with higher numbers then perhaps we should also add the LDM database partition as an exposed device.  It would make it much easier for an LDM userspace tool to work with it if it could simply be pointed at the LDM database by specifying it is on /dev/hdaN or something...  What do you think?

Definitely a good idea to expose the database partition. That saves the
userspace tool from having to decode the GPT partition table again.
The only partition where I see potential trouble is the LDM data
container partition. Do you know if the LDM data container partition
(AF9B60A0-1431-4F62-BC68-3311714A69AD) has some free space at the
beginning before the first volume? If yes, we could expose the LDM data
container partition as well. If not, automatic filesystem type detection
tools (blkid etc.) in Linux would mistakenly detect the LDM data
container partition as a partition containing a filesystem, and might
even automount it with potentially disastrous effects.

Another semi-related question: Is there any way by which some Linux
distro partitioning tool (or whatever software people use to detect
filesystems and mount them) can tell that a volume is part of a
stripe/raid5/... set and should definitely not be mounted lest it
destroys the filesystem?

Regards,
Carl-Daniel

> On 21 Jul 2012, at 09:47, Anton Altaparmakov wrote:
>> On 21 Jul 2012, at 09:30, Anton Altaparmakov wrote:
>>> On 21 Jul 2012, at 03:24, Carl-Daniel Hailfinger wrote:
>>>> There is also the question of naming the devices... hiding all GPT
>>>> partitions is a bit inconvenient if someone wants to access the GPT BIOS
>>>> Boot Partition or the GPT EFI System Partition (e.g. for installing a
>>>> bootloader). Device names like hdXldmN (hdaldm1) instead of hdXN (hda1)
>>>> are really clumsy, but I don't have a better suggestion.
>>> Not really. I doubt you would ever have a GPT BIOS partition or a GPT EFI System partition on a Windows Dynamic Disk.  The only partitions are the LDM database partition, the Microsoft reserved partition, and the LDM data partition which covers the entire disk (except for the little bits occupied by the LDM database partition and MS reserved partition).  So I don't think this is an issue as there will be no other partitions that you would want to expose.
>>
>> Hm.  I was wrong.  (-;
>>
>> Just googled this incredibly useful FAQ from Microsoft:
>>
>> http://msdn.microsoft.com/en-us/library/windows/hardware/gg463525.aspx
>>
>> It explains bootable dynamic disks.  Fortunately it explains that the EFI system partition (ESP) must be the first partition and same for any other non-LDM partitions.  So my example code needs changing so that it allows "efi.c" normal code to work (i.e. call put_partition(), etc) UNTIL an LDM database partition is found at which point it should switch to LDM handling and if that fails it should retry with pure GPT but only starting at the LDM database partition as the others are already done.
>>
>> And the LDM database partition detection needs to change to no longer insist on "i" being zero as that clearly doesn't need to be the case.
>>
>> So I think there is no need to change the names.  We just have hdX1, 2, 3, 4, etc.  where 1 is the ESP, 2..N are other partitions, N+1 is the first LDM partition, etc.  I think that works well.  So it treats LDM partitions effectively like extended partitions on MBR.  What do you think?  I think it is neater than having to do hdXldmY...

--
http://www.hailfinger.org/


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
Reply | Threaded
Open this post in threaded view
|

Re: Windows 7 Dynamic Disks and LDM

Carl-Daniel Hailfinger-2
Hi Anton,

Am 22.07.2012 02:29 schrieb Carl-Daniel Hailfinger:

> Am 21.07.2012 22:07 schrieb Anton Altaparmakov:
>> One further thought. If we go down the route of allowing access to
>> other partitions followed by access to LDM partitions in the same
>> namespace, just with higher numbers then perhaps we should also add
>> the LDM database partition as an exposed device. It would make it
>> much easier for an LDM userspace tool to work with it if it could
>> simply be pointed at the LDM database by specifying it is on
>> /dev/hdaN or something... What do you think?
>
> Definitely a good idea to expose the database partition. That saves
> the userspace tool from having to decode the GPT partition table
> again. The only partition where I see potential trouble is the LDM
> data container partition. Do you know if the LDM data container
> partition (AF9B60A0-1431-4F62-BC68-3311714A69AD) has some free space
> at the beginning before the first volume? If yes, we could expose the
> LDM data container partition as well. If not, automatic filesystem
> type detection tools (blkid etc.) in Linux would mistakenly detect
> the LDM data container partition as a partition containing a
> filesystem, and might even automount it with potentially disastrous
> effects.
>
> Another semi-related question: Is there any way by which some Linux
> distro partitioning tool (or whatever software people use to detect
> filesystems and mount them) can tell that a volume is part of a
> stripe/raid5/... set and should definitely not be mounted lest it
> destroys the filesystem?

The more I think about it, the less convinced I am about implementing
this in the kernel. Windows LDM is essentially the same as Linux LVM.
You mentioned that there is a LDM userspace tool which can read the LDM
database. Would it be possible to have that tool read all LDM databases
(all disks, last megabyte on MBR and LDM database partition on GPT),
then output table files which can be fed into dmsetup create? We would
get RAID and all other fun stuff (volume spanning multiple disks etc.)
supported automatically with no manual assembly required, and the
process would be very similar to how Linux LVM works right now.

Regards,
Carl-Daniel
--
http://www.hailfinger.org/

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
Reply | Threaded
Open this post in threaded view
|

Re: Windows 7 Dynamic Disks and LDM

Anton Altaparmakov
In reply to this post by Carl-Daniel Hailfinger-2
Hi Carl-Daniel,

On 22 Jul 2012, at 01:29, Carl-Daniel Hailfinger wrote:

> Hi Anton,
>
> Am 21.07.2012 22:07 schrieb Anton Altaparmakov:
>> One further thought.  If we go down the route of allowing access to other partitions followed by access to LDM partitions in the same namespace, just with higher numbers then perhaps we should also add the LDM database partition as an exposed device.  It would make it much easier for an LDM userspace tool to work with it if it could simply be pointed at the LDM database by specifying it is on /dev/hdaN or something...  What do you think?
>
> Definitely a good idea to expose the database partition. That saves the
> userspace tool from having to decode the GPT partition table again.
> The only partition where I see potential trouble is the LDM data
> container partition. Do you know if the LDM data container partition
> (AF9B60A0-1431-4F62-BC68-3311714A69AD) has some free space at the
> beginning before the first volume? If yes, we could expose the LDM data
> container partition as well. If not, automatic filesystem type detection
> tools (blkid etc.) in Linux would mistakenly detect the LDM data
> container partition as a partition containing a filesystem, and might
> even automount it with potentially disastrous effects.

I do not see any point in exposing the LDM data partition.  What would anyone do with it?!?

Whether it have any free space at the front or not is I can only assume random depending on the contained volumes and alignments applied.  I don't think you can rely on there being free space.

> Another semi-related question: Is there any way by which some Linux
> distro partitioning tool (or whatever software people use to detect
> filesystems and mount them) can tell that a volume is part of a
> stripe/raid5/... set and should definitely not be mounted lest it
> destroys the filesystem?

I don't know.  Linux has traditionally always ignored the partition type (on GPT the partition GUID but still ignored by Linux) so I doubt there is anything we can do about that.

Best regards,

        Anton

>
> Regards,
> Carl-Daniel
>
>> On 21 Jul 2012, at 09:47, Anton Altaparmakov wrote:
>>> On 21 Jul 2012, at 09:30, Anton Altaparmakov wrote:
>>>> On 21 Jul 2012, at 03:24, Carl-Daniel Hailfinger wrote:
>>>>> There is also the question of naming the devices... hiding all GPT
>>>>> partitions is a bit inconvenient if someone wants to access the GPT BIOS
>>>>> Boot Partition or the GPT EFI System Partition (e.g. for installing a
>>>>> bootloader). Device names like hdXldmN (hdaldm1) instead of hdXN (hda1)
>>>>> are really clumsy, but I don't have a better suggestion.
>>>> Not really. I doubt you would ever have a GPT BIOS partition or a GPT EFI System partition on a Windows Dynamic Disk.  The only partitions are the LDM database partition, the Microsoft reserved partition, and the LDM data partition which covers the entire disk (except for the little bits occupied by the LDM database partition and MS reserved partition).  So I don't think this is an issue as there will be no other partitions that you would want to expose.
>>>
>>> Hm.  I was wrong.  (-;
>>>
>>> Just googled this incredibly useful FAQ from Microsoft:
>>>
>>> http://msdn.microsoft.com/en-us/library/windows/hardware/gg463525.aspx
>>>
>>> It explains bootable dynamic disks.  Fortunately it explains that the EFI system partition (ESP) must be the first partition and same for any other non-LDM partitions.  So my example code needs changing so that it allows "efi.c" normal code to work (i.e. call put_partition(), etc) UNTIL an LDM database partition is found at which point it should switch to LDM handling and if that fails it should retry with pure GPT but only starting at the LDM database partition as the others are already done.
>>>
>>> And the LDM database partition detection needs to change to no longer insist on "i" being zero as that clearly doesn't need to be the case.
>>>
>>> So I think there is no need to change the names.  We just have hdX1, 2, 3, 4, etc.  where 1 is the ESP, 2..N are other partitions, N+1 is the first LDM partition, etc.  I think that works well.  So it treats LDM partitions effectively like extended partitions on MBR.  What do you think?  I think it is neater than having to do hdXldmY...
>
> --
> http://www.hailfinger.org/
>

Best regards,

        Anton
--
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
Reply | Threaded
Open this post in threaded view
|

Re: Windows 7 Dynamic Disks and LDM

Anton Altaparmakov
In reply to this post by Carl-Daniel Hailfinger-2
Hi Carl-Daniel,

On 22 Jul 2012, at 02:34, Carl-Daniel Hailfinger wrote:
> The more I think about it, the less convinced I am about implementing
> this in the kernel. Windows LDM is essentially the same as Linux LVM.
> You mentioned that there is a LDM userspace tool which can read the LDM
> database. Would it be possible to have that tool read all LDM databases
> (all disks, last megabyte on MBR and LDM database partition on GPT),
> then output table files which can be fed into dmsetup create? We would
> get RAID and all other fun stuff (volume spanning multiple disks etc.)
> supported automatically with no manual assembly required, and the
> process would be very similar to how Linux LVM works right now.

I am afraid the device mapper / dmsetup is pretty useless in such a context.  I mean, yes you are right that you could read the LDM database and call dmsetup but dmsetup only supports concatenations and striping.  No mirroring and no RAID-5.  So that would not be a great solution as it would have no way of supporting mirroring and RAID-5 which are both used with dynamic disks.

It is worth pointing out that the current LDM driver only exposes the underlying LDM partitions, it does not do any of the LVM/RAID bits.  That is left for the user to do and indeed it is done using Software RAID (MD driver) using correct invocations of mdadm by hand by people who need to do it.  I don't see any particular reason not to continue to do so unless you want to put in all the extra work required to actually decode the volume descriptions and assemble them.  I certainly am far too busy to start doing something like that...

Moving LDM completely into userspace also has another problem for bootable MBR based dynamic disks.  There the bootable Windows partition is described both by the LDM database and by an entry in the MBR partition table.  So if you do not handle the disk in the kernel the standard MBR partition code in the kernel will create the partitions for the disk and the NTFS volume could even be mounted.  At the same time the userspace LDM would read the LDM database and create a new device also describing the same NTFS volume and then that could be mounted and suddenly you have the same file system being written to by two separate mount instances which would be a complete disaster.

And I can't see any way around this if LDM is not parsed in the kernel...

Best regards,

        Anton
--
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
Reply | Threaded
Open this post in threaded view
|

Re: Windows 7 Dynamic Disks and LDM

Anton Altaparmakov
On 22 Jul 2012, at 14:54, Anton Altaparmakov wrote:

> Hi Carl-Daniel,
>
> On 22 Jul 2012, at 02:34, Carl-Daniel Hailfinger wrote:
>> The more I think about it, the less convinced I am about implementing
>> this in the kernel. Windows LDM is essentially the same as Linux LVM.
>> You mentioned that there is a LDM userspace tool which can read the LDM
>> database. Would it be possible to have that tool read all LDM databases
>> (all disks, last megabyte on MBR and LDM database partition on GPT),
>> then output table files which can be fed into dmsetup create? We would
>> get RAID and all other fun stuff (volume spanning multiple disks etc.)
>> supported automatically with no manual assembly required, and the
>> process would be very similar to how Linux LVM works right now.
>
> I am afraid the device mapper / dmsetup is pretty useless in such a context.  I mean, yes you are right that you could read the LDM database and call dmsetup but dmsetup only supports concatenations and striping.  No mirroring and no RAID-5.  So that would not be a great solution as it would have no way of supporting mirroring and RAID-5 which are both used with dynamic disks.

Just re-read the dmsetup manpage and it says there is a mirror target though it is not described how to use it.  That still leaves no RAID-5 though which people do use.

Unless there is a RAID-5 extension like for mirroring?  The man page at least doesn't mention one.

Best regards,

        Anton


> It is worth pointing out that the current LDM driver only exposes the underlying LDM partitions, it does not do any of the LVM/RAID bits.  That is left for the user to do and indeed it is done using Software RAID (MD driver) using correct invocations of mdadm by hand by people who need to do it.  I don't see any particular reason not to continue to do so unless you want to put in all the extra work required to actually decode the volume descriptions and assemble them.  I certainly am far too busy to start doing something like that...
>
> Moving LDM completely into userspace also has another problem for bootable MBR based dynamic disks.  There the bootable Windows partition is described both by the LDM database and by an entry in the MBR partition table.  So if you do not handle the disk in the kernel the standard MBR partition code in the kernel will create the partitions for the disk and the NTFS volume could even be mounted.  At the same time the userspace LDM would read the LDM database and create a new device also describing the same NTFS volume and then that could be mounted and suddenly you have the same file system being written to by two separate mount instances which would be a complete disaster.
>
> And I can't see any way around this if LDM is not parsed in the kernel...
>
> Best regards,
>
> Anton

--
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
Reply | Threaded
Open this post in threaded view
|

Re: Windows 7 Dynamic Disks and LDM

Carl-Daniel Hailfinger-2
Hi Anton,

Am 22.07.2012 16:06 schrieb Anton Altaparmakov:

> On 22 Jul 2012, at 14:54, Anton Altaparmakov wrote:
>> On 22 Jul 2012, at 02:34, Carl-Daniel Hailfinger wrote:
>>> The more I think about it, the less convinced I am about implementing
>>> this in the kernel. Windows LDM is essentially the same as Linux LVM.
>>> You mentioned that there is a LDM userspace tool which can read the LDM
>>> database. Would it be possible to have that tool read all LDM databases
>>> (all disks, last megabyte on MBR and LDM database partition on GPT),
>>> then output table files which can be fed into dmsetup create? We would
>>> get RAID and all other fun stuff (volume spanning multiple disks etc.)
>>> supported automatically with no manual assembly required, and the
>>> process would be very similar to how Linux LVM works right now.
>> I am afraid the device mapper / dmsetup is pretty useless in such a context.  I mean, yes you are right that you could read the LDM database and call dmsetup but dmsetup only supports concatenations and striping.  No mirroring and no RAID-5.  So that would not be a great solution as it would have no way of supporting mirroring and RAID-5 which are both used with dynamic disks.
> Just re-read the dmsetup manpage and it says there is a mirror target though it is not described how to use it.  That still leaves no RAID-5 though which people do use.
>
> Unless there is a RAID-5 extension like for mirroring?  The man page at least doesn't mention one.

Admittedly the documentation is not really up-to-date nor particularly
helpful. From
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=drivers/md/dm-raid.c
"raid1",    "RAID1 (mirroring)"
<http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=drivers/md/dm-raid.c#l79>"raid4",    "RAID4 (dedicated parity disk)"
<http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=drivers/md/dm-raid.c#l80>"raid5_la", "RAID5 (left asymmetric)"
<http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=drivers/md/dm-raid.c#l81>"raid5_ra", "RAID5 (right asymmetric)"
<http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=drivers/md/dm-raid.c#l82>"raid5_ls", "RAID5 (left symmetric)"
<http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=drivers/md/dm-raid.c#l83>"raid5_rs", "RAID5 (right symmetric)"
<http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=drivers/md/dm-raid.c#l84>"raid6_zr", "RAID6 (zero restart)"
<http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=drivers/md/dm-raid.c#l85>"raid6_nr", "RAID6 (N restart)"
<http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=drivers/md/dm-raid.c#l86>"raid6_nc", "RAID6 (N continue)"
<http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=drivers/md/dm-raid.c#l87>There
also are pending patches to support RAID10, so I think most bases are
covered.

I have written dmsetup tables by hand in the past (and wrote some
generator scripts which handle split image concatenation etc.), and I
think I could do the same to support most LDM configurations.


>> It is worth pointing out that the current LDM driver only exposes the underlying LDM partitions, it does not do any of the LVM/RAID bits.  That is left for the user to do and indeed it is done using Software RAID (MD driver) using correct invocations of mdadm by hand by people who need to do it.  I don't see any particular reason not to continue to do so unless you want to put in all the extra work required to actually decode the volume descriptions and assemble them.  I certainly am far too busy to start doing something like that...

Ah OK, I had assumed that decoding the volume descriptions was already
available with the LDM userspace tool. Is there some cvs/svn/git/hg tree
for the LDM userspace tools you mentioned?

While I'm busy as well, I think I may have enough time to attempt
automatic assembling of volumes if I get decoded descriptions from
somewhere.

 
>> Moving LDM completely into userspace also has another problem for bootable MBR based dynamic disks.  There the bootable Windows partition is described both by the LDM database and by an entry in the MBR partition table.  So if you do not handle the disk in the kernel the standard MBR partition code in the kernel will create the partitions for the disk and the NTFS volume could even be mounted.  At the same time the userspace LDM would read the LDM database and create a new device also describing the same NTFS volume and then that could be mounted and suddenly you have the same file system being written to by two separate mount instances which would be a complete disaster.
>> And I can't see any way around this if LDM is not parsed in the kernel...

IIRC Linux device-mapper marks any slave device nodes as in-use, and it
also checks whether the accessed device node is already mounted, so in
theory that should be caught by existing sanity checks.

Thanks again for all the helpful info!

Regards,
Carl-Daniel

--
http://www.hailfinger.org/


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
Reply | Threaded
Open this post in threaded view
|

Re: Windows 7 Dynamic Disks and LDM

Anton Altaparmakov
Hi Carl-Daniel,

On 22 Jul 2012, at 22:04, Carl-Daniel Hailfinger wrote:
Ah, ok, I didn't realize that it does all of that, too!  In that case indeed it can do everything LDM can.

> I have written dmsetup tables by hand in the past (and wrote some
> generator scripts which handle split image concatenation etc.), and I
> think I could do the same to support most LDM configurations.

Cool.

>>> It is worth pointing out that the current LDM driver only exposes the underlying LDM partitions, it does not do any of the LVM/RAID bits.  That is left for the user to do and indeed it is done using Software RAID (MD driver) using correct invocations of mdadm by hand by people who need to do it.  I don't see any particular reason not to continue to do so unless you want to put in all the extra work required to actually decode the volume descriptions and assemble them.  I certainly am far too busy to start doing something like that...
>
> Ah OK, I had assumed that decoding the volume descriptions was already
> available with the LDM userspace tool. Is there some cvs/svn/git/hg tree
> for the LDM userspace tools you mentioned?

It's a bit complicated.  There are two tools.  I hacked one of them in a few minutes so I could run it against the LDM database itself when exposed as its own partition or when dumped to a file.  However that does not display any information about how sub-component volumes connect together.  That is the tool ldminfo in the ldminfo directory in the tar ball I am sending you off list.

The other tool I cannot get to compile easily on Mac OS X.  I haven't used it in ages but IIRC it does decode a lot more useful information from the LDM database.  It is in the test directory in the tar ball.  (This tool is also called ldminfo.)  Hopefully you can get it to build on Linux more easily...

> While I'm busy as well, I think I may have enough time to attempt
> automatic assembling of volumes if I get decoded descriptions from
> somewhere.

I guess in that case it will depend on how much time you need to invest in getting test/ldminfo to compile on a modern Linux distro...

>>> Moving LDM completely into userspace also has another problem for bootable MBR based dynamic disks.  There the bootable Windows partition is described both by the LDM database and by an entry in the MBR partition table.  So if you do not handle the disk in the kernel the standard MBR partition code in the kernel will create the partitions for the disk and the NTFS volume could even be mounted.  At the same time the userspace LDM would read the LDM database and create a new device also describing the same NTFS volume and then that could be mounted and suddenly you have the same file system being written to by two separate mount instances which would be a complete disaster.
>>> And I can't see any way around this if LDM is not parsed in the kernel...
>
> IIRC Linux device-mapper marks any slave device nodes as in-use, and it
> also checks whether the accessed device node is already mounted, so in
> theory that should be caught by existing sanity checks.

Ok, cool.  That sounds good.

> Thanks again for all the helpful info!

You are welcome!

Best regards,

        Anton

> Regards,
> Carl-Daniel

--
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
Reply | Threaded
Open this post in threaded view
|

Re: Windows 7 Dynamic Disks and LDM

Carl-Daniel Hailfinger-2
Hi Anton,

Am 22.07.2012 23:18 schrieb Anton Altaparmakov:
> On 22 Jul 2012, at 22:04, Carl-Daniel Hailfinger wrote:
>>>> It is worth pointing out that the current LDM driver only exposes the underlying LDM partitions, it does not do any of the LVM/RAID bits.  That is left for the user to do and indeed it is done using Software RAID (MD driver) using correct invocations of mdadm by hand by people who need to do it.  I don't see any particular reason not to continue to do so unless you want to put in all the extra work required to actually decode the volume descriptions and assemble them.  I certainly am far too busy to start doing something like that...
>> Ah OK, I had assumed that decoding the volume descriptions was already
>> available with the LDM userspace tool. Is there some cvs/svn/git/hg tree
>> for the LDM userspace tools you mentioned?
> It's a bit complicated.  There are two tools.  I hacked one of them in a few minutes so I could run it against the LDM database itself when exposed as its own partition or when dumped to a file.  However that does not display any information about how sub-component volumes connect together.  That is the tool ldminfo in the ldminfo directory in the tar ball I am sending you off list.

Yes, I got that one to work.


> The other tool I cannot get to compile easily on Mac OS X.  I haven't used it in ages but IIRC it does decode a lot more useful information from the LDM database.  It is in the test directory in the tar ball.  (This tool is also called ldminfo.)  Hopefully you can get it to build on Linux more easily...

I invested a few hours to get it to compile, and frankly, it's not that
easy.  Besides that, its code is missing a few fixes present in the
mainline Linux kernel sources. I'm tempted to forward-port the changes
and test-boot a kernel with it. That might be easier than getting the
code to compile in userspace (which may have worked at the time of
2.6.0, but not that much later). I can still work on userspace
compilation once I've seen the code work.
That said, do you know why the code hooking up LDM with MD was never
merged to mainline?

Regards,
Carl-Daniel

--
http://www.hailfinger.org/


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
Reply | Threaded
Open this post in threaded view
|

Re: Windows 7 Dynamic Disks and LDM

Anton Altaparmakov
Hi Carl-Daniel,

On 7 Aug 2012, at 21:37, Carl-Daniel Hailfinger <[hidden email]> wrote:

> Hi Anton,
>
> Am 22.07.2012 23:18 schrieb Anton Altaparmakov:
>> On 22 Jul 2012, at 22:04, Carl-Daniel Hailfinger wrote:
>>>>> It is worth pointing out that the current LDM driver only exposes the underlying LDM partitions, it does not do any of the LVM/RAID bits.  That is left for the user to do and indeed it is done using Software RAID (MD driver) using correct invocations of mdadm by hand by people who need to do it.  I don't see any particular reason not to continue to do so unless you want to put in all the extra work required to actually decode the volume descriptions and assemble them.  I certainly am far too busy to start doing something like that...
>>> Ah OK, I had assumed that decoding the volume descriptions was already
>>> available with the LDM userspace tool. Is there some cvs/svn/git/hg tree
>>> for the LDM userspace tools you mentioned?
>> It's a bit complicated.  There are two tools.  I hacked one of them in a few minutes so I could run it against the LDM database itself when exposed as its own partition or when dumped to a file.  However that does not display any information about how sub-component volumes connect together.  That is the tool ldminfo in the ldminfo directory in the tar ball I am sending you off list.
>
> Yes, I got that one to work.

Great.

>> The other tool I cannot get to compile easily on Mac OS X.  I haven't used it in ages but IIRC it does decode a lot more useful information from the LDM database.  It is in the test directory in the tar ball.  (This tool is also called ldminfo.)  Hopefully you can get it to build on Linux more easily...
>
> I invested a few hours to get it to compile, and frankly, it's not that
> easy.

Indeed.  I know that...  But it is the more useful tool.  The alternative is to write your own...

> Besides that, its code is missing a few fixes present in the
> mainline Linux kernel sources. I'm tempted to forward-port the changes
> and test-boot a kernel with it.

I am confused a bit.  The interesting bit is the "test" directory which is not something you can boot a kernel with as it is just a userspace tool that dumps the database.  It just happens to re-use the kernel driver code but running in userspace (test/compat.c reimplements needed kernel functions in user space).

> That might be easier than getting the
> code to compile in userspace (which may have worked at the time of
> 2.6.0, but not that much later). I can still work on userspace
> compilation once I've seen the code work.
> That said, do you know why the code hooking up LDM with MD was never
> merged to mainline?

What code?

/me looks at the contents of the linux/fs/partitions directory.  *Boggle*  I had no idea that code existed / was there...  I thought Jakob wrote the ldmutil/ldminfo.  I had no idea he had extended the kernel driver, too.

Looking at it very quickly it seems to only support very simple forms of raid.  It really should support the other types, too...

But in general I had no idea that code was there, or at least I have no recollection of it.

Best regards,

        Anton

> Regards,
> Carl-Daniel

--
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
Reply | Threaded
Open this post in threaded view
|

Re: Windows 7 Dynamic Disks and LDM

Carl-Daniel Hailfinger-2
Am 07.08.2012 23:50 schrieb Anton Altaparmakov:
> On 7 Aug 2012, at 21:37, Carl-Daniel Hailfinger <[hidden email]> wrote:
>> Am 22.07.2012 23:18 schrieb Anton Altaparmakov:
>>> The other tool I cannot get to compile easily on Mac OS X.  I haven't used it in ages but IIRC it does decode a lot more useful information from the LDM database.  It is in the test directory in the tar ball.  (This tool is also called ldminfo.)  Hopefully you can get it to build on Linux more easily...
>> I invested a few hours to get it to compile, and frankly, it's not that
>> easy.
> Indeed.  I know that...  But it is the more useful tool.  The alternative is to write your own...

That tool only compiles if you have a really old Linux kernel around. I
tried with Linux 2.6.10 and that one was too new. Older Linux kernels
don't compile with my current gcc, so I'll have to do some more
massaging. I'm making progress, though.


>> Besides that, its code is missing a few fixes present in the
>> mainline Linux kernel sources. I'm tempted to forward-port the changes
>> and test-boot a kernel with it.
> I am confused a bit.  The interesting bit is the "test" directory which is not something you can boot a kernel with as it is just a userspace tool that dumps the database.  It just happens to re-use the kernel driver code but running in userspace (test/compat.c reimplements needed kernel functions in user space).
>
>> That might be easier than getting the
>> code to compile in userspace (which may have worked at the time of
>> 2.6.0, but not that much later). I can still work on userspace
>> compilation once I've seen the code work.
>> That said, do you know why the code hooking up LDM with MD was never
>> merged to mainline?
> What code?
>
> /me looks at the contents of the linux/fs/partitions directory.  *Boggle*  I had no idea that code existed / was there...  I thought Jakob wrote the ldmutil/ldminfo.  I had no idea he had extended the kernel driver, too.

That kernel extension is for Linux 2.5.0 (yes, it was written in 2001)
and apparently it worked back then. The compilation dependencies are fun.


> Looking at it very quickly it seems to only support very simple forms of raid.  It really should support the other types, too...
>
> But in general I had no idea that code was there, or at least I have no recollection of it.

Yes, that code was never posted anywhere except linux-ntfs CVS and maybe
the linux-ntfs mailing list. Jakob seems to have disappeared from the
net a few years later, and I didn't have any success contacting him.

Anyway, I think I'm close to getting this tool to compile, and will
report back soon.

Regards,
Carl-Daniel

--
http://www.hailfinger.org/


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
Reply | Threaded
Open this post in threaded view
|

Re: Windows 7 Dynamic Disks and LDM

Carl-Daniel Hailfinger-2
Am 09.08.2012 21:03 schrieb Carl-Daniel Hailfinger:

> Am 07.08.2012 23:50 schrieb Anton Altaparmakov:
>> On 7 Aug 2012, at 21:37, Carl-Daniel Hailfinger <[hidden email]> wrote:
>>> Am 22.07.2012 23:18 schrieb Anton Altaparmakov:
>>>> The other tool I cannot get to compile easily on Mac OS X.  I haven't used it in ages but IIRC it does decode a lot more useful information from the LDM database.  It is in the test directory in the tar ball.  (This tool is also called ldminfo.)  Hopefully you can get it to build on Linux more easily...
>>> I invested a few hours to get it to compile, and frankly, it's not that
>>> easy.
>> Indeed.  I know that...  But it is the more useful tool.  The alternative is to write your own...
> That tool only compiles if you have a really old Linux kernel around. I
> tried with Linux 2.6.10 and that one was too new. Older Linux kernels
> don't compile with my current gcc, so I'll have to do some more
> massaging. I'm making progress, though.

Short update:
To compile the tool, you need:
linux-2.6.10 sources, configured for your native architecture (it's one
of the very few 2.6 versions which even start to build with current
gcc/gnumake), and which has started to build (you can abort after header
and symlink generation).
Point the include path in the main ldm Makefile to those linux-2.6.10
sources.
Fix hardcoded i386 target architecture for test/ldminfo compilation.
Forward-port the test/ldminfo code from Linux 2.5.something to Linux
2.6.0, enjoy the first working compile.
Forward-port to 2.6.10, notice that the code can't deal with LDM created
by Vista or newer. Forward-port to 2.6.26 (but keep pointing the include
path to the configured linux-2.6.10 sources, otherwise you'll get
compile errors).
Add/change lots of stuff in test/extra.c.
Get the following output for a Windows 7 dynamic disk with MBR (not GPT)
partitioning scheme:

# test/ldminfo /dev/hda
ldm_parse_tocblock(): Cannot find TOCBLOCK, database may be corrupt.
ldm_parse_tocblock(): Cannot find TOCBLOCK, database may be corrupt.
ldm_parse_vol5(): len 83 > BE32(buffer + 0x14) 82
ldm_parse_vblk(): Failed to parse VBLK 0xc (type: 0x51).
ldm_partition(): Failed to read the VBLKs from the database.
Something went wrong, skipping device '/dev/sda'


I have not yet managed to get the code to work based on a 2.6.36 or
later ldm.c.
My 3.1.10 kernel still complains about the TOCBLOCK during boot (LDM
debugging is disabled in that kernel, though), so rebasing to 3.1 won't
be enough.
Next step: Rebase the code to ldm.c from Linux 3.5.

Do you have any idea what might cause the TOCBLOCK problems?


Regards,
Carl-Daniel

--
http://www.hailfinger.org/


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
Reply | Threaded
Open this post in threaded view
|

Re: Windows 7 Dynamic Disks and LDM

Carl-Daniel Hailfinger-2
Hi Anton,

I'm making progress... slower than I hoped, but still...

Right now the code is at the level of Linux 2.6.30, and I have made
progress with 2.6.36 (compiles, have to check if it works). I also got
rid of the Linux 2.6.10 kernel header requirement, now you only need one
kernel version to compile the code.

The output below is still present with my current sources. Any insight
would be appreciated.

Am 11.08.2012 00:10 schrieb Carl-Daniel Hailfinger:

> Get the following output for a Windows 7 dynamic disk with MBR (not GPT)
> partitioning scheme:
>
> # test/ldminfo /dev/hda
> ldm_parse_tocblock(): Cannot find TOCBLOCK, database may be corrupt.
> ldm_parse_tocblock(): Cannot find TOCBLOCK, database may be corrupt.
> ldm_parse_vol5(): len 83 > BE32(buffer + 0x14) 82
> ldm_parse_vblk(): Failed to parse VBLK 0xc (type: 0x51).
> ldm_partition(): Failed to read the VBLKs from the database.
> Something went wrong, skipping device '/dev/sda'
>
>
> My 3.1.10 kernel still complains about the TOCBLOCK during boot (LDM
> debugging is disabled in that kernel, though), so rebasing to 3.1 won't
> be enough.
>
> Do you have any idea what might cause the TOCBLOCK problems?

Regards,
Carl-Daniel

--
http://www.hailfinger.org/


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Linux-NTFS-Dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
12