Skip to content

Instantly share code, notes, and snippets.

@darwin

darwin/readme.md

Last active April 9, 2024 22:30
Download ZIP

Revisions

  1. darwin revised this gist Nov 15, 2019. No changes.
  2. darwin created this gist Nov 15, 2019.
    369 changes: 369 additions & 0 deletions readme.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,369 @@
    Today I wanted to move existing APFS-resident macOS Catalina installation to a new disk. I upgraded my late 2014 Mac Mini with
    a shiny new 1TB SSD. This took way too many hours of my life I will never get back. Hope this saves some time to you.

    Good news:
    1. it is possible to create a DMG image from existing APFS container with macOS Catalina installation including metadata
    needed for complete restore (the DMG contains OS, OS Data, Preboot, Recovery and VM volumes)
    2. it is possible to restore this DMG image into empty APFS container and get a bootable copy of the original system

    This information is relevant for Catalina (I'm currently running macOS 10.15.1).

    There are major tricks how to make it work, the second one quite recent and third one is its consequence:

    #### Trick 1

    Source of restore operation must be a synthetised disk mounted from the DMG. Not the DMG file itself!

    #### Trick 2

    "[APFS Volume Groups](https://bombich.com/kb/ccc5/working-apfs-volume-groups)", a new feature introduced in Catalina does not
    play well with current `Disk Utility.app`, so you have to drop to command-line and force using legacy method of restoring the
    container.

    #### Trick 3

    Source DMG must not contain any APFS Volume with APFS snapshots.

    ---

    ## Let's create the source DMG

    This can be done comfortably from `Disk Utility.app`. You must be booted into some other system than your main system. In the
    left sidebar of `Disk Utility.app` switch to "Show All Devices". And then right-click the whole APFS container containing your
    system. It should have a name like "container diskN" and none of its volumes should be currently mounted. In the context menu
    you should see an available option to "Image from container diskN".

    Wait, hold your horses. Before making image you might consider deleting all existing snapshots in all APFS Volumes in the
    container. It will save you headaches later. Better finish reading the article first.

    > Can I boot into RecoveryOS to do this DMG image?
    No. RecoveryOS resides in one of APFS Volumes in the container you want to image so this is not possible, I think (not tested).

    > Oh, what to do if I don't have a secondary system?
    I would recommend using a recent macOS system installed on an external drive and boot from there.
    Alternatively, you can downsize the existing APFS container, create a new APFS container on your main disk and install a
    secondary macOS there and boot into it. This is quite easily doable just by clicking in `Disk Utility.app` if you have enough
    space on the main drive and a macOS installer at hand.

    ## Clone (restore) the source container

    `Disk Utility.app` seems to be using `asr` command under the hood. You can `man asr` to read the docs, especially the sections
    about APFS restore.

    When restoring you specify source and target. `asr` supports different ways how to specify source and target to cover
    scenarios related to previous filesystems. Source could be a DMG image file path or some existing disk in the form of
    `/dev/diskN`.

    One might expect that specifying a DMG image file as source and some existing empty APFS container as target should do the
    right thing. No! You get cryptic errors. The trick is to:

    1. first, mount your DMG file => it creates a synthetised disk
    2. second, figure out which `/dev/diskN` corresponds to the `synthetised` disk created in step 1
    3. finally, use this synthetised disk as source for your `asr` restore operation

    Make sense?

    Let's be more concrete with my case. My DMG file mounted as `disk9` which created synthetised `disk10`. As a target I created
    an empty APFS container in `disk0` with single empty APFS volume via `Disk Utility.app`. The synthetised target container had
    identifier `/dev/disk1`:

    ```
    > diskutil list
    /dev/disk0 (internal, physical):
    #: TYPE NAME SIZE IDENTIFIER
    0: GUID_partition_scheme *1.0 TB disk0
    1: EFI EFI 209.7 MB disk0s1
    2: Apple_APFS Container disk1 1000.0 GB disk0s2
    /dev/disk1 (synthesized):
    #: TYPE NAME SIZE IDENTIFIER
    0: APFS Container Scheme - +1000.0 GB disk1
    Physical Store disk0s2
    1: APFS Volume NewEmptyOS
    ...
    /dev/disk9 (disk image):
    #: TYPE NAME SIZE IDENTIFIER
    0: +499.2 GB disk9
    /dev/disk10 (synthesized):
    #: TYPE NAME SIZE IDENTIFIER
    0: APFS Container Scheme - +499.2 GB disk10
    Physical Store disk9
    1: APFS Volume MinimeOS - Data 311.1 GB disk10s1
    2: APFS Volume Preboot 82.2 MB disk10s2
    3: APFS Volume Recovery 528.5 MB disk10s3
    4: APFS Volume VM 4.3 GB disk10s4
    5: APFS Volume MinimeOS 10.9 GB disk10s5
    ```

    To execute restore I had to use `asr` command from command-line:

    ```bash
    > sudo asr restore --source /dev/disk10 --target /dev/disk1
    ...
    Volume replication failed - Read-only file system
    ```

    The `--useInverter` is the tricky option here. Replication functionality which is enabled by default is quite recent and it
    does not play well with APFS Volume Groups. It fails when trying to do something with read-only system volume:

    I consider this to be a bug and to be resolved in some future macOS version. Unfortunately `Disk Utilty.app` seem to be using
    `asr` without the `--useInverter` flag and gives me the same error in the GUI when trying to restore mounted DMG into existing
    container.

    So I ran it and got:

    ```
    > sudo asr restore --source /dev/disk10 --target /dev/disk1 --useInverter
    ...
    APFS inverter failed to invert the volume - Invalid argument
    ```

    Uh, oh!? Trying to run the restore again with `--verbose` and `--debug` flag gave me a hint:

    ```
    > sudo asr restore --source /dev/disk10 --target /dev/disk1 --useInverter --verbose --debug
    ...
    *** Mounting inner volume (ContainerToInvert)...
    ...
    mount_inner_volume:884: Inner volume has snapshots
    APFS inverter failed to invert the volume - Invalid argument
    ```

    Ah, now I remember that the man page stated that `--useInverter` does not work with snapshots.
    The error does not tell me which volume has this issue but sure I can figure this out:

    ```
    > diskutil apfs listSnapshots disk10s1
    Snapshots for disk10s1 (20 found)
    |
    +-- 9801FBE4-E581-4B95-A055-8A94A9C2ABE9
    | Name: com.apple.TimeMachine.2019-11-13-000654.local
    | XID: 2200904
    | Purgeable: Yes
    | NOTE: This snapshot limits the minimum size of APFS Container disk10
    |
    ...
    |
    +-- 46C315AB-6014-4275-AF94-B284E476718A
    | Name: com.apple.TimeMachine.2019-11-13-220854.local
    | XID: 2211035
    | Purgeable: Yes
    |
    +-- 72FE9F27-A358-42F0-91CE-CBC0E8140B46
    Name: com.apple.TimeMachine.2019-11-13-231202.local
    XID: 2212680
    Purgeable: Yes
    > diskutil apfs listSnapshots disk10s5
    Snapshots for disk10s5 (3 found)
    |
    +-- B07AE72B-1EE9-4145-827F-CB31E06635BB
    | Name: com.apple.TimeMachine.2019-11-13-211042.local
    | XID: 2210200
    | Purgeable: Yes
    | NOTE: This snapshot limits the minimum size of APFS Container disk10
    |
    +-- D124E17B-B061-4143-B69D-6626E5E8B41A
    | Name: com.apple.TimeMachine.2019-11-13-220854.local
    | XID: 2211035
    | Purgeable: Yes
    |
    +-- 1430DB4E-6527-4FA3-B226-77006055CA99
    Name: com.apple.TimeMachine.2019-11-13-231202.local
    XID: 2212676
    Purgeable: Yes
    > diskutil apfs listSnapshots disk10s4
    No snapshots for disk10s4
    > diskutil apfs listSnapshots disk10s3
    No snapshots for disk10s3
    > diskutil apfs listSnapshots disk10s2
    No snapshots for disk10s2
    ```

    So the Data volume `disk10s1` and OS volume `disk10s5` seem to have some snapshots created by TimeMachine.

    My task now is to delete them and try to run restore again. The problem is that my `disk10` disks are read-only because they
    were mounted from read-only DMG.

    ... few hours of banging my head into a wall

    Ok, I will spare you and won't describe all my trial and errors here.

    Finally I found a way how to strip all snapshots from `disk10s1`:

    ```
    > hdiutil attach -owners on /Volumes/W1/backups/mos.dmg -shadow
    expected CRC32 $0A545D85
    /dev/disk11
    /dev/disk12 EF57347C-0000-11AA-AA11-0030654
    /dev/disk12s1 41504653-0000-11AA-AA11-0030654 /Volumes/MinimeOS - Data 1
    /dev/disk12s2 41504653-0000-11AA-AA11-0030654 /Volumes/Preboot 1
    /dev/disk12s3 41504653-0000-11AA-AA11-0030654 /Volumes/Recovery 1
    /dev/disk12s4 41504653-0000-11AA-AA11-0030654 /Volumes/VM 1
    /dev/disk12s5 41504653-0000-11AA-AA11-0030654 /Volumes/MinimeOS 1
    ```

    The magical `-shadow` option mounts the image as read-write and keeps diff of changes in `/Volumes/W1/backups/mos.dmg.shadow`
    file. So if you try this at home make sure you have enough space available for the shadow file. Not sure if `-owners on` is
    important but this worked for me after some previous problems. Also note that an alternative solution would be to use
    `hdiutil convert -format UDRW ...` and continue with converted read-write image. See the Q/A section below.

    Anyways, having mounted writable disk allowed me to enumerate all snapshots and delete them:
    ```bash
    # this is just a dry-run to see all snapshot GUIDs to be deleted
    > diskutil apfs listsnapshots /dev/disk12s1 | grep "+--" | cut -d" " -f2
    9801FBE4-E581-4B95-A055-8A94A9C2ABE9
    DA44B969-EBA6-4EFE-ADCA-B4BB43FF6B37
    8AEFF1F4-BC35-4C05-A21C-3122A97569B4
    0149B1B9-9095-4896-B883-CD976802E419
    A6E252E6-2007-4DD5-9313-EE2156C0C813
    F3C38242-8AF7-42A8-9EC6-CF6381F647C0
    D6391D9B-4ADB-44D4-9F05-C58FABC730AD
    9281E5D8-867D-423D-A95F-7C4E3B4F34E2
    82EA6CA1-FB41-40DC-8F80-B8C34311E491
    EEE95161-ABFA-4E99-B893-2887DF4EC317
    779FAA27-DACD-41C2-A408-E731DC6E96A1
    36E7F342-9E5E-43F4-BA45-0441F9067244
    8359CA22-79A6-43DF-A265-35BEB8D4FC7E
    70105B62-19B0-4F14-9FFE-E73C6E32C74A
    3C581209-FC4B-4D14-9491-95AE656B3335
    068E2DD8-B6CF-454E-A5CC-8E68261EC9A6
    422788AF-2A6E-40D6-976D-CC3F93A925CA
    B647F1C2-E4CF-4F90-8A77-3C1CB2B992C9
    ```

    ```bash
    # delete them all
    > diskutil apfs listsnapshots /dev/disk12s1 | grep "+--" | cut -d" " -f2 | xargs -I{} sudo diskutil apfs deleteSnapshot disk12s1 -uuid {}
    Deleting APFS Snapshot DA44B969-EBA6-4EFE-ADCA-B4BB43FF6B37 "com.apple.TimeMachine.2019-11-13-011547.local" from APFS Volume disk10s1
    Started APFS operation
    Finished APFS operation
    Deleting APFS Snapshot 8AEFF1F4-BC35-4C05-A21C-3122A97569B4 "com.apple.TimeMachine.2019-11-13-030651.local" from APFS Volume disk10s1
    Started APFS operation
    Finished APFS operation
    ...
    Deleting APFS Snapshot B647F1C2-E4CF-4F90-8A77-3C1CB2B992C9 "com.apple.TimeMachine.2019-11-13-211042.local" from APFS Volume disk10s1
    Started APFS operation
    Finished APFS operation
    ```

    To double-check the result, I ran:
    ```bash
    > diskutil apfs listsnapshots /dev/disk12s1
    No snapshots for disk12s1
    ```

    I repeated the same snapshot purging with `disk12s5` and confirmed:
    ```bash
    > diskutil apfs listsnapshots /dev/disk12s5
    No snapshots for disk12s5
    ```

    Ok, hopefully there are no snapshots in any volume in whole container and I can run the restore again using the massaged `disk12`:
    ```
    > sudo asr restore --source /dev/disk10 --target /dev/disk1 --useInverter
    Validating target...done
    Validating source...done
    Validating sizes...done
    Restoring ....10....20....30....40....50....60....70....80....90....100
    Verifying ....10....20....30....40....50....60....70....80....90....100
    Inverting target volume...done
    Restoring ....10....20....30....40....50....60....70....80....90....100
    Verifying ....10....20....30....40....50....60....70....80....90....100
    Inverting target volume...done
    Restored target device is /dev/disk1s2.
    ```

    The disk is bootable and seems to be 1:1 copy of the original. Sure not really a device-level block-by-block copy, but good
    enough.

    ## It was easy, any questions?

    > How do I mount the DMG?
    #### Case 1: if your DMG does not contain any snapshots:

    ##### Option 1: via Finder.app

    Simply double-click the DMG file and it should mount. Then use `diskutil list` or `Disk Utility.app` to figure out which
    synthetised disk it was mounted as. It should have "APFS Container Scheme".

    ```
    /dev/disk9 (disk image):
    #: TYPE NAME SIZE IDENTIFIER
    0: +499.2 GB disk9
    /dev/disk10 (synthesized):
    #: TYPE NAME SIZE IDENTIFIER
    0: APFS Container Scheme - +499.2 GB disk10
    Physical Store disk9
    1: APFS Volume MinimeOS - Data 311.1 GB disk10s1
    2: APFS Volume Preboot 82.2 MB disk10s2
    3: APFS Volume Recovery 528.5 MB disk10s3
    4: APFS Volume VM 4.3 GB disk10s4
    5: APFS Volume MinimeOS 10.9 GB disk10s5
    ```

    ##### Option 2: via command-line

    ```
    > hdiutil attach /Volumes/W1/backups/mos.dmg
    Checksumming whole disk (Apple_APFS : 0)…
    ....................................................................................................................
    whole disk (Apple_APFS : 0): verified CRC32 $09A8F547
    verified CRC32 $0A545D85
    /dev/disk10 EF57347C-0000-11AA-AA11-0030654
    /dev/disk10s1 41504653-0000-11AA-AA11-0030654 /Volumes/MinimeOS - Data
    /dev/disk10s2 41504653-0000-11AA-AA11-0030654
    /dev/disk10s3 41504653-0000-11AA-AA11-0030654
    /dev/disk10s4 41504653-0000-11AA-AA11-0030654
    /dev/disk10s5 41504653-0000-11AA-AA11-0030654 /Volumes/MinimeOS
    /dev/disk9
    ```

    #### Case 2: if your DMG has some snapshots, you must mount it as read-write with a shadow file

    ```
    > hdiutil attach -owners on /Volumes/W1/backups/mos.dmg -shadow
    expected CRC32 $0A545D85
    /dev/disk11
    /dev/disk12 EF57347C-0000-11AA-AA11-0030654
    /dev/disk12s1 41504653-0000-11AA-AA11-0030654 /Volumes/MinimeOS - Data
    /dev/disk12s2 41504653-0000-11AA-AA11-0030654 /Volumes/Preboot
    /dev/disk12s3 41504653-0000-11AA-AA11-0030654 /Volumes/Recovery
    /dev/disk12s4 41504653-0000-11AA-AA11-0030654 /Volumes/VM
    /dev/disk12s5 41504653-0000-11AA-AA11-0030654 /Volumes/MinimeOS
    ```

    > How to convert read-only DMG to writable DMG?
    This is what worked for me:
    ```
    > hdiutil convert /Volumes/W1/backups/mos.dmg -format UDRW -o /Volumes/W1/backups/mos-rw.dmg
    nx_kernel_mount:1387: : reloading after unclean unmount, checkpoint xid 2213158, superblock xid 2213151
    Reading whole disk (Apple_APFS : 0)…
    ....................................................................................................
    Elapsed Time: 3h 14m 56.431s
    Speed: 40.7Mbytes/sec
    Savings: 0.0%
    created: /Volumes/W1/backups/mos-rw.dmg
    ```

    You can then follow case 1 to mount `/Volumes/W1/backups/mos-rw.dmg` instead of `/Volumes/W1/backups/mos.dmg`.

    As you can see from the output this conversion can take quite some time if the file is on external/slower disk, so it is
    preferable to use `-shadow` mounting as described in the article.

    > I've launched hdiutil/asr and it does seem to be stuck, what now?
    With larger images/disks it might take up to several hours for operation to complete. The progress indicator in terminal
    output is not granular enough to display any action for several minutes. I would recommend to open `Activity Monito.App`
    switch to `Disk` view and observe most busy reader/writer processes. This should give you sense if your command is doing any
    work or is slacking. It could give you also rough lower bound estimate how far it is in progress because it usually needs to
    read or write its source or target data at least once. So if you have 300GB DMG and see reading speed 1GB/s you can get an
    idea if you can take a nap or not. Also better disable power-saving because your Mac could hibernate unexpectedly while
    working.