Sparse bundle thinks it's out of space.

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Sparse bundle thinks it's out of space.

Michael_google gmail_Gersten
I'm running into an issue with sparse bundles. I am using 10.9.5.

The issue is this: deleting files off of a sparse bundle does not actually free up space, or at least does not appear to.

Specifically: the partition that I'm using for Time Machine had a sparse bundle created in it. The sparse bundle was the size of the partition minus a small amount (default sizing). Due to the poor speed of Time Machine with sparse bundles and the default band size, I used the technique seen in various places to make the band size bigger (copying it with hdiutil and specifying a new band size).

"hdiutil compact" will not free up unused bands when they are bigger than the default (128 mb in use). I did not think this would be a concern, because even if all of the drive image space is allocated, there will still be space on the virtual disk that is unused, freed up, and will be reused. The worst that would happen, I figured, would be that the band files would be rewritten repeatedly with new backup data.

This is not what happened. Issue number one, is that the free disk space that the operating system reports is based on the free disk space of the underlying physical disk, regardless of how many gigabytes of data have been deleted off of the this image. (As you can guess, this means that my entire Time Machine history is now gone.).

Issue number two? For some reason, just ignoring the disk free result, attempting to write to the image drive anyways fails. Nevermind that the HFS filesystem contained on the drive image should have terabytes of unused space, writes fail. It started with backupd complaining that it could not write its log file, and now I cannot even use a simple "echo test > out" without a complaint.

First question: is there any way to make a sparse image drive reuse space in the bands when they are freed up?

Second question: is there any way to compact a sparse image when the band size used is bigger than normal?

(A monthly maintenance reboot into 10.12, or something similar, is not out of the question. For normal operation, I need features of 10.9.5 that go away in 10.10.)
---
Entertaining minecraft videos
http://YouTube.com/keybounce

_______________________________________________
MacOSX-talk mailing list
[hidden email]
http://www.omnigroup.com/mailman/listinfo/macosx-talk
Reply | Threaded
Open this post in threaded view
|

Re: Sparse bundle thinks it's out of space.

@lbutlr
On 2018-02-24 (09:55 MST), Michael <[hidden email]> wrote:
>
> First question: is there any way to make a sparse image drive reuse space in the bands when they are freed up?
>
> Second question: is there any way to compact a sparse image when the band size used is bigger than normal?

Have your tried with a non-hax0red sparse bundle using the default 8MB band size?

--
A TRAINED APE COULD NOT TEACH GYM Bart chalkboard Ep. AABF15

_______________________________________________
MacOSX-talk mailing list
[hidden email]
http://www.omnigroup.com/mailman/listinfo/macosx-talk
Reply | Threaded
Open this post in threaded view
|

Re: Sparse bundle thinks it's out of space.

Michael_google gmail_Gersten
> On 2018-02-24 (09:55 MST), Michael <[hidden email]> wrote:
>>
>> First question: is there any way to make a sparse image drive reuse space in the bands when they are freed up?
>>
>> Second question: is there any way to compact a sparse image when the band size used is bigger than normal?
>
> Have your tried with a non-hax0red sparse bundle using the default 8MB band size?

Sure. Works. Will compact. But slow. Very slow.

And this is a normal, supported HDI image bundle, made by the normal Apple tools.

>
> --
> A TRAINED APE COULD NOT TEACH GYM Bart chalkboard Ep. AABF15
>
> _______________________________________________
> MacOSX-talk mailing list
> [hidden email]
> http://www.omnigroup.com/mailman/listinfo/macosx-talk

---
Entertaining minecraft videos
http://YouTube.com/keybounce

_______________________________________________
MacOSX-talk mailing list
[hidden email]
http://www.omnigroup.com/mailman/listinfo/macosx-talk
Reply | Threaded
Open this post in threaded view
|

Re: Sparse bundle thinks it's out of space.

Michael_google gmail_Gersten
>>> First question: is there any way to make a sparse image drive reuse space in the bands when they are freed up?
>>>
>>> Second question: is there any way to compact a sparse image when the band size used is bigger than normal?
>>
>> Have your tried with a non-hax0red sparse bundle using the default 8MB band size?
>
> Sure. Works. Will compact. But slow. Very slow.
>
> And this is a normal, supported HDI image bundle, made by the normal Apple tools.

To clarify: with the default 8 MB band size, the same behavior of assuming that the space on the image is the same as the space on the host drive, and as far as I can tell not reusing the band space when a file is deleted off the image drive.


_______________________________________________
MacOSX-talk mailing list
[hidden email]
http://www.omnigroup.com/mailman/listinfo/macosx-talk
Reply | Threaded
Open this post in threaded view
|

Re: Sparse bundle thinks it's out of space.

Macs R We
Of course, in a TM image, one has to be sure one is actually deleting the large file, and not just one of the possibly many hardlinks to the large file residing in another backup layer.

> On Feb 24, 2018, at 5:03 PM, Michael <[hidden email]> wrote:
>
>>>> First question: is there any way to make a sparse image drive reuse space in the bands when they are freed up?
>>>>
>>>> Second question: is there any way to compact a sparse image when the band size used is bigger than normal?
>>>
>>> Have your tried with a non-hax0red sparse bundle using the default 8MB band size?
>>
>> Sure. Works. Will compact. But slow. Very slow.
>>
>> And this is a normal, supported HDI image bundle, made by the normal Apple tools.
>
> To clarify: with the default 8 MB band size, the same behavior of assuming that the space on the image is the same as the space on the host drive, and as far as I can tell not reusing the band space when a file is deleted off the image drive.
>
>
> _______________________________________________
> MacOSX-talk mailing list
> [hidden email]
> http://www.omnigroup.com/mailman/listinfo/macosx-talk

_______________________________________________
MacOSX-talk mailing list
[hidden email]
http://www.omnigroup.com/mailman/listinfo/macosx-talk
Reply | Threaded
Open this post in threaded view
|

Re: Sparse bundle thinks it's out of space.

@lbutlr
In reply to this post by Michael_google gmail_Gersten
On 2018-02-24 (17:03 MST), Michael <[hidden email]> wrote:
>
> To clarify: with the default 8 MB band size, the same behavior of assuming that the space on the image is the same as the space on the host drive,

You said "The sparse bundle was the size of the partition minus a small amount ", so what are you expecting the free space to report?

I only  use a few sparse bundles, they are set to be maximum of 4.7GB and they report the proper amount of free space as maximum Size of bundle - space used on bundle, as I would expect.

If you are making a custom sparse bundle with a custom band size I would not be at all surprised if something like Time Machine got very confused by that. Time machine is not just any app, and it has deep hooks into the system.

As for deleting data from Time Machine, this is often tricky and just because you think you deleted something doesn't mean you did. What tool are you using to delete files to free up space?

AFAIK, Time Machine only supports using sparse bundles that 1) it creates and 2) are on network shares. Could be wrong, it's not something I've spent a lot of time on. Time Machine is my primary backup, but that is only because it is convenient. The backup I rely on is Backblaze.

--
"...Life is not a journey to the grave with the intention of arriving
safely in one pretty and well-preserved piece, but to slide across the
finish line broadside, thoroughly used up, worn out, leaking oil, and
shouting GERONIMO!!!" -- Bill McKenna

_______________________________________________
MacOSX-talk mailing list
[hidden email]
http://www.omnigroup.com/mailman/listinfo/macosx-talk
Reply | Threaded
Open this post in threaded view
|

Re: Sparse bundle thinks it's out of space.

Michael_google gmail_Gersten

On 2018-02-24, at 6:04 PM, @lbutlr <[hidden email]> wrote:

> On 2018-02-24 (17:03 MST), Michael <[hidden email]> wrote:
>>
>> To clarify: with the default 8 MB band size, the same behavior of assuming that the space on the image is the same as the space on the host drive,
>
> You said "The sparse bundle was the size of the partition minus a small amount ", so what are you expecting the free space to report?
>
> I only  use a few sparse bundles, they are set to be maximum of 4.7GB and they report the proper amount of free space as maximum Size of bundle - space used on bundle, as I would expect.
>
> If you are making a custom sparse bundle with a custom band size I would not be at all surprised if something like Time Machine got very confused by that. Time machine is not just any app, and it has deep hooks into the system.
>
> As for deleting data from Time Machine, this is often tricky and just because you think you deleted something doesn't mean you did. What tool are you using to delete files to free up space?
>
> AFAIK, Time Machine only supports using sparse bundles that 1) it creates and 2) are on network shares. Could be wrong, it's not something I've spent a lot of time on. Time Machine is my primary backup, but that is only because it is convenient. The backup I rely on is Backblaze.

I am certain that I'm actually deleting the large file. The tool that I am using is Time Machine itself.

Console log shows:

Feb 23 21:23:44 keybounceMBP com.apple.backupd[26765]: Deleted backup /Volumes/TimeMachine/Backups.backupdb/Keybounce's MacBook Pro/2017-11-09-235918 containing 51.36 GB; 56.22 GB now available, 7.69 GB required
Feb 23 21:23:44 keybounceMBP com.apple.backupd[26765]: Deleted 3 backups containing 53.48 GB total; 56.22 GB now available, 7.69 GB required
Feb 23 21:23:44 keybounceMBP com.apple.backupd[26765]: Backup date range was shortened: oldest backup is now Nov 16, 2017
Feb 23 21:23:48 keybounceMBP com.apple.backupd[26765]: Copied 8 items (1 KB) from volume Macintosh HD. Linked 21.
Feb 23 21:25:01 --- last message repeated 4 times ---
Feb 23 21:27:47 keybounceMBP com.apple.backupd[26765]: Copied 405 items (1.54 GB) from volume New Main. Linked 6550.
Feb 23 21:28:08 keybounceMBP com.apple.backupd[26765]: Copied 2 items (Zero KB) from volume UserData. Linked 13.
Feb 23 21:28:11 keybounceMBP com.apple.backupd[26765]: Copied 2 items (Zero KB) from volume rawTimeMachine. Linked 5.
Feb 23 21:28:16 keybounceMBP com.apple.backupd[26765]: Copied 5 items (33 bytes) from volume Sierra10-12test. Linked 19.
Feb 23 21:29:32 keybounceMBP kernel[0]: hfs: set LowDisk: vol:TimeMachine, freeblks:17670, warninglimit:19200
Feb 23 21:29:32 keybounceMBP KernelEventAgent[136]: tid 54485244 received event(s) VQ_LOWDISK (4)
Feb 23 21:29:32 keybounceMBP KernelEventAgent[136]: tid 54485244 type 'hfs', mounted on '/Volumes/TimeMachine', from '/dev/disk4s2', low disk
Feb 23 21:29:33 keybounceMBP kernel[0]: hfs: set VeryLowDisk: vol:TimeMachine, freeblks:12710, dangerlimit:12800
Feb 23 21:29:33 keybounceMBP kernel[0]: hfs: set VeryLowDisk: vol:TimeMachine, backingstore b_avail:78246, tag:16
Feb 23 21:29:33 keybounceMBP KernelEventAgent[136]: tid 54485244 received event(s) VQ_LOWDISK, VQ_VERYLOWDISK (516)
Feb 23 21:29:33 keybounceMBP KernelEventAgent[136]: tid 54485244 type 'hfs', mounted on '/Volumes/TimeMachine', from '/dev/disk4s2', low disk, very low disk
Feb 23 21:29:34 keybounceMBP mds[131]: (Warning) Volume: Indexing reset and suspended on backup volume "/Volumes/TimeMachine" because it is low on disk space.
Feb 23 21:29:36 keybounceMBP com.apple.backupd[26765]: Stopping backup.
Feb 23 21:29:36 keybounceMBP com.apple.backupd[26765]: Error: (-34) SrcErr:NO Copying /Volumes/rawKleimanMovies/KleimanMovies.sparsebundle/bands/32c9 to /Volumes/TimeMachine/Backups.backupdb/Keybounce's MacBook Pro/2018-02-23-212008.inProgress/103200FE-3AA8-4766-918F-D60F6A88AAF0/rawKleimanMovies/KleimanMovies.sparsebundle/bands
Feb 23 21:32:10 keybounceMBP com.apple.backupd[26765]: Copied 14 items (1.48 GB) from volume rawKleimanMovies. Linked 9498.
Feb 23 21:32:10 keybounceMBP com.apple.backupd[26765]: Copy stage failed with error:28
Feb 23 21:32:12 keybounceMBP com.apple.backupd[26765]: Backup failed with error 28: 28


In fact, over the rest of the night and the next morning, the log shows complaints of no space for the indexer, deleting a backup to make room for the indexer, resuming the indexer, and the indexer failing for no disk space.

Right now, there is only one backup on the sparse bundle.

What do I expect to show in free space? The underlying partition is 3.7 TB. The drive image, if I remember correctly, was made at 3.5 TB. The free space should be 3.5 TB minus the used (one backup). In other words, the space that the disk image claims is available, with a maximum of the actual allocated and unused band space plus the available on the underlying partition.

But the other issue is that if someone chooses to not look at free space, and just write, the filesystem should happily reallocate locations in the already allocated band files. The write should succeed.

And yes, I also use Backblaze. But I discovered that Backblaze has some serious limitations. It cannot restore file permissions, in particular it does not restore the execute bit. This wrecked all of my shell scripts, for example. I rely on Time Machine for easy access to historical data, such as restoring Firefox browsing history when a Firefox crash results in the places.sqlite file being ruined.

Meanwhile: I just used disk utility and explicitly told it to shrink the partition contained inside the sparse bundle file, to a size that I know is small enough to fit on the drive. I'll report if this works. The goal here is to force the file system to rewrite the allocated space in the bands files, and not try to make new bands files.


_______________________________________________
MacOSX-talk mailing list
[hidden email]
http://www.omnigroup.com/mailman/listinfo/macosx-talk
Reply | Threaded
Open this post in threaded view
|

Re: Sparse bundle thinks it's out of space.

Michael_google gmail_Gersten
In reply to this post by @lbutlr
Oh, this is fun.

Feb 24 20:29:49 keybounceMBP kernel[0]: hfs: cat_resolvelink: can't find inode=dir_2 on vol=TimeMachine

...
The good news: using disk utility to explicitly shrink the partition results it working properly, showing plenty of free space as expected.
The bad news: that error message


_______________________________________________
MacOSX-talk mailing list
[hidden email]
http://www.omnigroup.com/mailman/listinfo/macosx-talk
Reply | Threaded
Open this post in threaded view
|

Re: Sparse bundle thinks it's out of space.

Michael_google gmail_Gersten
Oh crud.

/dev/disk3s3    3.4T  3.4T  492M 100% /Volumes/rawTimeMachine
/dev/disk4s2    3.3T  3.3T     0 100% /Volumes/TimeMachine

So this morning, I saw something in the console log that made me worried. Another one of those "cannot cat this inode" issues.

A verification of the Time Machine image showed that it was fine. So now let's verify the underlying physical drive.

Oh, it's live. Unmount the Time Machine volume.

Nope, still live.

A quick check showed that the diskimage-helper program was still running. Right, I have to actually eject the Time Machine volume.

The raw volume is fine. Remount Time Machine, and oops.

Remember, yesterday I shrank the diskimage with disk utility, and at that point the reported size was working correctly. It was ignoring the space on the underlying physical drive, and reporting that it had about 2 TB of unused space inside the diskimage. The only thing I did since then it was a complete flush of the cache.

The underlying physical drive still has space. The Time Machine volume is ignoring that, and only using its own space. This is good.

But in the process of completely cleaning out the cache, and reading the data off the drive, it went from seeing a large amount of space to seeing no space.

There seems to be a cache consistency problem in the diskimage system somewhere.

I'm probably going to reformat this thing, and start over. Having lost the two month history, there is no reason not to.

EDI: Or, you know, ...

keybounceMBP:TimeMachine michael$ du -s lost+found/
du: cannot read directory 'lost+found/dir_11663332': Permission denied
du: cannot read directory 'lost+found/dir_13655795': Permission denied
du: cannot read directory 'lost+found/dir_14096703/com.apple.documentVersions': Permission denied
... 456 total lines ...
du: cannot read directory 'lost+found/dir_4871298/Pictures': Permission denied
du: cannot read directory 'lost+found/dir_4871298/Public/Drop Box': Permission denied
du: cannot read directory 'lost+found/dir_8696987': Permission denied
915271136       lost+found/
keybounceMBP:TimeMachine michael$

Yea ...

---
Entertaining minecraft videos
http://YouTube.com/keybounce

_______________________________________________
MacOSX-talk mailing list
[hidden email]
http://www.omnigroup.com/mailman/listinfo/macosx-talk