Crash during restore from backup

[patrickplaggenborg]

No matter what I do, Arc keeps crashing (or gets killed by iOS) during Restoring from a backup.

All my backup files have been successfully downloaded from iCloud. Arc goes straight to the progress.

The remedy a couple of times has been to touch the screen a few times to keep the app ‘active’. I’ve also set iOS to always keep the display on.

I started out around 1548 timeline weeks (which would be 30 years of data, which sounds strange to me). Some weeks have been restored. I did see some errors in the process but everything moved forward.

Now I’m at 1098 and stuck, no matter what I do. I do see the Error count go up after some time. Sometimes it reaches 4, or 13. But it never goes to the next week.

I guess there is little to do about this, since it’s too much for Arc or iOS to handle.

When looking at the error log I do notice that the same error gets processed every time I retry. So even if Arc sometimes gets past some of this data in this specific week, after it crashes it looks like it will reprocess that week from scratch.

Is there some way to make Arc process the restore and continue where it left off also for data that has already been processed previously within the current processed week?

[matt]

Hmm, well that’s not great. Though we can work around it.

The Restore view is really just a super simplified UI that does the exact same thing as the File Importer view, accessible from Arc’s Settings tab → Backup, Import & Export. Inside the File Importer view you can see the full list of files in the backup, and can tap on each individually to download and import. The Restore view is literally just doing that (as well as automatically retrying after download completions, etc).

In the File Importer view, don’t bother tapping on Place or Timeline Item files to import. When you tap on a Samples file to import that week of samples, it will automatically find the matching Timeline Item and Place files for each sample in the file, and import them at the same time. So importing a Samples file also automatically imports all the dependent other files.

Oh, the Note and Day Summary files can be imported in any order though - before, after, during. Those ones stand independent of the others.

I’d avoid trying the “Import All” button on the Sample Files list though. That’s literally what the Restore view is doing, so if that’s stalling / failing on the Restore, you’ll run into the same stall on the File Importer view too. It’ll be best to try each file one by one, then there’ll be either a tick mark for full success, or an error button that will take you to a list of errors that occurred when trying to import that specific file (and/or its dependent Timeline Item and Place files).

You can import the Sample files in any order. I think the Restore view tries to do them from most recent first, so that the user can start to see their most recent data first, while the restore is in progress. But it makes no difference to the process which order you try in yourself.

[matt]

Oh I just tried the Import All buttons for the Notes files and Day Summary files on my test device, and that went smoothly and completed almost immediately (aside from a bunch which still haven’t finished downloading, after days of Arc asking iOS to download them. Grr). Those files are trivially easy to import because they have no dependent files.

And I just remembered the one significant difference between the File Importer view and the Restore view. The Restore view will delete the files after import, as long as the file has been fully, successfully imported. The File Importer instead doesn’t delete any files, and just leaves them in the list but with a tick mark on success.

The reason the Restore view deletes each file after success is that this “Import/Restore” folder is a special case folder for the managed restore process, which is meant to be a copy of the backup files. So once each file has been successfully imported it has no reason to exist anymore. The File Importer view is instead designed to pick up any importable files in the “Import” folder (including the “Restore” subfolder), and import them on demand, with no knowledge of whether they’re your only copy of the file or not, so deleting the file after success doesn’t make sense for the File Importer view.

[matt]

Ah, I just remembered another useful detail of the File Importer:

If a Samples file fails to import some of its samples due to missing dependent files, there will be a new button, “Retry import ignoring missing dependents”, available on the error/info view (tap the exclamation mark icon at the end of the row to open this view).

That button will attempt the import again, but this time importing each sample regardless of whether their dependent Timeline Item or Place file exists or not. This allows you to get around problems with incomplete backups that are missing some files.

Arc’s timeline processing is designed to self heal from all sorts of situations, and ultimately only the Samples themselves are necessary - all the rest will be rebuilt automatically when the timeline self heals. It might mean you’ll need to reconfirm a Place assignment for a Visit, or potentially repeat some other manual cleanup / reshaping you’d made to that timeline data in the past, but as long as the Samples are there, everything can be put back to how you had it before.

[patrickplaggenborg]

That’s awesome Matt! Very happy to hear that. Thanks for all the details on how this works. I was scared to lose all of my data and I’m happy to see there will be a way to fix this.

I’m looking at the File Importer right now and notice that it’s not seeing any of the files in the backup folder.

Arc itself does see the backup and allows me to restore.

Do I need to copy anything from the backup folder in iCloud to the Import folder for it to work?

[matt]

Ah, give it a minute. It’s loading the files list, which can take a while depending on how many files. The File Import view needs a loading spinner

[patrickplaggenborg]

Ahh yeah! So there was indeed something happening.

I just came back to that screen on a later moment and saw this. Thanks!

Loading spinner would be super helpful indeed!

[patrickplaggenborg]

Clicking on items separately works indeed.

(It wasn’t immediately clear to me the separate list items were also clickable. They looked like a list only, and maybe could use a small light grey icon behind them)

It’s nice to be able to dive into that file after an error!

It seems like there are missing place files.

If I don’t want to lose those places, is there anything I can do to solve these errors?

[patrickplaggenborg]

Other files show a grey icon. What does the grey icon mean?

Tapping it again forces a retry so it looks like an error that made it not complete the import and the icon allows you to try again.

Same result after retry, so not sure what I can do differently to have it succeed.

[patrickplaggenborg]

I accidentally tapped the first row instead of the error. That triggered a re-import.

Now the first file also shows the grey icon:

[matt]

Just checking the code, the exclamationmark.arrow.circlepath icon means the task is in “waiting” state, which means it’s waiting for a dependent file to download, ie a TimelineItem or Place file. Hmm.

That shouldn’t be possible if the Files app has already downloaded everything to local. It might be worth swiping the app closed and relaunching it, to reset any file system state awareness.

@patrickplaggenborg

If I don’t want to lose those places, is there anything I can do to solve these errors?

For missing Place files… you could theoretically mock up a Place json file, to match the UUID. But it’d almost certainly create a lot more trouble than it’s worth. Better to just let the processing engine self heal around it.

For data structure context:

• A Place is something like “Starbucks” or “Central Station”
• A TimelineItem is either a Visit or a Path, so these are the objects you see on the daily timeline view - either a Visit to a Place, or a Path between Visits.
• A Visit item usually has a Place assignment (Visit → Starbucks)

If a Visit doesn’t have an assigned place, Arc will either look for a suitable match and auto assign it, or mark the Visit as unconfirmed and ask you to assign a Place manually (eg the “1 unconfirmed item” button).

So losing the Place file for a minor place that you’ve only visited once is no big deal. There’ll just be a Visit somewhere in the timeline that will either try to pick up a Place automatically, or ask you to assign one.

Although losing the Place file for somewhere that you’ve visited hundreds of times could be a bit of a pain in the arse, because then there’ll be hundreds of Visits wanting a Place assignment. If it’s a nice n’ easy, clear cut auto assignment situation, then auto assignment will be able to clean that up on its own. But if it’s a fiddly situation like a Place that overlaps lots of other nearby Places that you also regularly visit, then it could cause a minor cleanup headache.

Though if a Place file is missing from the backups, I can’t imagine it being an important Place. Given backups run every day, it’d be very weird for a Place with many visits to somehow slip through the cracks. It’s more likely to be some Place that you only visited once… hopefully.

[patrickplaggenborg]

Thanks that explains. Happy to know the actual sample data is most important and will be saved.

Just tried force quitting Arc and retrying. The same thing happened.

I got an error on the first sample. Then retried while ignoring missing place data. This resulted in the grey icon with exclamation mark.

Now trying to redownload in the Files app. Maybe something was purged or wasn’t downloaded properly.

Noticing this difference that might be helpful to determine whether an iCloud folder or file has actually been synced with your phone.

Assuming that an actual calculated file size is a successfully synced folder. Also, the ‘Download Now’ button is not displayed for a synced folder.

I’ll keep you posted.

[matt]

@patrickplaggenborg

Now trying to redownload in the Files app. Maybe something was purged or wasn’t downloaded properly.

There is also a rare case where iCloud Drive gets into some sort of limbo with a file download, where the file isn’t downloaded but also can’t be requested to be downloaded. Presumably some sort of internal error state that we don’t get to see. I think from memory the workaround of restarting the phone gets around that problem, but it’s a rare enough occurrence that I haven’t seen it / heard of it in quite a while, so can’t remember the details.

[patrickplaggenborg]

Hmmmmmm. I’ve tried reprocessing the individual sample files.

Looks like the pattern is:

• Process
• Red error (missing Place files)
• Reprocess by ignoring errors
• Error with grey exclamation mark
• Tap again to reprocess
• Error with grey exclamation mark
• Stuck…

I’ve tried after force quiting the Files app.
I’ve tried after redownloading/syncing the files in the Arc iCloud folder.
I’ve tried after force quiting Arc.
I’ve tried after turning my iPhone off and on.

It looks like the import process crashes on something.

Is there a way we can debug or log why it’s failing?

See full video of what’s happening:

[matt]

Ugh, yeah seems like it’s falling into some gap in the logic there. I’ll have a poke around the code…

Ok so the red icon definitely means error, and that’ll be the missing Place files. That part makes sense. Then once you open the details view and tap to try again, ignoring dependents (aside: is it really opening up that details view as just an empty modal at first??), it’s going to the grey “waiting for dependents” icon. Which… that seems like the bug.

If you tell it to ignore dependents, then there should be nothing stopping the import, and it should just go all the way to the end and import every sample, regardless of whether there’s missing dependents or not.

Unless… it has done that, and the grey icon is just there as extra info, to say “okay I imported every sample this time, but some of them were missing dependents, just a heads up”. Lemme check that theory in the code…

Ugh, nope. It really is a bug. Found it. The “ignore missing dependents” means it won’t care if the TimelineItem file is missing, but if the TimelineItem file exists, it tries to import it, but doesn’t tell that import task to ignore the fact that the Place file is missing. So it gets stuck in a limbo of “have TimelineItem file, but importing the TimelineItem fails”.

K, I’ll get onto fixing that now. Uh, and will reply in a sec with a temporary workaround for you, so you don’t have to wait for the next release to get that samples file imported. Back in a min!

[matt]

Ok so the temporary workaround is this:

You need to find the TimelineItem file (or files) that are referencing the missing Place file, and remove these two lines:

Then the TimelineItem file won’t be dependent on the missing Place file anymore, and can import without problem.

The trick will be finding the TimelineItem file(s) referencing the Places. The import task details view isn’t telling you their UUIDs, which is … yeah, damn, wish I’d added that

Could try Spotlight file search, by just pasting the Place file UUIDs into Spotlight and seeing what files turn up. Assuming Spotlight is indexing iCloud Drive. Uh, assuming you’re on a Mac. But if on Windows, presumably the equivalent file system search should be possible…

[patrickplaggenborg]

Thanks Matt for the full explanation of what’s going on, and getting to the bottom of this.

And yes, the modal did load empty at first. Then I swiped it down and retried and that displayed the content correctly.

I haven’t been able to reproduce that, but it did catch my attention as well indeed.

The workaround you mentioned is helpful, but will be extremely time consuming as many of these errors are happening. It might be better for me to wait for your fix to be released.

See screenshot from the Backup Restore error log.

(Another small issue: the Error Log window title is black on a black background.)

I’m still wondering what caused all the place files to be missing by the way, because they are so many errors on those.

[matt]

@patrickplaggenborg

The workaround you mentioned is helpful, but will be extremely time consuming as many of these errors are happening. It might be better for me to wait for your fix to be released.

Yeah when I was describing it I was thinking “this will be a massive pain in the arse to do”. I’m prepping a new build to go live as soon as the current build’s 7 day staged rollout has finished (it’s on day 4 now). So it’ll definitely be less fuss to just wait until that update arrives next week.

@patrickplaggenborg

And yes, the modal did load empty at first. Then I swiped it down and retried and that displayed the content correctly.

Is the entire File Importer really sluggish too? I suspect what’s happening is the “file system change observing” that turns on when the importer view is open is clogging up the whole system. It was definitely a hindrance when I first built the File Importer, but on iOS 16 I’m finding it almost intolerably slow. So I’m working on ditching the use of iOS’s built in file system change observing, and just doing periodic checks every minute or so instead.

@patrickplaggenborg

(Another small issue: the Error Log window title is black on a black background.)

Ugh. That’ll be a bug due to Dark Mode sneaking through. Arc is supposed to ignore Dark Mode completely (at least in current builds - I’m working on an update that supports Dark Mode right now). But some views in the app somehow ignore the app-wide setting and try to use Dark Mode, fail miserably in the attempt.

@patrickplaggenborg

I’m still wondering what caused all the place files to be missing by the way, because they are so many errors on those.

I think I know the answer to this, and it’s a convoluted one. The short version is: I suspect the missing Place files are totally irrelevant Places and aren’t even needed.

But then how is there backup data referencing irrelevant/unnecessary Place files? The failure there is in me not noticing that iOS at some point changed how it names files that aren’t synced locally to the phone.

At some stage earlier on they were named “normal_filename.icloud”. So I could detect whether I file needed downloading by looking for that “.icloud” on the end, and distinguish between entirely missing files and files that just need downloading in the same way. But at some point iOS started naming them “.normal_filename.icloud”. Notice the dot at the start - I didn’t!

So at some time in the past, the backup system went from dutifully downloading existing backup files before updating them, to thinking the file was missing completely, and recreating it as new. That has resulted in masses of duplicate backup files, eg:

1. 2016-W17.json.gz
2. 2016-W17 2.json.gz
3. 2016-W17 3.json.gz
4. etc, etc

Now, the restore system doesn’t care about the duplicates. Whenever it imports one of those files it checks each object’s lastSaved and skips over the import data if it’s older than the data in the database. No harm done.

But, annoyingly, the checks for dependent files happen before the check for newest lastSaved. So the importer will error out, saying “ugh, missing dependent” before it realises it didn’t even need that out of date import data anyway.

So yeah, what I think those missing Place files are are Place files that are referenced by out of date data, but not referenced by up to date data. So the backup system at some stage probably deleted the Place file, thinking “don’t need that anymore - no one uses it”.

Aside: All these duplicate files are also slowing down the restore/import process massively. On my test device (that I’m still trying to get the restore finished on!) I’m seeing sometimes up to 30 or 40 duplicate files. Each of which the managed restore has to try to import (even if it ends up skipping every sample inside the file, due to being out of date).

Oh, don’t go deleting those duplicates by the way! There’s no way of knowing which will be the newest. Well… I assume the newest is the one with the highest number. But I wouldn’t want to gamble on that.

Heh. Yeah, I am really bad at waffling on endlessly Though these longwinded explanations do kind of act as rubber duck debugging for me - I’m selfishly using them as way to think through the problems and make sure I’ve definitely understood them correctly myself.

[patrickplaggenborg]

@matt

Is the entire File Importer really sluggish too?

The File Importer itself is not too slow for me. It takes quite some time to load all files in the beginning. But it stays quite responsive.

I would say the Restore Backup function is definitely sluggish and becomes completely unresponsive for minutes on end very often.

@matt

Ugh. That’ll be a bug due to Dark Mode sneaking through.

Ahhh! I saw that in a few other places in the timeline as well. I’ll send more feedback on that once you released your Dark Mode update.

@matt

So yeah, what I think those missing Place files are are Place files that are referenced by out of date data, but not referenced by up to date data. So the backup system at some stage probably deleted the Place file, thinking “don’t need that anymore - no one uses it”.

Sounds like a sound hypothesis. I noticed my iCloud folder contains a lot of these duplicates indeed.

@matt

Heh. Yeah, I am really bad at waffling on endlessly Though these longwinded explanations do kind of act as rubber duck debugging for me - I’m selfishly using them as way to think through the problems and make sure I’ve definitely understood them correctly myself.

I actually enjoy reading that a lot. It makes me feel like you put a lot of effort getting to the bottom of the problem. It allows me to follow your train of thought and often times I see myself thinking very similarly. It also allows me to actually understand better (to a certain extent) what’s going on so I don’t have questions about stuff you already explained. Any next questions or suggestions from me can then be more relevant.

[patrickplaggenborg]

Tried the new restore functionality. Nice to see the feedback of what’s happening in the UI!

https://streamable.com/2xsfu2