Yeah, much better for showing that progress is still being made.
Still struggling to restore this backup. From 1098 down but still 1028 weeks to process. Each week takes about an hour.
It takes long but thatâs fine.
The problem is that after a few minutes the app crashes. So it needs my intervention every time to restart the restore.
This is almost impossible during the day because it renders my phone useless in that meantime.
Having the backup run throughout the night would be perfect, but then I canât be there to restart the restore.
Is there any way I can help to debug the crashes during restore so we can just leave it running?
o
Unfortunately it wonât be actual crashes (which is something I can fix), itâll be iOS terminating the app due to high energy use in the background.
Thatâs something that we really canât work around, because itâs just how iOS is designed. iOS takes an extremely strict position on energy use in the background, and will aggressively terminate apps that use sustained energy/CPU over a short period of time (for example a threshold of sustained 80% CPU over 60 seconds).
The method that iOS offers to apps to get around this is scheduled background tasks, which can be flagged as âneeds powerâ, meaning that it should only be run when the phone is plugged in to power. That flag gives that background task permission to use high CPU. But the catch is iOS wonât start that task when you actually ask it to be started, itâll start it at any time after that, when it feels in the mood. And it will then stop that task as soon as the mood takes it. So itâs unworkable for the restores, because it might not even start until hours later, then iOS might only run the restore for a few minutes before stopping it again.
I think the best workaround for now is to use the manual importer, from Arcâs Settings tab â Backup, Import & Export, File Importer. Then in there tap on the âSample Filesâ a few at a time, and keep doing that until theyâre all done. Thatâs basically exactly what the managed restore system is doing - itâs just doing the equivalent of tapping âImport Allâ, which queues them all up at once. But if you do it manually yourself, one or two at a time (or even âImport allâ, just to see what happens), you can keep an eye on the progress much more closely.
Hmmm with 1028 weeks to go for the restore, and each week taking 1 hour to complete, thatâs already 42 full days of 24 hours to restore it.
Then option 1 is the app being crashed by iOS, and me having to restart it every 5 minutes.
Option 2 is me manually processing all of it.
In both cases I would need my full attention on my phone.
Obviously I canât be doing this the full 24 hours. So I could set aside a full hour each day to do this.
That would take me almost 3 full years of dedicating one full hour a day of my life to restore this backup.
Iâm sure youâll agree with me that this isnât the best solution.
Isnât there another smart solution to create and process these backups? I thought you mentioned that you were going to overhaul the entire backup system.
Some ideas:
- process the backup/restore server side
- process the backup on my MacBook Pro saving it in iCloud
Maybe a simple macOS companion app that would help solve this?
Any other ideas?
One week should take a few seconds to complete importing. Try in the File Importer view, tapping on one Samples week file, and see how long it takes.
A full restore of say 6 years of data should take less than an hour to complete.
It might be the case that thereâs a lot of duplicate week files in your backup. Only one of those files needs to be imported, but the managed restore process will queue them all up because it doesnât know which one to trust. You could speed the process up by going through the samples folder and finding the newest of each week file, then deleting the rest. (Make sure to do this on a copy of your backup, not the original - you donât want to accidentally delete the wrong file then have it lost forever).
Iâll give it a try to check out the samples folder.
Is it possible to let the Arc app remove the duplicates instead of me having to do it manually?
Arc was built in a way that created these duplicate files in the first place. Feels a bit bad having to spend so many hours and hours already trying to fix that.
I see Arc has created a lot of duplicate files for a lot of weeks in the LocoMotionSample folder.
Some are small and have 2-3 duplicates. A lot of them are 2-3MB and have up to 14 duplicates.
I manually deleted the duplicates where the filesize looked similar.
When file sizes dit not look similar I kept the one with the highest duplication number.
This cleaned up the folder from 1GB to about 450MB. And 449 files instead of about 1040.
I tried a couple of manual imports and this now went pretty fast for the weeks I tried. I now started the full restore and this also looks like it works smoother now. Still in the middle of the process, weâll see.
Letâs hope this whole process wonât create new duplicates anymore.
I would suggest that Arc would do a clean up of the folder before starting the full restore by removing any duplicates and keep the latest one for each week.
I considered adding this in the update where I found and fixed the duplication bug. But in practice the duplicates at most doubled or tripled the backup restore duration in my testing, even when there were sometimes 5+ duplicates of some weeks. So I weighed the risk of accidentally deleting the wrong duplicate, thus risking data loss, and decided it was better to just leave it.
If I do end up doing something about the duplicates, what Iâll most likely do is just automatically trigger the start of a fresh backup. That way the previous backup with duplicates can be preserved unchanged, in a âPrevious Backupsâ folder, thus not risking accidentally deleting the wrong duplicates. And the fresh backup will be clean without duplicates.
Aside: The extremely long restore times are largely due to the app being terminated by iOS, with the duplicate files being a secondary cause. If the app is terminated during restore and youâre not there to notice, it can be hours until you resume the restore. That adds a lot more time to the restore than the duplicates.
Of course because the duplicates slow down the process, thereâs more risk of the restore being left unattended (eg in the background) and iOS then terminating the app. So one feeds into the other, but ultimately itâs the terminations that really cost big in time.
Ah, hereâs the trap! You have to keep the newest duplicate, which might be the one with the highest number at the end, or the one with no number at all. Itâs the âlast modifiedâ date that you need to check, and keep the file that was most recently modified.
Definitely new duplicates wonât be created. That bug has been fixed.
Happy to hear the cause of the duplications is solved. So it will not return.
Think removing the duplicates solved the issue for a large part! It brought down the amount of weeks to restore from 1000+ to 400+ now.
Also, Iâve noticed many of the weeks indeed restore in less than 10 seconds. Feels great and smooth.
Looks like there is a second reason for slowing down:
Some of the weeks being restored are taking a very very long time. See video below.
As you can see in this video the progress bar is taking a very very long time to move forward. At some point, iOS crashes it.
What could be the cause of this, and how to solve it?
Very hard to know. The restore view doesnât say any files are downloading, which would be the usual reason for a week to restore slowly. Itâs possible it had already restored part of that week before, so this time through itâs skipping most of the samples after seeing they already exist, which might be less efficient than just restoring them. But thatâs not a particularly convincing explanation to me. I donât have any good guesses, Iâm afraid.
I wouldnât bother thinking too hard on it. Itâs the kind of thing where Iâd go digging with the debugger, then find something that both makes sense and is also unavoidable. All the easy optimisations have already been done.
Iâve hammered the restore process with literally months of testing, doing tens or even hundreds of restores on multiple test devices with various different backup sets (the most boring, tedious days of my life. heh). So thereâs very unlikely to be any major bugs or obvious inefficiencies left in it.
I could possibly tweak it further to get another 10-20% more efficiency out of it, but really the bottlenecks now are unavoidable IO - itâs reading and writing gigabytes of data, and giving the CPU a sustained hammering, which are both things that iPhones really hate you doing for any extended period (eg more than 30-60 seconds at a time).
If I were to do anything more with the restore process other than just find new ways to squeeze a bit more of efficiency out of it, Iâd probably take a completely different approach - making it slower instead of faster, and making it a hidden background process that trickles away over weeks, gradually restoring the data while youâre not paying attention, at a low enough IO and CPU level such that iOS doesnât get angry.
That way of doing it would make the restore take far longer, but also likely much more reliable (or at least feel more reliable). When you have a UI thatâs showing you directly whatâs happening, taking up the whole screen, itâs unavoidably obvious when itâs failing, needing to stop and resume, getting stuck on some slow part, etc, etc. You literally canât avoid seeing the pain points. But when itâs hidden in the background it can chip away at the job, getting to the finish line slower but in a much less visibly frustrating way.
So yeah, that would probably be where Iâd go next with it. That would appease iOS, getting rid of the frustrating cycle of app terminations and manual resumes of the restore. Instead it would quietly and slowly work away at it in the background (or more likely in the foreground, but without showing you any obvious UI, so that you donât have to waste any thought on it).
A lot of this stuff really does come down to how you communicate whatâs happening, and sometimes itâs better to not communicate the gritty details. An example I learnt years back was CPU meter widgets - if you have a CPU meter visible at all times on your computer, youâre likely to feel like things are going slower, because you can visibly see when the computer is straining through a heavy workload, even if that workload isnât causing any lag in UI interaction or tasks youâre performing. If instead thereâs no CPU meter, youâll probably be oblivious to those heavy workloads, and if they werenât measurably impacting UI responsiveness anyway your experience of the overall performance will be better 
Hmmm. That basically means Iâm stuck. Because as I calculated before, the current setup takes an amount of time one canât reasonably ask anyone to spend manually. And I think I am already going way beyond reasonable to get this to work.
Is there a way to move this process to macOS so I can just let my laptop crunch it without being closed by iOS?
Arc Appâs strict approach to privacy means that your database is only stored on your phone. The restore has to happen on your phone.
I canât speak to your calculations, but I can assure you that even a restore with many duplicated files, with six years of data, will take at most a few hours to complete. The delays are due to the app being terminated in the background, during which time the restore makes no progress. If the restore is able to continue uninterrupted in the foreground, it will complete in a matter of hours.
I do recommend using the manual restore process in the File Importer view (Settings tab â Backup, Import & Export â Open File Importer). That will give you better insight into whatâs going on, and granular control over which week files are being imported. You can tap âImport Allâ on the sample week files section, but I recommend tapping a few rows at a time, to get a feel for how fast your phone can import each week. Some weeks will go very fast, and some quite slow (which can make calculating an overall duration tricky).