I have been using Arc for years starting around the time when Moves went under. After many years, and new iPhones, at some point, I noticed a 51 month data gap and was too hesitant to actually to sit down and try to recorver the “missing data”
I recently had to have my iPhone replaced and decided to start from scratch by downloading my apps one by one and NOT restoring everyone from an iCloud backup.
This is when the data gap grew from ~51 months to ~66! I began to panic and came to terms with the fact that I need to solve this problem.
I briefly researched how to restore from a backup and moved the backups into the import folder. I then imported “manually” by cgoign into the File Importer and clicking “Import” next to each section. I did the smaller ones first, Notes etc. But one of them had so many items to import that it would crash.
This is when I did more research and learned to change the name of the backup in the Import folder.
There were two backups and both took about a week to auto restore.
Now everything seems to be restored.
BUT I now have other issues from possibly the way I restored.
The 66 month gap persists. There is a grey line with 66 months next to it, at he bottom of each day that has been restored from those 66 months. And the map looks like a child drew on it because I assume it is trying to load all of those 66 months visually.
Now I have ~3 duplicated locations on most days of the restored data gap. OF the three, two will have the same start time but one will begin an hour before the others.
My app has not backed up since this was all restored, about 2 weeks ago.
If I leave the app open on Timeline, “Updating Metadata” and “Processing Timeline” and the purple “X Unconfirmed Items” are constantly alternating, which I assume what is chewing through my battery life.
This is my favorite app. Period. The equivalent of writing I na daily journal. The only one I willingly, and proudly pay for. It has been on my home screen since I first downloaded it. Please Matt, or anyone, help me.
I’m guessing that 66 month gap is some sort of mega timeline item, that somehow the imported data has been merged into. So the data will be there, just all squished into a single item. That will definitely upset things!
It’s likely that you won’t be able to fix that without doing a fresh install and fresh restore from backups. You could try tapping into that mega item and then the Individual Segments view and trying to break it apart that way, but I’m guessing the amount of data loaded would cause the app to be terminated.
When doing a fresh restore, start by copying in the largest and most recent “Previous Backups” folder into the “Import” folder. Then in Arc open the File Importer view from Settings tab, and import the Sample Weeks files one by one, from newest to oldest. That way you can check in the timeline view if things are still looking relatively sane after each week or few has imported.
Though before you start, it’ll be worth making sure you at least have your most recent data backed up, so you can restore that too. You can check which data has backed up by looking in the Backups folder, then in LocomotionSample folder, and checking which year-week numbers are in there. Those sample files are the heart of your data, and the most important. So as long as the week files are there (and dated recently, if it’s the current week), you’re all good.
Basically the process you want to do is restore the Sample Files (the ones in LocomotionSample folder) one by one, from newest to oldest, from whichever backup folder has the most recent or most complete version of that week.
It’s ok to restore from multiple Previous Backups folders, if for example one appears to have older but more complete data, then another has never but incomplete data. The importer will check for existing data when importing, and will only overwrite the existing data if the data in the file is more up to date.
If you do import from multiple folders, do them one at a time. By that I mean copy one of them into the Import folder, then import those files in the File Importer view, then remove that folder and copy in the next one, so that there’s only one Previous Backups folder in the Import folder at a time.
Anyway, let me know any questions you have and how you get on!
Matt! Hi there. Great app. Absolutely my favorite, by far. Thank you for all the amazing work you and your team do. Your app (and Apple’s iOS) updates are the ONLY ones I truly get excited for anymore.
So, to business it appears to be two simultaneous reads at once. The mega 66 month one and the standard daily timeline. My phone will normally crash within moments when I am anywhere in the 66 month range.
I have not been able to backup for the past 15 days so I am missing “2024 Week 8 & 9” and I am afraid to delete the app then try restoring again out of fear of entirely losing the last 2 weeks.
Am I supposed to delete the app, and do a fresh reinstall? Will it being pulling fro the Locomotion samples automatically?
For the past two weeks, I have tried leaving the app open in the foreground on the charger and in the background, trying to back up but it usually crashes eventually. If there is another way to force a backup or see the progress, that would be great. As of right now Im at the mercy of your magic as I sit here with my fingers crossed.
As a precaution, I have backed up the Locomotion folder elsewhere just in case and turned the backup toggle off and on again as I type this.
I appreciate any and all of your help. Thank you for your time.
P.S. Before I forget, I wanted to suggest a desktop companion app for viewing / managing the data on a larger screen with more functionality / customization / search-ability. I remember years ago with Move there was a website out there that could access the Moves data, although I don’t recall what it was, and it allowed a large Google Map type view with hot spots on it of frequently visited places etc. It could be something else to sell, or upgrade to in addition to the app!
I just remembered there are two options below “iCloud Drive Backups” so I enable the Daily and Monthly JSON exports. Assuming your answers is yes, to reinstall the app and restore from the Locomotion iCloud backup folder, I will use the recent JSON exports to restore If the iCloud backup does not work.
Ok that does sound like the processing has messed up and merged a large amount of data (well, 66 months of data!) into a single item. That mystery occurrence can be fixed with editing when it’s small, but when it’s that big, yeah, there’s no hope, as you’ve seen. It tries to load all that data into memory then gets killed by iOS due to going over memory use limits.
Eek. Yeah, if you haven’t been able to get it to back up the most recent data, then you definitely don’t want to be deleting and reinstalling yet. We’d better make sure that data is safe first.
When we do do the delete and reinstall, I’ll get you to import the backup data manually, rather than with the “managed restore” process that it offers when it sees the previous backup. The manual import will allow us to step through week by week around the danger point, where things went wrong last time.
That manual restore will be from the “File Importer” on the Settings tab → Backup, Import & Export. But we won’t be getting to that yet. First we need to make sure all the data is going to be there ready for restore - we need the most recent data backed up.
You’ve been doing the right thing - leaving it plugged in and Arc in the foreground forces a backup to start, and continue until complete.
As long as it hasn’t crashed / been terminated yet, it will be still working away at that backup. So what will be super useful at this stage is being able to see just how much work it has left to do. Arc has debug views for that info, but they’re not enabled in normal builds. They are however visible in Arc Mini. If you haven’t already got it installed, that’ll be the next step.
In Arc Mini, tap the ellipsis button and open “System Debug Info”. In there you’ll see a Backups section, which lists out how many pending items there are of each type. The “Samples pending backup” is the one we want to focus on.
As a rough rule of thumb we could maybe say one day of data is around 1000 to 5000 samples, depending on the amount of movement in the day. So if that number of pending samples is lower than maybe 10k we’re looking pretty good.
The pending Items should also be less than 100, and on a normal day would typically be around 10-50, until the next backup completes. But those item objects are the easiest to back up, so I’d expect that number to be low already. Really it’s the samples that are the heart of the data, and the one of most concern.
I just double checked the code, and it doesn’t care about date order when backing up the weeks. I was concerned that it might be trying to back up from newest to oldest, which would mean the two most recent weeks being missing would be a worry. But it’s not caring about date order, so it might still be working successfully through doing a full fresh backup and just hasn’t got to those weeks yet.
Heh. You and me both! Unfortunately because of Arc’s strict approach to privacy, the data is only stored on the phone and in the iCloud Drive backup. Which makes it hard for any app on another device to get at it. Technically another app on another device could read the iCloud Drive backups, but given that they’re hundreds of megabytes, it’d be slow going. And editing the data on another device wouldn’t be possible to sync back to the phone. So at this stage it’s a feature out of reach. It’s just too much data to sync between devices. But hopefully someday!
That’ll probably be Move-o-Scope, which I loved! It was actually the reason why I started building Arc in the first place! I could see the potential for how much more Moves app could be doing, and wanted to build something to make that possible.
A purely viewer / visualiser focused app like that could actually be possible, by reading the iCloud Drive backups. I believe one or two have actually been made for Arc’s data formats. Although I can’t remember their names off the top of my head - I haven’t seen them in quite a while. It’s certainly something I’d love to do myself someday, if I ever get the time.
I downloaded Arc Mini. I had hear about it over the years in the forums but never knew what it was. This is almost exactly what I was hoping before because it allowed me to see the progress. Over the past two days I have left Arc running in the foreground as much as I could and if you can see I nthe attached screenshots, its went through a lot!
I have looked into the iCloud files and W08 and W09 now exist, which is great!
The 66 month thing still exists and I’m now up to 17 days not backed up in the Arc > Settings > Backup screen. “Backup in progress” is displayed so Im just letting it run.
Should l continue to wait for Arc to say “Backup is up to date” in the settings screen or is it now safe to proceed with a delete and reinstall?
Phew! Yeah, the same files are always the last to catch up. Though once I see “Items pending backup” down to under 10-20 I know it’s on the home stretch.
Hmm. It looks to me though the backup should be almost finished by now. What’s the “Samples pending backup” down to now?
Aside: One thing that can happen with the scheduled background tasks is that iOS will start the task, then “expire” only a few seconds after, deciding that it doesn’t want to let the app finish doing the task to completion.
Arc can’t immediately stop what it’s doing - the various tasks get broken up into batches of work and it can only stop between batches. So once Arc gets told the task has expired (and is required by iOS to stop doing any more of it) it still finishes up the current batch of work. If there was only one batch of work to be done, then technically the job will be complete, even though iOS marked it as expired. So it can end up showing as incomplete even though all the work was finished.
That is very common with the coreMLModelUpdates task you see in the list. iOS almost always expires that task only seconds after starting it (super annoying!). But often that task only had one thing to do, so it’ll still end up having finished all of its work, even though it says “overdue by 19 days”.
The same can happen with the backups task. Although that’s much less common. So typically when the backups task says it’s overdue by X days, that’s usually correct, in the sense that it hasn’t managed to fully finish the job in that many days. Although the actual data in the backup might be almost up to date - it just hasn’t been given a chance to fully finish the job in that many days.
I think it’s best to wait. Or at least, wait until “Samples pending backup” is down really low. Like, maybe once it’s under 10,000 or so. Or if it says it’s actually completed the backups in the last day.
