This restore from backup process is now verging on impossibly tedious. Going through and restoring week-by-week (which I still have no evidence is actually successfully happening, given I don’t see that data populating in the calendar view, nor the timeline view).
Where I’m up to now is slowly going through each week and importing them individually. Each week takes somewhere between 6-15 minutes to restore. But what is continually happening is the app will crash (or iOS terminates it) when I’m trying to import a week. The week’s data the crash occurs on is not constant and might import just fine the next round of importation attempts. So it’s not like I can even weed these weeks out.
What is super frustrating here is that the file importer, nor the calendar view, nor the timeline view shows any history that ANYTHING has been imported before the crash. The list of ticks next to each week vanishes. So I start from the very beginning again. Though one slight blessing is that subsequent re-imports of weeks imported prior to the last crash import in only a few seconds - so maybe I don’t need to reimport those weeks and they are actually already there?
I’m now on about round 45 of the app crashing and me starting from the beginning of the process again, unsure if the weeks successfully imported before the crash are still in fact there. So I go from the top again. IT’S INFURIATING and very reminiscent of the experience of @patrickplaggenborg
Each round I move slightly further down the intimidatingly long list of Sample Files, diligently navigating its endless list of duplicates, with some weeks having over 25 duplicate sample files. All the while I keep my phone connected to a charger with the display perma-illuminated, using an iPad as a step-in replacement iOS device from which I can continue to run my life, hardly ideal. The process is excruciating. I’m really at a loss for what to do next and close to just giving up on my 10+ years of data. This would be an EXTREMELY disappointing outcome. Please Matt, I’m a paid up lifetime subscriber. I like so many things about Arc and the principle of privacy on which it is built. There must be a better way.
2 Likes
Sorry to hear it’s giving you trouble!
Yep. When importing a file, the importer first checks to see if the data is already in the database. And will only reimport if the data in the JSON file has newer dates than the data in the database, otherwise it will just quickly skip over it.
So once a week has imported, there’s no need to do it again.
Though we do need to figure out why the calendar view isn’t updating correctly! Something fishy going on there.
Unfortunately what we’re dealing with here is more data than iOS is built to cope with. Arc database files can be upwards of 6 GB or larger (my own is 6 GB, with data from 2016 onwards). That’s just too big for what iOS expects apps to handle.
With the rebuild of LocoKit, Arc’s underlying recording and processing engine, in Arc Timeline Recorder (and the upcoming Arc Timeline Editor) I’m also restructuring the database, to hopefully better manage the scale of data. But it’s ultimately never going to be a simple thing to restore this much data from backup.
One of the options that sometimes gets brought up is simply copying the entire database file, in its native state, and using that as the backup. But the time it takes to do that for the large databases most of us have is longer than iOS will allow, so any attempt to do that will result in the app being terminated and the copy failing.
The current process is not ideal, but there are also no easy alternatives. Arc simply goes too big on data, compared to what iOS wants to handle (or allows us to handle).
Anyway, back on topic: First let’s figure out what’s going on with the calendar view, and that cached “earliest data date”.
Thanks for the quick reply, especially on a Sunday Matt 
I must apologise if my frustration with the process was shining through a little too brightly in that last message. Thanks for the context.
Yes, I agree getting to the bottom of the calendar issue should happen before anything else. I need to see evidence I’m not just wasting my time and actually see my hard fought imports appearing in the calendar view 
I’ve force quit the app to see if that changed anything, alas the calendar view still appears as in my screen shot from earlier.
No worries. I understand it’s a frustrating process. I’ve sat through it a hundred times during testing. It can really be a slog, when things don’t go right!
Hmm. This is confusing. That cached “earliest data date” is for the earliest TimelineItem found in the database. Which could be missing if the TimelineItem files aren’t there. But if the TimelineItem files are missing, then the weekly samples file import would error, due to missing dependents. And your screenshot shows that that hasn’t happened - the weekly sample files are importing without error.
So something is happening that I don’t understand…
What happens if you try to swipe back on the timeline view, to before the earliest date the calendar view currently allows you to navigate to? Or if you tap the “<” button to do the same as swiping back?
What you’re seeing is something I’ve never seen before, and I can’t explain it yet. So I’m going to keep thinking…
Oh, do you still have the old phone, with Arc on it? If so, you could do a fresh full backup on that phone, to get a fresh, clean backup with no duplicate files. That at least would import/restore much faster. It’s typically the duplicate weekly files that slow things down.
But that’s an aside as to this calendar view / earliest date thing. The data is clearly importing without error, but… why isn’t it accessible? Yeah, I’m going to keep pondering.
If I try and swipe back on the timeline view it won’t allow me to go back any further than the day I last installed the app - so it’s only showing new data. Again the same with the “<” icon.
Alas, I don’t have the old phone. Apple took it away and replaced it with a new unit.