It seems that possibly my issues might be covered in other posts. So I apologize if this is a duplicate. My concern is losing data. I was affected by the 3.6.0 crash. Now that the app stays open, it doesn’t seem to be recording at all, and I can’t seem to browse any past data. If I hit the back arrow, nothing happens. If I go to Calendar, it doesn’t show anything except this month.
Some other things to note:
Prior to 3.6.0 I wasn’t really opening the app that often as much as I used to, or correcting any issues.
I only started opening it once 3.6 hit and I was constantly getting the notification that iOS was stopping the App.
I should have a lot of data from years past
Its an iPhone 12 Pro Max
The Arc app is using 32GB of data of storage on my iPhone
The iCloud Drive Backup is only 30MB
The iCloud data is using 1.5GB
I don’t have an unusual amount of movements and places
Should I uninstall and reinstall? Should I be concerned about losing past data?
It sounds like perhaps the database has got messed up in some way. And given that you can’t browse the timeline data, it’s not going to be viable to try to clean it up within the app. Which leaves us the option of a fresh install then restore from backup.
Though to do this, it’ll make sense to do a manual restore of the data, week by week, rather than a normal “managed restore” of the full backup. Doing it bit by bit will allow you to spot problematic data and deal with it before it creates a bigger compounding mess.
Here’s the steps I think would be best:
Make sure you’ve got as much possible backed up, ready for restore. To do this, make sure the Settings → Backups, Import & Export view says it’s got a recent backup, eg “Last backup completed 1 day ago”.
If the last backup completed too long ago, we’ll want to get it up to date first before doing any delete and reinstall things. So to do this:
i. Turn off recording at the bottom of settings tab (it sounds like it’s failing to record clean new data anyway, and having no new data going in to the database will make it faster to catch up the backups).
ii. Plug the phone into power and leave Arc in the foreground. Ideally turn off screen lock in your phones settings, so Arc can stay in foreground uninterrupted until it finishes the backup. You can tell it’s actively backing up by checking the Settings → Backup, Import & Export view again, where it’ll say something like “Backing up now”.
Once you’ve got that full recent backup, it’ll be safe to delete the app then reinstall. Note that if you also use Arc Mini you’ll need to delete that too, otherwise iOS will keep the database on the phone, because shared “App Group” data isn’t deleted until all apps in the group are deleted (in this case Arc App and Arc Mini share an App Group).
Once you’ve got a fresh install of Arc App with no data in it, it’ll be able to record new data without problem. So we can then move on to restoring the backup week by week. To do this, don’t tap Restore on the pink bar at the top of the screen. That’s the “managed restore”, which will just put things back the way they were before, which isn’t where we want to end up. Instead to go the Settings tab → Backup, Import & Export, and open the File Importer.
Oh I missed a step. On iCloud Drive, your Backups folder will have been renamed to “Previous Backups” when you reinstalled Arc App. What you’ll want to do is make a copy of that Previous Backups folder, then move that copy into the “Import” folder. The “File Importer” view looks for anything in the “Import” folder, so it’ll see all the backup files in there and allow you to import them one by one.
In the “File Importer” view, don’t tap “Import All” on anything. That’s basically just what the managed restore does - it taps “Import All” for you and lets it run. In this case we don’t want that, we want to import the LocomotionSample files one by one. Each file is a week of data, so you can tap on one of the files, wait for it to finish importing, then go check on that data in the timeline view.
Look over that imported data in the timeline view and make sure it’s sane. It’s likely that there’s going to be a mess somewhere in there that’s screwed up the database in such a way that the processing engine can’t recover. You’ll know it when you see it - it’ll probably be a mega visit, like a visit to one place that spans way too many days. To clean up a mega visit:
i. Tap into the visit’s details view from timeline view, then tap Edit, then “Edit individual segments”. From there you’ll want to tap on segments that you know should be other separate visits to other places, and correct them as appropriate. You can probably also do this by tapping on the circles on the map on details view, if you recognise obvious visit circles in there that can be assigned to more appropriate places.
ii. Just keep repeating that cleanup of individual segments until the mega visit is back to being a sensible visit of sensible size and content. Once that’s done, the messy data that was confusing/breaking the processing engine will be resolved and won’t get in the way of recording/processing new data.
Keep repeating this week by week restore in “File Importer” for each LocomotionSamples week file until they’re all done. Tap to import, wait for import to complete, check the deta in timeline view and clean up as appropriate.
In File Importer you can safely tap on “Import All” for “Notes” files and “Day Summary Files”. Those backup files are safe and easy to restore all in one go, at any stage of the restore process. If you use notes at all you’ll definitely want those restored! The “Day Summary Files” are less important - they’re pretty much just there to remember which days you’ve marked as favourite days.
Note: Don’t try to tap on “Place” files or “Timeline Item” files in the File Importer. Those are imported automatically, as dependents of the LocomotionSample files, so there’s nothing gained by trying to import those files manually.
Ok, that should be the whole of it. Definitely let me know how you get on!
I’ve started the import but I noticed quickly that some weeks have a lot of files associated with it. The first few weeks was only one file but then went to about 18 files per week. Attached is a screenshot. I looked at the whole history and there are not many weeks that only contain one file. I started importing a week that had several files. The first file only had a day or so. The second file looked like it contained most of the week but then as I scroll through it, it crashes.
Ugh. Sorry about that, my bad. There was a bug in the backup system that made it not realise a file already existed if the file wasn’t locally on the phone, and needed to be downloaded from iCloud first. (It was checking for the file on iCloud Drive, but a typo meant it was missing them). The bug is luckily harmless, but made one hell of a mess of the backup folders
That bug was fixed a few updates ago, but existing backups still have the clutter. Anyway, you can either tap to import all of the duplicate files for each week, or you can find the newest one and just import that. Importing all the duplicates is fine, because the importer will only import over the top of existing data if it’s newer than what’s already in the database. So for each sample inside the file it checks to see if it’s newer or not, updating the existing data if so, and skipping it if not. So by importing the duplicated files you still end up with the newest versions of all the data in the end, but it’s going to be tedious.
So to find the newest version of each week file… okay, damn, looks like the process is quite tedious in the Files app. You have to hold on each file until the menu comes up, then select “Get info” and look at the “modified” date. That’s going to be beyond tedious for all those files. Let’s see what it looks like on the iCloud.com website… Okay, nope, that’s not any better. If you have a Mac you’ll be able to do it much easier in Finder, by viewing the folder sorted by modified date descending, like this:
Or if you’re on a PC, presumably there’s also a way to get iCloud Drive connected on Windows, and view the files in there. Either Mac or PC will be much easier than in the Files app or on iCloud.com
You could also just import the week file with the highest trailing number (eg 2022-W38 18.json.gz). That should be the newest file, given iOS automatically gives each duplicate a new number one higher than the previous. I’m just a bit wary of trusting iOS outright, given I work with the guts of iOS every day and it’s really not a job that builds trust with iOS
I’ve been painstakingly going through and import each week. At first I was correct all of the locations and unconfirmed items, But that was taking way too long, considering I have about 7 years of data to go through.
I recently ran into some weeks that could not be imported. I tried the “retry import ignoring missing dependents.” and force closing the app, but I get the same error everytime.
I am thinking of just skipping these weeks. It probably wouldn’t be a big deal. Thoughts?
Yeah it’s a lot of work. Probably best to just check that the data came in looking reasonably sensible and not entirely broken, then move to the next week.
Yep, that’s fine. I’m not sure why they’re not finishing importing with the “ignoring missing dependents” button - that should get the import to complete regardless. But it’s fine to skip those weeks, given they’re already 99% imported. You can see on the details view it says “Existing samples skipped: 6601”, which means 6601 samples from the file have already been imported, and “Errored samples” is only 1. One missing sample is no big deal, given each sample is a snapshot of only a few seconds of recorded time (each sample is generally a snapshot of 2 to 60 seconds of incoming raw data).
It’s curious though that it also says “Deferred samples”. Deferred means it’s waiting for a dependent file, and it knows that file exists, but it won’t proceed with importing those samples until their dependent files have downloaded from iCloud Drive. The File Importer will be asking iCloud Drive to download those files each time, but downloads can get stuck in some sort of limbo sometimes.
You could try going into the Files app on your phone and long pressing on the Import folder then selecting “Download Now”, to force it to download all files. And if that still doesn’t fix the “deferred samples” within 5-10 mins or so, you could try restarting the phone. If iCloud Drive gets stuck in a weird limbo, not being able to download some files, it generally snaps out of it after the phone is restarted.
But in general, yeah, those week files look pretty good already - there’s only 1 errored sample (errored meaning the dependent file doesn’t exist so it can’t be imported), and only 32 to 51 samples waiting for dependent files. In that grand scheme of things that’s a tiny amount of data to miss. If it were me, I’d try the “Download All” trick in Files, and restarting the phone if that doesn’t work, then if it still doesn’t work I’d just call it a small loss and move on. Not worth losing even more time over a few minutes of data out of seven years worth.
Alright so I have finally finished importing everything and everything is recording normally.
After I finished, I turned on the iCloud drive backup and it took a few hours to complete when i plugged it in and turned off the auto-lock. I also had to tell the auto importer to ignore the previous backups folder.
Here are the storage numbers:
Arc and Arc mini are using 1225MB and15 MB of iPhone storage
Arc App is using 1.8GB on iCloud
Arc and Arc mini are each using 3.37GB and 3.37GB in my phones iCloud backup
The iCloud Drive Arc App Backups folder is using 288MB.
Those numbers look a bit confusing to me though. The 1.8GB on iCloud seem oddly high, given the iCloud Drive backups folder is 288MB. Unless there’s a bunch of “Previous Backups” folders taking up the rest of the space?
The 3.37GB in your phone’s iCloud backup sounds about right. Though that’s likely shared between the two apps, so only 3.37GB in total rather than 3.37GB each. That space will be the database that contains all your timeline data, which is shared between the two apps.
This sounds a bit weird too. Mini using 15MB sounds great, but Arc App using 1225MB seems oddly high. That number won’t be including the shared database (the 3.37GB, which will be listed separately somewhere in iPhone Storage), so over a gigabyte for just Arc App excluding the shared database is … quite high. I’m not sure what that’s about.
But other than taking up a bit of mystery extra space, it looks and sounds all good! Yay
Ah yes, I misspoke earlier. The arc app is only using 95MB of iphone storage. I do see the shared database listed as Digital Nomad that is using 3.55GB. Yeah I am not sure what is going on with the iCloud storage.
