I’m trying to migrate my Arc data to a new phone and I am very stuck. I thought it was just crashing due to importing but it seems like it’s crashing just in general as well.
I tried the Restore route but noticed that I’d keep having to start the process again not long after, and Arc would start the Restore from scratch each time.
I tried going the manual importing route (after renaming the Restore folder back to Previous Backups but Arc always crashes (or freezes then crashes) while trying to import a sample file.
Out of curiosity, I opened Arc then just left it open and it would consistently crash after about 45 seconds.
I’ve also tried going into the Import modal and waiting for the files to load (which takes awhile, especially since the ones I want, i.e. the Sample Files, load last!), and once I finally manage to tap a file, it crashes between 10-60 seconds later.
I thought maybe it was an attention thing, i.e. it would crash 45 seconds after I stopped looking at my phone with Arc open, but even opening it and looking at the app it crashed after 35 seconds.
This is on an iPhone 15 Pro Max and iOS 17.0.2.
I’m at a loss for what to do to get my data transferred over!
Step 1: Restart your phone. If anything really weird and unreliable is happening with an iPhone, it’ll probably go away with a reboot. Especially if it’s soon after an iOS update. (Though if it’s a new phone, perhaps it was already on latest iOS?)
For the restore from backup, the managed restore (the one started from the pink button at the top of timeline view) won’t actually restart from the beginning if it gets interrupted. It’ll look like it’s restarting from the beginning, but really it is picking up where it left off. So it will eventually get to the end, if you keep restarting it.
Though if iOS is terminating the app after 45 seconds that’s going to be a lot of restarts
I think if it were me, and I couldn’t find a way to stop iOS terminating the app, I’d go with the manual restore process, but only tap on one samples week file at a time, waiting until one completes before attempting the next. Then gradually inch my way through like that.
Though really we need to figure out why the app is getting killed so often. Especially on a new iPhone 15 Pro Max, that really really shouldn’t be happening. Terminations during restore are typically due to either the app using too much memory, or using too much CPU, neither of which are very likely with a 15 Pro Max!
It could possibly be that the restore files aren’t downloaded locally yet, and are instead still only in remote iCloud Drive. It could be that iOS is stalling on the downloading of the files, which then triggers a “system watchdog” (a system that watches out for apps doing the wrong thing for too many seconds, like freezing or sustained high CPU or some such).
If it’s a case of the iCloud Drive download stalling things out, then you can get around that by going into the Files app, navigating to the Arc App folder, then long pressing on “Backups” and selecting “Download Now”. That will queue up all the files for immediate download, and for whatever reason the Files app can get away with doing that without getting terminated (while Arc seemingly can’t).
Good to know that Restore isn’t starting from scratch each time.
For both Restore and Import, part of the issue is that it takes long enough for either process to start (Arc is laggy and stalls) and the scanning for importable files process takes awhile, so that it’s a miracle if I even manage to make it to tapping on the appropriate file in Import before it crashes!
Feedback on File Import: if the Samples are really the important ones to import first, could those be loaded first? Because right now I’m just staring at Arc while it loads the Importable Files from the bottom up starting with Places and hoping that it makes it to Samples before another crash.
I did try the Download Now in Files, but that’ll be more relevant to whenever I can hopefully get Arc to stay open long enough in general. (Thanks for that tip too. I was doing a long-winded attempt at downloading the entire Restore folder yesterday by navigating into each folder manually to try to tell iCloud I wanted these files asap please! While watching TV because it would’ve been mind-numbing otherwise.)
So next steps: I’m now thinking that I should just reset my phone and start from scratch, but I need your help understanding how this works on the Arc end.
if I have Arc on both my old phone and new phone, are they backing up to the same folder or different ones?
my old phone has everything backed up for Arc with the exception of yesterday, where my new phone was the one recording.
my new phone says “first backup incomplete” but I don’t know if that means it hasn’t started yet or if that data is mixed in with the old phone’s data.
I only want yesterday’s data from my new phone. Because when I first started trying to restore, it started with a file from 2019 so now thinks that in between every day in the past week, I’ve flown cross country multiple times. Ie the new phone Arc aside from yesterday’s daytime outings is a mess!
So! If I export yesterday’s data within Arc as GPX or JSON, would it make sense to then delete Arc & Mini, do a backup of my phone to iCloud, then wipe and reset my phone and just not open Arc for as long as possible and hope it just opens with the correct data in it? And then I’m not sure how to manually replace the data from yesterday.
All this is because Arc is unusable in its current state. Even just editing yesterday’s outings took a lot of toggling between Mini and Arc. (Mini didn’t seem to crash as quickly or perhaps since it’s Mini, I could get to where I needed more quickly before the crash.)
Ok! So I do see two backup folders now, so I guess each phone is making their own backup?
And I tested the single day export from my new phone and tried to do an import on my old phone but the gpx/json files don’t show up so there so there goes that theory.
Now I’m really wishing I had kept my old phone recording!
Is there a way to add yesterday’s data from my new phone with all the data from my old phone? With the caveat that the new phone’s data is messed up aside from that one day.
I was just about to make this change (it sounds a good idea to me), when I realised it won’t help. To import Sample files, the dependent TimelineItem and Place files need to be there, ready to go. So even if the Sample files loaded first, you’d still be stuck waiting for the others before you could do a successful import.
That’s what you used to have to do, before iOS 16 added the “Download now” option. Extremely tedious!
I wouldn’t bother going that far for Arc’s sake. Unless there’s other things going wrong on the phone, that make you think a full fresh iOS install might help.
For Arc, if you’re thinking the database has become a mess, you can just delete Arc (and Mini) to get rid of the local database, then reinstall them to start a fully fresh restore.
Same folder. So you ideally don’t want the app running on both phones at the same time! It’ll just make a mess of the Backups folder. Though for your restore you’ll be working from a “Previous Backups” folder, which isn’t being modified by anything. (The “managed restore” will delete files as it progresses through the restore, but it’s deleting files from a copy, not the original Previous Backups folder).
Ok, in that case I’d say make sure you get a copy of the current Backups folder, so you can import that using the File Importer manually, after you’ve restored the rest.
It won’t be aware of the backup data from the old phone. It only keeps track of the “last backed up” date on each piece of data in the local Arc database. Though it definitely will have started! It just means it hasn’t got to the end of the line yet.
But as to how much progress it has made so far, Arc can’t tell you that (it’s too time/energy expensive to calculate), but you can tell by looking at the LocomotionSamples folder in the Backups folder, to see which weeks are/aren’t there.
Though from the new phone you’ll only really want the most recent Samples week (and the dependent TimelineItem and Place files). Backing up the TimelineItem and Place files is very quick, so those ones should hopefully already be complete. It’s the LocomotionSamples files that take the longest, so if that’s there then you should be ok. And if the TimelineItem and/or Place files are missing, those can be recreated by Arc’s processing engine if necessary. (Though that does create a minor mess, with your timeline item editing lost, and the items essentially reverting to how they would have appeared right after recording, without any cleanup work done on them).
Yeah that sounds a mess. My gut instinct is I’d want to start again, just in case the processing engine has taken it upon itself to try to make sense of any of the big gaps, and done something silly.
My instinct would be to start with a fresh Arc install (also deleting Mini, so that the local database is definitely deleted), then do a manual restore with the File Importer, one week at a time, working from most recent week backwards. (Which I think is what the managed restore is supposed to do anyway, but when things are going weird I feel more comfortable doing it myself, double checking that each step has succeeded).
Unfortunately Arc can’t import from GPX yet, nor (embarrassingly) its own ArcJSON export files. It can only import from the JSON files in the iCloud Drive backups. (That second one is high on the todos, but weirdly hardly anyone ever notices or cares, so I never get around to it )
But in general yeah your plan sounds good. If the old phone has done a proper full iOS iCloud Backup (ie not Arc’s backup but iOS’s own), then you’ll have the old phone’s full Arc database in iCloud, and ready to restore onto the new phone with a fresh iOS install. And holding off on opening Arc (or Mini) for as long as possible will definitely also help.
Basically what I think goes wrong with setting up new phones from an iOS iCloud Backup is that Arc (or Mini) gets opened (or launched automatically by iOS) before the database has been restored from the iCloud Backup, which then results in Arc creating a new empty database. At which point all is lost - the iOS iCloud Backup restore will give up on restoring the full database, and you’ll be stuck with the new empty one. (I haven’t figured out any way around this problem yet - it’s a nasty one.)
If you’ve got a copy of the current Backups folder from the new phone, you can use the File Importer to put that data back, on top of the restored data from the old phone. Just copy it into the Import folder on iCloud Drive and the File Importer will see those files and let you import them. So it’s possible to import/restore from multiple backups, even from different devices.
Sorry about all this! Really not sure why Arc (and Mini too??) are getting killed like that. It really complicates an already quite large task!
Oh one other option: Do a backup of your old phone to your computer over USB, the old Finder/iTunes way. Then setup the new phone by restoring from that backup instead of an iCloud Drive backup.
That should be much faster, and less at risk of the problem of Arc/Mini getting launched and creating a new empty database before the restore has finished.
Though I haven’t done one of those USB Finder/iTunes backups and restores in years, so I don’t really know much about them anymore. But my hunch is that would make it both much faster and much less at risk of the above problem.
Question in case I can catch you while you’re still online: where’s the current Backups folder housed? (My iCloud Arc App folder now shows 4 Previous Backups folders. But I don’t understand where to look for the current Backup.)
I’m trying to understand how Arc will know which data is correct. Ie I’m scared to import week 39 from the new phone into the old phone data lest it also brings in the wonky “you’ve been traveling for 50 months cross country” at the start of every new day mess!
And thank you for your detailed response! I’ll reply more completely when my brain is fully functioning!
Edit: Follow up question: Could I just cut Thursday’s data only (accounting for the UTC time) from my new phone’s 2023-W39.json.gz file, save it as a new 2023-W39.json, put that into the Import folder, then import that file onto my old phone? Would it just replace/update the Thursday data only (and leave the rest of the week alone since there’d be no data from the rest of that week in that edited file)?
It’ll be in iCloud Drive -> Arc App -> Backups. Arc’s backups aren’t turned on by default (because they can consume such a large amount of the default free iCloud storage limit), so if the folder is missing then it’ll mean it’s not turned on yet.
Yep! You could hand edit the JSON LocomotionSamples file. I’ll just have a poke around in one of mine to get a sense of the needed edits now…
Ok, eek, it’s kind of overwhelming. But… it looks like you’d “just” need to delete the samples with a “date” value older than the boundary date you want to work from.
The importer works from the LocomotionSamples file as the basis of what to import, so if something isn’t in the samples file, it’s not going to get imported, regardless of whether there’s a TimelineItem file for it. (The TimelineItem files get imported based on a sample referencing that file, so if no sample to import, no timeline item gets imported).
You can un-gzip the file just by double tapping it in Finder. But to re-gzip it you probably need to do that from Terminal. Just trying now to fuss about and actually find the iCloud Drive folder from command line is proving to be not fun, so I think it’d be easiest to copy the file to somewhere accessible first (like say your Desktop folder), un-gzip it, edit it, re-gzip it there, then copy it back over the original on iCloud Drive. To re-gzip it, it’s just gzip 2023-W39.json in Terminal - that part at least is easy.
I think I’d been turning the backups on and off because I couldn’t decide my old phone new phone strategy but I did see it pop back in, thank you!
And it worked!! I figured out which segment from the new phone I needed then stuck that into the old phone’s JSON. I also had to put the new phone’s “Previous Backups” folder into the Import folder so that it could pull in whatever additional files were being referenced which is what I think you said as well. I did have to go in and re-parse out the events of that day, but that was definitely much easier than trying to go back in on my new phone to re-parse out 50 months of locations! And turns out I didn’t need to re-gzip it, Arc imports it just fine as json!
Question: I’m now trying to get my old phone to do a full clean backup. Arc says the backup finished 2 hours ago yet the Backups folder is quite sparse. Is it only backing up things that have changed? If so, how do I go about getting a full and complete backup made? (So that when I reinstall Arc on my new phone, it ideally just goes, “oh look there’s my data!” Or if that’s far too idealistic, that I can just do the Restore process with the one Backups folder.)
(And replying more completely is still on the docket. Figuring out the above took all my brain power today… JSON files and formatting are not even remotely part of my normal life!)
Hmm. That sounds like what would happen if the Backups folder was deleted. Arc would think that it’d backed up a whole lot of stuff but that stuff wouldn’t be there anymore, and only the data that’d been backed up after the folder was deleted would be there.
If you toggle off Arc’s backups setting, then wait a few minutes (there’s a stand down timer, to avoid immediately clobbering the backups state when people flick the setting on and off while thinking) Arc will reset all of the “last backed up” dates in its database, marking everything as needing to be backed up again. Doing that will also rename the Backups folder to “Previous Backups”, so you can know the stand down timer has finished once the Backup folder has been renamed. Then you can turn it back on and it’ll start a completely fresh backup with fresh Backups folder.
Unfortunately that does mean you’ll have to wait through a full backup from the beginning. But you can speed that up by turning the phone’s screen auto lock off and leaving Arc in the foreground while plugged in to power. A full backup can be churned through in a couple of hours (if the stars align and you’re especially lucky with iOS’s mood) if Arc is in the foreground the whole time. Oh, also it’d probably help to have recording turned off, given it’s the old phone and you don’t want it creating new data, further delaying the backup completion.
OH that explains why I suddenly have so many Previous Backups folders. It’s my own doing!
Okay so I turned off the backup, waited for the Backups folder to be renamed, then turned the backup back on. Is it normal for it to still have the “last backup # hours ago” message?
Hmm. If no new Backups folder yet, then it hasn’t started doing a new backup yet. I wonder why…
I wonder if it thinks it finished the last backup too recently (like within the last day), so it’s not going to try to start backing up in the foreground. Backing up when the app is in the foreground is something it only does if the backup completion is overdue - it’s not good to have Arc doing heavy stuff like that while you’re trying to use it, so the preference is to only do the backups during the normal scheduled background task.
But if you’ve reset the … ooh, I see it in the code now. It’s using the date of the last time the scheduled background task completed for that “last backup completed” thing. Which isn’t aware of the backups being reset. Ugh.
Ok, I found where it decides whether to run the backups in the foreground. It’s got a 12 hour limit - so if the backups haven’t completed within the past 12 hours, it’ll also run the backups in the foreground. 12 hours doesn’t seem too bad. How far off that is it?
No dice! It looked like it was going well and then it decided that it was complete an hour ago even though it just has 2019 and some of 2020 backed up. Any other ideas for how to get an actual full backup going?
Side note: There’s a bug in the onboarding where the Health permissions pops up briefly but disappears too quickly for any action. I then get stuck on the onboarding Health explanation screen with no way to move forward except to exit the
It’s more about being able to navigate to the desired Sample file while the dependents are loading. While troubleshooting, I had to wait for the many many dependents to load before the Sample files loaded and then it took me awhile to navigate to an older 2019 file. (This is when I was trying to “close the gap” of the 50-month conglomerate segment by loading the Sample file from the week that 50-month segment started.) In a similar vein, I deleted Arc to start from scratch and started with a Restore with all files completely downloaded (to avoid what I thought was the issue before, i.e. a 2019 file getting imported because more recent files weren’t available) and Arc STILL came back with that 50-month segment at the start of every day! So now it looks like I’ll have to do this manually, i.e. import 1 week then exit, then navigate to that week to see if it’s the file causing the issue or not… this isn’t going to be a quick process in one sitting, so that means that every time I go back to do it, I’ll have to wait for the dependents to load before I can find the next file I need. And it makes more sense to spend the time navigating to the next file while Arc is loading dependents below.
Yeah, I realized that while most things on my phone would be fine, I would lose multiple days of iMessages which is a nonstarter, so hopefully I can get Arc to work!
So since the weird 50-month conglomerate segment returned again even after making sure that all Sample files were on my phone (and deleting both Arc and Mini, does this mean the issue is in my backup files?
Still stuck on this one. The most complete backup I have is from before my new phone. On the bright (?) side, I’ve been sick for all but one day since then so I haven’t gone anywhere so that one missing day is still the only day that I’d have to figure out. On the not bright side, it’s a bit moot if I can’t get rid of that 50-month mega segment on the new-phone import.
I care!! I care greatly!! Make that many votes from me! This would be amazing because then I could export a specific ArcJSON file (for the week even?) so that I don’t have to wait for the full backup (if it ever happens) just to get a single week’s data i.e. Sample file. And I wouldn’t be freaked out if I was missing a day in one device’s data, I could just export it. (While I was successful in getting the missing day onto my old phone, I haven’t been able to get a backup containing the week with that day, nor was I able to replicate my success in adding that missing day back to my new phone. Agh!
Next time I get a new phone I’ll do this
Interestingly enough, it created a new Backups folder and started backing up sooner! I still haven’t been able to get a complete Backup though.
Wow. Uh… I’ve never heard of that happening before. That should be impossible. Just trying to think of any way that could happen. Just looking at the code, yeah, really has to be impossible. All the backupLastSaved values in the entire database are nulled out, 30 seconds after the backups setting is toggled off. That nulling out is done as a single database transaction, all database rows at once. The database engine (SQLite) is designed to make it impossible for a transaction like that to only partially complete.
Unless… Are you looking at the files in iCloud Drive on your computer or phone? It might be that the files are all there on the phone, but haven’t synced to iCloud Drive yet. It’ll be worth checking in Files app on the phone, just to be sure. Otherwise we’re in a mystery situation where I’m at a loss as to how it could happen or how to get out of it.
If the files are missing on the phone too, then the only thing I can think to try is to toggle off the backups setting again, wait 1-2 minutes (the stand down timer is only 30 seconds, I noticed just now, but give a minute or two to be on the safe side), then toggle it back on again
Hmm. Another mystery! I’ll do some testing here to see if I can reproduce the problem.
If the weird 50-month segment isn’t on your old phone, then it won’t be in the backup files from that phone either. Though it could get into backup files created by the new phone - ie those backups reflecting the new weird timeline state. So you’d want to avoid restoring from those ones.
I think best way to avoid that mega gap thing would be to import the weeks from most recent first. Those sorts of mega gap items can happen when there’s some timeline data from long ago, then also new recent timeline data, and the processing engine then tries to bridge the gap between them. So if the data is imported from newest first, then there’s no gap to bridge.
Ah yeah, that’ll have been the scheduled background task then! At least that ran on time for once
Sorry about all this fuss btw. It really is a nightmare situation. But unfortunately one that it’s very difficult to fully avoid, given the large amounts of data involved.
Related anecdote: I was working last month on adding iCloud sync to one of my new apps (This & That - the Health data correlations app). But because that app creates such a large amount of data (120 different correlations between every combination of Health app data types, so about 3 million correlations), iCloud’s CloudKit service could never get even close to finishing the sync. It was absolutely impossible, and I had to give up on the feature. And Arc’s database is orders of magnitude larger than that one!
Arc’s current backup system is adequate and efficient when everything goes right. Which has taken years to fine tune and achieve. But when it doesn’t go right, the struggle to work around it and clean it up is … yeah, really not fun. It’s just too big
Definitely phone. The same phone that’s doing the backup!
So question: can I combine files from various backups and still have dependents pull in accurately? The backup that decided that it finished 3 hours ago has a lot more samples than prior backup attempts, and between the last two, and the original backup that ended last week, I might actually be fully covered for samples. The most recent backup attempt only has a handful of the dependents though. So could I pull dependents in from whatever prior backup has the most dependent files? Basically merging several backups into one then using that to manually import. (Oof.)
Hmm, in my most recent attempt to Restore, it was a newly installed Arc using the last complete backup from my old phone (ie missing a few days / from before I ever turned on the new phone).
This is my next attempt for today!
I appreciate you trying to troubleshoot with me! I’ve definitely learned a lot more about the backup system. And I am trying to accept potentially messed up / inaccurate / missing data depending on how successful today’s attempts are! In the grand scheme of things it is so not a big deal, but convincing my Type A self is another matter.
I’m trying the combo method (using the latest [incomplete] backup from my old phone and combining that with the last full backup from my old phone (that goes up til last Thursday). It’s mostly working for the weeks I’ve tried though it’s going slowly due to having to check that none of the weeks are triggering a mega segment moving forward, plus there are a lot of confirmations that I have to redo.
Problem/Question: Oddly enough, the week 39 import resulted in a partial Friday (it’s missing all the out-of-home activity) and empty Saturday on my new phone, despite my old phone now having the full Friday and Saturday data. I checked the JSON file and it HAS that data in it but it’s not importing! I tried “import ignoring missing dependents” at first. Then when Friday still showed up mostly empty, I tracked down the missing dependents and put them in the import folder under “TimelineItem”. Then I tried renaming the file, then saving a separate file with only that missing data, then changing the lastSaved date, all to no avail. Is there something wrong with the backup file? I’m out of ideas for how to hack the Friday data back in!
I’ve been trawling a lot of old threads on backups and imports for troubleshooting ideas. There was a time when that Arc JSON import was high on your todos so it’d be amazing if it could reclaim that high position. I get that todo priorities are highly influenced by which features get the most votes but given that this is something that was intended to be part of the original Backup/Import features (and totally understandable why it was left out at the time), it’d be amazing to get it reinstated.
I’ve only imported a month and a half so far, but I’m already seeing days that didn’t import well and maybe it wouldn’t make a difference, but it seems like it would be helpful to be able to export that specific day from my old phone and import it directly into the new phone. Not to mention the ongoing issue of my missing Friday from last week. (Nothing as terrible as that 50-month mega segment, thankfully, but, for example, what used to be 12 segments on my old phone imported as 4 on my new phone that I guess I have to manually go in and separate out.)
)
The most complete backup I have is from before my new phone. On the bright (?) side, I’ve been sick for all but one day since then so I haven’t gone anywhere so that one missing day is still the only day that I’d have to figure out. On the not bright side, it’s a bit moot if I can’t get rid of that 50-month mega segment on the new-phone import.
