Apologies if this is documented somewhere, but I can’t seem to find the answer.
How does the new import screen work? Arc missed a walk I took earlier today, fortunately I had a gpx from a workout app, which I placed in iCloud Drive/Arc App/Import, but the importer doesn’t see it.
Is gpx the correct format?
Thanks!
Bill
I’d also be keen to know how this works. I’ve also got some old Moves data that I’d like to import as well.
Hi! Apologies for the late reply. I’ve been having some health issues lately, which have destroyed my productivity 
There’s currently two separate importers - the old Moves data importer, and the new Arc JSON importer. Unfortunately there’s not yet a GPX importer, but that’s definitely on the todo list!
For importing Arc JSON files, create a new folder in the Import folder, named after the object type you’re importing. For example if you’re importing a Note file, then make a “Note” folder (ie “Arc App/Import/Note”), and drop the JSON file in there. The other object types are LocomotionSample, TimelineItem, and Place.
You can find the manual import view in Settings → Backup, Import & Export → Open File Importer. It will then show you a list of the files it’s found, and you can tap on them individually to import them.
@Camson, for the Moves data importer, you either need to put your moves_export.zip file into the Import folder, or if your moves_export is already unzipped, then Arc will look for it in a folder named moves_export (ie “Arc App/Import/moves_export”). Once Arc finds either the zip or the folder, it will automatically start the import process.
Thanks for replying Matt. Sorry to hear you’ve been unwell.
I have tried both the uploading the zip and unpacking the folder with the correct name and in the right subfolder, but neither method seem to start the import. I’m trying to remember whether this is because possibly I may have run the moves import in the past (possibly when I first started using Arc). I’m not entirely sure as I do remember trying a number of Moves replacements at the time.
Is there some way to check what the issue is?
Thanks.
Yeah I think that would stop it running a second time. I’ll check the source now - it’s literally years since I last looked at that code!
Ok, found it. Yeah it won’t start the “managed” Moves import if it’s already been done before (or rather, last run was with the most recent version of the importer).
But you can queue up individual Moves files for individual import, regardless of previous import status. The Moves importer looks for Moves storyline JSON files in the root of the Import folder, and imports them one by one (then deletes the file on success, so only drop copies in the root, not originals!)
So basically you can copy in individual Moves storyline JSON files and drop them directly into the Import folder, and Arc will start importing them on next … appear, on next foregrounding/appear of the app (I double checked, and it doesn’t wait for a fresh launch - it checks for the files on each foregrounding).
From memory, I think the weekly files (if there are any for Moves?) or the monthly files are best. The larger the JSON file, the greater the risk that it’ll consume too much memory and iOS will kill the app. The daily files are a bit tedious (and probably have a lot of overlap, due to visits overlapping midnight thus appearing in two day files), but the week and month files are more efficient, and if I remember right, the monthly files still fit within memory limits.
Thanks Matt. I dropped a copy of the weekly files from the Moves Export zip to the Import folder on iCloud and they all imported successfully while the app was open over a couple of hours. I’m now able to see the old data in Arc! Thanks for your help. 
Hi matt, I can’t seem to move my arc data to a new phone that I can’t use normal icloud restore with.
The importer finds the data when I put them in the folders as you wrote them (I think you should put this crucial info on how to use the importer in the importer itself, by the way), but then it goes into “errored” saying that the data is in the wrong format. It’s straight from arc’s exports – compressed or uncompressed, it doesn’t matter, it doesn’t work.
Do you have any suggestions on how to move the data?
Thanks
I agree! One of the many things I’ve fallen behind on 
For the import format, Arc can import the JSON files created by its own iCloud Drive backups system. On the old phone, turn on the iCloud Drive backups in Arc’s backup settings, then wait until it completes.
For importing JSON files created by Arc’s manual exporter, here’s some information on how to adapt the format to match the backups format:
ahhhh! so it’s from the backups, not from the json export! I’ll try. Thanks for the answer <3
Hi Matt,
so moving the backup to this new phone is an absolute pain, because the way you structured the backups, it’s a strain on the filesystem. Most ways of moving files around will start to panic with thousands of small files and folders, and it took me two days to actually find a way to move the four folders. Zipping them and unzipping doesn’t work, the iOS zip helper crashes (can you believe this? fucking 2022, from a $2 trillion company).
The worst offender is timelineitems, it just breaks everything.
Can you please consider making the backups a more consolidated package? Thanks
And I guess you already thought about making the importer accept arc’s own jsons 
Ugh, that sounds a hassle!
Where were you moving them from? Was it from different iCloud accounts?
When testing the new backup system, I had to repeatedly import whole backup folders, many times. So the way I did it was on my laptop. I’d copy the test backup folder into Arc’s iCloud Drive folder in Finder on my laptop, then it’d sync to iCloud over the next while. I forget how long it took each time - it was long enough to be annoying, but short enough that I could do a bunch of full restore tests each day.
It moving the folder from one phone to another, with different iCloud accounts… yeah, I bet that’d be a nightmare 
I’d probably want to use a Mac as an intermediary, like I did in testing. Syncing large folders (and as your found, dealing with large zips!) is much easier from a Mac, even though iPhone specs are insanely high these days.
Aside: The problem with iPhones and performance these days is that iOS is designed to only allow apps short bursts of high CPU/energy consumption, and then punishes the app for it if it oversteps (either immediately or later). So we’ve got all this incredible power in our pockets, but can only get away with using it for short bursts at a time.
Anyway, back on topic, yeah I’d definitely prefer TimelineItems to be grouped into one file per week, same as samples. The problem there is the importer needs the TimelineItem file for the sample that it’s importing (because TimelineItems are the parent objects for LocomotionSamples), and if the items aren’t one per file, I can’t use the filename as a quick way to find the right file.
It kind of gets fiddly, with different optimisations opening up new sets of problems. But yeah, would definitely be nice to do something about all those TimelineItem files!
This is how I did it, using iCloud with an intermediary mac! The problem is that there is no “download all” function on icloud on iphone, so in order to make the iphone download all 130000 files, I had to go through each subfolder and scroll to the end and wait for the files to download. That’s for each of the 256 folders. I just finished doing so, this whole thing is taking a little bit more than I expected
Now the importer is working but when I tap import all on the “2187 sample files”, some of the items will get an exclamation mark with a counter-clockwise arrow around it icon. If I tap those items, they just go back to the clock icon.
What’s going on?
I completely understand the reasoning for the splitting, can you consider putting the whole actual database in the backup then in the future? If it is for avoiding the multigigabyte updates of the file, icloud is delta so not to worry. You know, just to avoid having to use the filesystem as a database it’s probably easier to import, too.
Oh, Arc will do that for you! For each file it needs, it’ll start it downloading from iCloud if it’s not local. (Though there’s a weird performance problem in the iOS layer, which then causes high CPU usage in Arc if there’s a lot of files downloading, which led to me having to do multiple import attempts due to iOS terminating the app, if there were a lot of files not yet local. I might have worked around that though - it’s been a while since I worked on that code).
Just checking the code now, and that sounds like exclamationmark.arrow.circlepath, which means it’s “waiting”, meaning it’s waiting for a dependent file to be downloaded from iCloud. So either a sample waiting for a TimelineItem file, or a TimelineItem waiting for Place file. The dependent file will have been queued up for download, so iCloud should bring it local once it feels in the mood, then you can tap to try again.
Heh. I’d love to. But doing that would require copying the database file, which can be more than 1-2GB! That copy tends to go over iOS’s CPU time thresholds and causes the app to get terminated (and then subsequently punished, with higher risk of being terminated again later, for more minor infractions). So it’s something that can’t be safely done in the background except for with new/small databases. Yet another reason why I tend to grumble about iOS’s resource use limits - how we’ve got absurdly powerful pocket computers that we’re only allowed to use the power of for very short bursts
Aside: It’s been on my todos for more than a year now to break the main database file up by year. With one db file per year, doing whole file backups would probably (well, hopefully) be doable in the background. Plus it’d reduce index sizes and potentially give a nice performance boost.
Thanks for the replies!!
Unfortunately until i went in files and made it download all the files, the importer was having tons of issues saying it was missing the files… After I went and downloaded everything manually, it started to actually import stuff. Weird?
I’m now done transferring all of the data, it took many hours of babysitting.
i split the locomotion items by year and usually about a third actually imported, but if i redid an import all, I would always get the same “waiting” errors (keep in mind there are no files to be actually downloaded from Icloud) and eventually the app would crash.
So i would move the files that were imported and then restart the app and usually the remaining files would import without error.
It was… both intense and boring as hell 
I’ve read in another thread about the termination issues (another problem I keep on having), I guess there’s nothing much to do then.
I imagine you stagger the copy of the split files to avoid that cpu overloading…I hope that when you’ll come around to do the database split by year, you’ll find a way to stagger the db queries so that youll be able to have the same control on the cpui usage.
good luck
Yeah that is weird. Though… not entirely outside what I’ve experienced. Which is why I said above " iCloud should bring it local once it feels in the mood". During my testing I’d have test runs where all the files queued up for download, then downloaded at a reasonable pace and imported without fuss, then other times when everything seemed to stall and go nowhere.
iCloud does have rate limiting, which is visible to the app when using a service like CloudKit. But not visible when using iCloud Drive (the syncing/downloading is all managed at OS level not app level). So my assumption has been that it’s rate limiting kicking in. But it never felt a convincing argument, because I could do lots of intensive import/restore testing with no problems at all, then the next day have problems on the first attempt 
I experimented a lot when doing the restore testing with how many concurrent file imports the app should run (I think it’s set to only 2 at the moment). I could make the restore/import run much faster if I increased the concurrency up to as many cores as the device has, but then the risk of angering iOS with resource use thresholds also went through the roof, so it wasn’t worth it. And then as you found, sometimes swiping the app closed and starting again would get different results. There’s no winning 
Yeah I have high hopes for the database sharding. I suspect a lot of current performance / stability problems come back to the database files (and subsequently indexes) simply getting too large on disk.