JSON Import Format

This would be amazing, would help me understand the import process and how to create valid import data.

If the way in the current Arc import process to bulk import timeline items? I can’t seem to import a timelineItems.json file with multiple timeline items in it, even if I used the export schema, and there’s no “Import All” button for timeline items in the UI, forcing me to tap each item manually, which get tedious pretty fast.

I’ve been struggling with this quite a bit. The only way I’ve managed to get imported data to appear in Arc is to create a valid linked timeline series, import it, and then import an update for an existing timeline item to set its previous timeline item to the imported data. I’m not sure if Arc is relinking in the background or not, there’s no UI to show the progress, it just looks like the import failed.

It would be nice if the import UI showed the progress on the re-linking. Looking through Arc Mini, it looks like timelines items should be relinked if they’re broken when you look at the correct day. But I’ve failed to induce this behaviour, additionally the data I want to import is from years back, so tap “back” over 600 times doesn’t really work for me.

That would be super nice. Would also be cool is Locomotion samples could be marked as dirty or something, then have those sample excluded from ML updates.

The trick is to import the LocomotionSamples first, and the importer will find the relevant TimelineItem files itself. When importing a sample, it looks at the timelineItemId, checks the db to see if it has that item, and if not, looks for a matching JSON file to import along with the sample(s).

Likewise Places - if a TimelineItem file references a placeId, it’ll check the db for existing, then check for a matching JSON file. So importing a LocomotionSamples file will also trigger the import of all referenced TimelineItem and Place files.

Hm, that sounds weird. I wonder if this due to importing TimelineItem files before samples? TimelineItems with no associated samples won’t show up in the UI, due to being empty / zero duration. (They’ll also get auto deleted later, during some cleanup process, if they still have no associated samples).

Starting the import from LocomotionSamples file first might get around that weirdness.

Ok so I’ve dumped into Arc Mini all the importer and related views from Arc App.

The code is literally just dumped in without edit at this stage, so it won’t compile, because it references a bunch of Arc App specific little helpers and whatnots. (I’ve also removed them from the project target, to avoid them breaking the build).

ArcImporter is the brains. It’s just an ugly “manager” dumping ground of all file importing functions. Not exactly elegant architecture, but it got the job done :joy:

ImportList is the manual importer view that you’ve been using.

RestoreView is the managed importer that’s available from fresh installs, and is literally just a stripped down version of the manual importer, that “pushes all the buttons” itself. So that one’s the least useful to sift through.

ImportTask is the wrapper / state machine around each file being imported, that ArcImporter uses, and ImportList uses for knowing what to show in the UI.

ErrorLogView is an unfinished/unused all-in-one view for showing all errors from a managed restore (ie RestoreView). It’s not used by the manual importer, which already lets you see errors individually per import task.

When it comes time to hack this stuff up to deal gracefully with different file formats, I think that big ugly single file ArcImporter will need to be split up.

It’s all working on the assumption it can just take a JSON file and directly decode it to the appropriate TimelineObject type (array of LocomotionSamples, a TimelineItem, Place, Note, etc). Which won’t be possible with other file formats - they’ll need an intermediary layer.

Which I do actually kinda have, in the very old Moves App data importer. But that was hacked up in a panic in a single weekend, after Facebook announced out of the blue that they were closing Moves down and my inbox filled up with 10,000 emails. It’s elegant work for a panicked weekend, but it’s not elegant enough to be reusable.

Actually I should still drop it into the Arc Mini repo. I’ll have a sift over it now, and see what it looks like…

Ahhhh! That’s what in was missing. I assume I need to name the timeline item files correctly? (Don’t answer that, I’ll dig through the import code!).

I’ll need to do a little more investigation on this. My description of the behaviour isn’t quite accurate. But without the import code it’s difficult to figure out why imports were failing. I’ll compare what I’m generating to what’s expected and see if I can figure out what’s breaking.

Oh perfect, this is gonna be so helpful!

Just as a heads up, I would love to dive into this and contribute back to the project. But it’s likely my employment contract prevents me contributing code directly :frowning:, I’ll double check, but I suspect I’m limited to giving pointers on bugs.

You’ve probably already figured it out, but yep. The importer looks for them by filename, to keep the process efficient. That’s also why there’s only one object per file, for TimelineItems and Places.

No worries. Such is life :wink:

Yeah figured that one out. It was the capitalisation that was throwing me off. I was generating UUIDs in lowercase and using that in the JSON and file names. Turns out the import code is parsing the UUID, then converting it back to string for the file lookup, but Swift uses uppercase UUID strings rather than lowercase (which Python does by default), so Arc wasn’t finding my files.

Pretty easy to fix on my part. But would be nice if the file search code could be made case insensitive.

Also one other thought, would be nice if the import code could import from a compressed file. iCloud is really slow at syncing thousands of small files, and the whole downloading stuff on iOS is a bit dodgy. Being able to create a single compressed file with right internal structure would make it much easier to push data over iCloud.

Am I right in saying the current import code doesn’t support the Arc JSON export format? To import Arc JSON, I would need to split the JSON into a LocomotionSample/samples.json file and a bunch of TimelineItem/<UUID>.json files?

Argh. That’d be the most annoying mystery bug to find :joy:

Agree.

Oh it doesn’t?? Wait, yes, the LocomotionSample files are gzipped. Ah, but you mean zipping up a bunch of single object files, like TimelineItem files, into a tar.gz or similar?

Yeah, that’d make sense. iCloud syncing has been one the biggest headaches of testing and debugging the backup system.

Yep :disappointed: To my shame.

I was writing all this new backup/restore system during Covid lockdowns, and got super burnt out on it (along with everything else in this currently very dull world). It was a struggle to even get it shipped in its current state. So even though “support Arc JSON” was literally a core initial requirement, I didn’t get to it before burning out completely.

Theoretically it should be trivial to make it happen. I mean, Arc JSON is literally just a serialised TimelineSegment, that could be decoded straight back into a tree of TimelineObjects. But yeah, need to muster up the energy to actually get it done.

Here a little bit of Python that takes an Arc JSON export and converts it into the separate TimelineItem LocomotionSample and Place files.

Anyone running this will need to create the three output folders (TimelineItem LocomotionSample Place) before they run the script on their data.

import json

with open('<input file>.json') as file:
    data = json.load(file)

places = {}
timelineItem = {}
samples = []

for item in data['timelineItems']:
    if item.get('place'):
        places[item['place']['placeId']] = item['place']

    if item.get('samples'):
        samples += item['samples']
        del item['samples']

    timelineItem[item['itemId']] = item

with open('LocomotionSample/samples.json', 'w') as file:
    json.dump(samples, file, indent=2)

for id, item in timelineItem.items():
    with open(f'TimelineItem/{id}.json', 'w') as file:
        json.dump(item, file)

for id, item in places.items():
    with open(f'Place/{id}.json', 'w') as file:
        json.dump(item, file)
1 Like

That’s brilliant! Thanks so much for this :smile:

Also goes to show how much more concise and efficient Python can be.

I’ve also got one that will take Arc exported GPX files and produce mostly ok JSON for import. I’ll throw that up here once I’ve given it a bit more of a clean.

1 Like

Thanks for the script.

Once before I had temporarily reverted to an older iOS version on another device due to some issues. Because of such, and various circumstances, my current Arc app data diverged from before with years of data. I have that backup and am ready to try a merging of such.

Any updates or precautions? From what I can tell, I can take the older backup with necessary folders, run this script, and then place the transformed files in the Import folder?

Some pretty simple precautions:

  • Make sure your ARC backup is up-to-date. You can force a backup by disabling backups, waiting a little while (there’s a thread somewhere with exact of time you need to wait) then re-enabling backups. That force an immediate foreground backup to happen
  • Makes sure you backup is outside of iCloud, so don’t accidentally overwrite while importing
  • Keep a separate backup of the data you wanna merge. The merging process can get fiddly, you don’t want to accidentally overwrite the data you’re merging during the processing process
  • Make sure you give iCloud enough to sync. It can take a while for everything to sync up, and it easy to make a mess by trying change stuff before iCloud has got all your devices in sync.

Beyond that, the importing works quite well. It can take a little while to complete, and transferring many small files via iCloud is slow. You really need to give it time.

Also the process of downloading the files onto your phone is a little unreliable. Arc asks iOS to download everything, then waits for confirmation. With many small files, iOS frequently forgets to actually download files, so the process gets stuck. Force quitting Arc, and re-starting the import is needed. After Arc requests a file to be downloaded enough times, iOS eventually sorts itself out. Thankfully killing an import and restarting is completely safe, you’ll probably need to do that a couple of time. You can also help this process along by manually downloading the files using the Files app.

Finally make sure that to force a new backup once you’ve got all your data imported and you’ve checked it! You don’t want to do this process more than once, it can take a while for Arc to chug through all the data.

1 Like

All excellent advice @AvianLyric! Thanks!

The grace period is 30 seconds. It waits 30 seconds to see if the user turns backups back on or not. Then if backups are still turned off, everything in the database is marked as not backed up, and it starts again from scratch. Oh and before starting a fresh backup it will rename the existing backup to “Previous Backups [some_random_text]”, to preserve it.

Important advice! Arc does work on copies when doing a managed restore, and doesn’t delete anything if doing manual imports, but you never want to trust your data solely to the brains of a computer - they’re untrustworthy beasts. Always work on a copy, not the original :+1:t3:

iOS 16 has a long overdue new feature in the Files app - you can now long press on a folder (in this case the Backups or Restore folder, for example) and choose “Download Now”. That will queue up and download the entire contents of the folder, and do it seemingly much faster and more reliably than when Arc asks iOS to do the same. So that trick helps to avoid a lot of wasted time, waiting for iOS to pay attention to Arc’s requests to download individual files.

Have you managed to tweak this yet?

Looking for this to bring in a years worth of data that I’ve been waiting to import for years :smiley:

I haven’t, honestly wasn’t sure there was anyone who wanted to use it.

I would say that importing years worth of GPX data might not be a good idea. It takes quite a bit of time to clean up after the import, even for a small amount of data.

If you’re still interested I’ll dig it out and have another look at it. But be prepared, you’ll need some technical chops to figure it all out, and I won’t be able to provide much support.

1 Like

For built-in importing of Arc’s ArcJSON export format, that’s very high on my todos.

I probably won’t get it into the next update. I want to get Dark Mode and a couple of other long overdue smaller features shipped, and ArcJSON import is quite fiddly and time consuming, so it’d hold up the release. But it’s definitely coming.

I haven’t had a chance to get around to it but indeed I have four or five years of old data to import. It seemed like the way with the current script is to run it on each of the monthly JSON exports and then import one at a time?

I can wait if you think sometime you might have a chance to improve it. Or if Matt might get around to import in a few months, perhaps that would be a better time.