# JSON Import Format **URL:** \ **Category:** Features Discussion\ **Created:** [9 January 2022 18:10 UTC](https://support.bigpaua.com/t/json-import-format/219 "2022-01-09T18:10:43Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![AvianLyric](https://avatars.discourse-cdn.com/v4/letter/a/b487fb/32.png) [@AvianLyric](https://support.bigpaua.com/u/AvianLyric)\ **Post date:** [9 January 2022 18:10 UTC](https://support.bigpaua.com/t/json-import-format/219/1 "2022-01-09T18:10:43Z")
Is the ARC JSON format documented anywhere? I’m wondering if it’ll be possible to build external transformers that can take other data formats and convert them in the ARC format. Then it would be possible for the community to build its own transformers for data sources like Google Timeline or GPX files. ---
**Author:** ![matt](https://dub1.discourse-cdn.com/flex013/user_avatar/support.bigpaua.com/matt/32/3_2.png) [@matt](https://support.bigpaua.com/u/matt)\ **Post date:** [10 January 2022 06:03 UTC](https://support.bigpaua.com/t/json-import-format/219/2 "2022-01-10T06:03:39Z")
That would be great! The format isn’t formally documented anywhere, but the code behind it is available on Github in the [LocoKit repo](https://github.com/sobri909/LocoKit/tree/master/LocoKit/Timelines/TimelineObjects), in the various TimelineObject classes (samples and timeline items). I can also answer any questions anyone has here! ---
**Author:** ![AvianLyric](https://avatars.discourse-cdn.com/v4/letter/a/b487fb/32.png) [@AvianLyric](https://support.bigpaua.com/u/AvianLyric)\ **Post date:** [12 January 2022 23:03 UTC](https://support.bigpaua.com/t/json-import-format/219/3 "2022-01-12T23:03:49Z")
Reading through the code, am I right in saying you would need to generate locomotion samples to create a travel path? The timeline objects seem simple enough, but I’m not sure the best way to create locomotion samples, as most data won’t include 90% of the info normally in a locomotion sample. Additionally would creating sparse locomotion sample throw of the ML training? I suspect data taken from 3rd party sources wouldn’t be good enough for Arc’s ML training process. [EDIT] looking through ArcMini it seems that only samples and timeline entries with the source `LocoKit` are used to update models, everything else is ignored. So I assume this means I can create sparse samples with a made up source, and they won’t end up polluting Arc’s training process? ---
**Author:** ![matt](https://dub1.discourse-cdn.com/flex013/user_avatar/support.bigpaua.com/matt/32/3_2.png) [@matt](https://support.bigpaua.com/u/matt)\ **Post date:** [13 January 2022 10:39 UTC](https://support.bigpaua.com/t/json-import-format/219/4 "2022-01-13T10:39:53Z")
> [@AvianLyric](#): > > Reading through the code, am I right in saying you would need to generate locomotion samples to create a travel path? Yep. LocomotionSamples contain the lat/long coordinates of samples recorded every 2 to 60 seconds. > [@AvianLyric](#): > > Additionally would creating sparse locomotion sample throw of the ML training? I suspect data taken from 3rd party sources wouldn’t be good enough for Arc’s ML training process. Yeah, the ML will largely skip over those samples, judging them not suitable for learning from. It might be able to classify them though, purely based on coordinates, but the results will be less than ideal. If the original data has known activity types, then you’d want to put that type in the `confirmedType` field for each sample, so that the classifier doesn’t bother trying to make sense of them. ---
**Author:** ![AvianLyric](https://avatars.discourse-cdn.com/v4/letter/a/b487fb/32.png) [@AvianLyric](https://support.bigpaua.com/u/AvianLyric)\ **Post date:** [14 January 2022 01:36 UTC](https://support.bigpaua.com/t/json-import-format/219/5 "2022-01-14T01:36:56Z")
I’ve managed to successfully massage a single GPX file into ArcJSON and import it. Of the back of this I’ve got a couple of questions: 1. Can you add the ability to set the source in imports? At the moment everything gets imported with the source `LocoKit`, which means that Arc will try to use the data for training the ML stuff. Ideally it wouldn’t do this, because the data quality is so low. 2. Is there a way to import timeline items with no next or prev timeline ID and have Arc figure that out? At the moment it’s difficult to attach new timelines items to the existing timeline in Arc. 3. Can you expose a more basic `location` attribute for imports that isn’t based on CLLocation object? I don’t have enough data for course, or lat/lon accuracy. It would be nice to omit this values instead of adding filler values. ---
**Author:** ![matt](https://dub1.discourse-cdn.com/flex013/user_avatar/support.bigpaua.com/matt/32/3_2.png) [@matt](https://support.bigpaua.com/u/matt)\ **Post date:** [14 January 2022 09:30 UTC](https://support.bigpaua.com/t/json-import-format/219/6 "2022-01-14T09:30:53Z")
> [@AvianLyric](#): > > Can you add the ability to set the source in imports? This is something that can be / probably should be done in the Arc Mini code on Github. Though I don’t think the new file import system is in the Arc Mini code yet. I’ll get on that. > [@AvianLyric](#): > > Is there a way to import timeline items with no next or prev timeline ID and have Arc figure that out? Yeah, it’s actually better to not give those fields a value at all. I have a hunch the importer actually always ignores them anyway, because the database will reject incorrect / impossible linked lists. So it imports them with no next/previous links, then lazily rebuilds those links based on nearest neighbour, after import. > [@AvianLyric](#): > > Can you expose a more basic `location` attribute for imports that isn’t based on CLLocation object? I don’t have enough data for course, or lat/lon accuracy. It would be nice to omit this values instead of adding filler values. Hmm. I thought it’d happily accept it as long as lat/long exists, but looks like [I’ve remembered wrong](https://github.com/sobri909/LocoKit/blob/develop/LocoKit/Base/LocomotionSample.swift#L281-L285). Could modify that so that, if full CodableLocation fails, it falls back to just using lat/long to initialise a basic CLLocation. ---
**Author:** ![AvianLyric](https://avatars.discourse-cdn.com/v4/letter/a/b487fb/32.png) [@AvianLyric](https://support.bigpaua.com/u/AvianLyric)\ **Post date:** [16 January 2022 22:22 UTC](https://support.bigpaua.com/t/json-import-format/219/7 "2022-01-16T22:22:25Z")
> [@matt](#): > > This is something that can be / probably should be done in the Arc Mini code on Github. Though I don’t think the new file import system is in the Arc Mini code yet. I’ll get on that. This would be amazing, would help me understand the import process and how to create valid import data. If the way in the current Arc import process to bulk import timeline items? I can’t seem to import a `timelineItems.json` file with multiple timeline items in it, even if I used the export schema, and there’s no “Import All” button for timeline items in the UI, forcing me to tap each item manually, which get tedious pretty fast. > [@matt](#): > > Yeah, it’s actually better to not give those fields a value at all. I have a hunch the importer actually always ignores them anyway, because the database will reject incorrect / impossible linked lists. So it imports them with no next/previous links, then lazily rebuilds those links based on nearest neighbour, after import. I’ve been struggling with this quite a bit. The only way I’ve managed to get imported data to appear in Arc is to create a valid linked timeline series, import it, and then import an update for an existing timeline item to set its previous timeline item to the imported data. I’m not sure if Arc is relinking in the background or not, there’s no UI to show the progress, it just looks like the import failed. It would be nice if the import UI showed the progress on the re-linking. Looking through Arc Mini, it looks like timelines items should be relinked if they’re broken when you look at the correct day. But I’ve failed to induce this behaviour, additionally the data I want to import is from years back, so tap “back” over 600 times doesn’t really work for me. > [@matt](#): > > Hmm. I thought it’d happily accept it as long as lat/long exists, but looks like [I’ve remembered wrong](https://github.com/sobri909/LocoKit/blob/develop/LocoKit/Base/LocomotionSample.swift#L281-L285). Could modify that so that, if full CodableLocation fails, it falls back to just using lat/long to initialise a basic CLLocation. That would be super nice. Would also be cool is Locomotion samples could be marked as `dirty` or something, then have those sample excluded from [ML updates](https://github.com/sobri909/ArcMini/blob/572b54920b6caa304ff07c81bf1cca14a2ac233f/Arc%20Mini/Models/ActivityType/UserActivityType.swift#L61-L65). ---
**Author:** ![matt](https://dub1.discourse-cdn.com/flex013/user_avatar/support.bigpaua.com/matt/32/3_2.png) [@matt](https://support.bigpaua.com/u/matt)\ **Post date:** [17 January 2022 04:07 UTC](https://support.bigpaua.com/t/json-import-format/219/8 "2022-01-17T04:07:34Z")
> [@AvianLyric](#): > > If the way in the current Arc import process to bulk import timeline items? I can’t seem to import a `timelineItems.json` file with multiple timeline items in it, even if I used the export schema, and there’s no “Import All” button for timeline items in the UI, forcing me to tap each item manually, which get tedious pretty fast. The trick is to import the LocomotionSamples first, and the importer will find the relevant TimelineItem files itself. When importing a sample, it looks at the timelineItemId, checks the db to see if it has that item, and if not, looks for a matching JSON file to import along with the sample(s). Likewise Places - if a TimelineItem file references a placeId, it’ll check the db for existing, then check for a matching JSON file. So importing a LocomotionSamples file will also trigger the import of all referenced TimelineItem and Place files. > [@AvianLyric](#): > > The only way I’ve managed to get imported data to appear in Arc is to create a valid linked timeline series, import it, and then import an update for an existing timeline item to set its previous timeline item to the imported data. Hm, that sounds weird. I wonder if this due to importing TimelineItem files before samples? TimelineItems with no associated samples won’t show up in the UI, due to being empty / zero duration. (They’ll also get auto deleted later, during some cleanup process, if they still have no associated samples). Starting the import from LocomotionSamples file first might get around that weirdness. ---
**Author:** ![matt](https://dub1.discourse-cdn.com/flex013/user_avatar/support.bigpaua.com/matt/32/3_2.png) [@matt](https://support.bigpaua.com/u/matt)\ **Post date:** [17 January 2022 04:28 UTC](https://support.bigpaua.com/t/json-import-format/219/9 "2022-01-17T04:28:14Z")
Ok so I’ve dumped into Arc Mini all the importer and related views from Arc App. > **[ArcMini/Arc Mini/Import at master · sobri909/ArcMini](https://github.com/sobri909/ArcMini/tree/master/Arc%20Mini/Import)** > > master/Arc%20Mini/Import The code is literally just dumped in without edit at this stage, so it won’t compile, because it references a bunch of Arc App specific little helpers and whatnots. (I’ve also removed them from the project target, to avoid them breaking the build). `ArcImporter` is the brains. It’s just an ugly “manager” dumping ground of all file importing functions. Not exactly elegant architecture, but it got the job done 😂 `ImportList` is the manual importer view that you’ve been using. `RestoreView` is the managed importer that’s available from fresh installs, and is literally just a stripped down version of the manual importer, that “pushes all the buttons” itself. So that one’s the least useful to sift through. `ImportTask` is the wrapper / state machine around each file being imported, that `ArcImporter` uses, and `ImportList` uses for knowing what to show in the UI. `ErrorLogView` is an unfinished/unused all-in-one view for showing all errors from a managed restore (ie `RestoreView`). It’s not used by the manual importer, which already lets you see errors individually per import task. ---
**Author:** ![matt](https://dub1.discourse-cdn.com/flex013/user_avatar/support.bigpaua.com/matt/32/3_2.png) [@matt](https://support.bigpaua.com/u/matt)\ **Post date:** [17 January 2022 04:32 UTC](https://support.bigpaua.com/t/json-import-format/219/10 "2022-01-17T04:32:46Z")
When it comes time to hack this stuff up to deal gracefully with different file formats, I think that big ugly single file ArcImporter will need to be split up. It’s all working on the assumption it can just take a JSON file and directly decode it to the appropriate TimelineObject type (array of LocomotionSamples, a TimelineItem, Place, Note, etc). Which won’t be possible with other file formats - they’ll need an intermediary layer. Which I do actually kinda have, in the very old Moves App data importer. But that was hacked up in a panic in a single weekend, after Facebook announced out of the blue that they were closing Moves down and my inbox filled up with 10,000 emails. It’s elegant work for a panicked weekend, but it’s not elegant enough to be reusable. Actually I should still drop it into the Arc Mini repo. I’ll have a sift over it now, and see what it looks like… ---
**Author:** ![AvianLyric](https://avatars.discourse-cdn.com/v4/letter/a/b487fb/32.png) [@AvianLyric](https://support.bigpaua.com/u/AvianLyric)\ **Post date:** [17 January 2022 09:27 UTC](https://support.bigpaua.com/t/json-import-format/219/11 "2022-01-17T09:27:01Z")
> [@matt](#): > > The trick is to import the LocomotionSamples first, and the importer will find the relevant TimelineItem files itself. When importing a sample, it looks at the timelineItemId, checks the db to see if it has that item, and if not, looks for a matching JSON file to import along with the sample(s). Ahhhh! That’s what in was missing. I assume I need to name the timeline item files correctly? (Don’t answer that, I’ll dig through the import code!). > [@matt](#): > > Hm, that sounds weird. I wonder if this due to importing TimelineItem files before samples? TimelineItems with no associated samples won’t show up in the UI, due to being empty / zero duration. I’ll need to do a little more investigation on this. My description of the behaviour isn’t quite accurate. But without the import code it’s difficult to figure out why imports were failing. I’ll compare what I’m generating to what’s expected and see if I can figure out what’s breaking. > [@matt](#): > > The code is literally just dumped in without edit at this stage, so it won’t compile, because it references a bunch of Arc App specific little helpers and whatnots. Oh perfect, this is gonna be so helpful! > [@matt](#): > > When it comes time to hack this stuff up to deal gracefully with different file formats, I think that big ugly single file ArcImporter will need to be split up. Just as a heads up, I would love to dive into this and contribute back to the project. But it’s likely my employment contract prevents me contributing code directly ☹, I’ll double check, but I suspect I’m limited to giving pointers on bugs. ---
**Author:** ![matt](https://dub1.discourse-cdn.com/flex013/user_avatar/support.bigpaua.com/matt/32/3_2.png) [@matt](https://support.bigpaua.com/u/matt)\ **Post date:** [18 January 2022 06:49 UTC](https://support.bigpaua.com/t/json-import-format/219/12 "2022-01-18T06:49:09Z")
> [@AvianLyric](#): > > Ahhhh! That’s what in was missing. I assume I need to name the timeline item files correctly? (Don’t answer that, I’ll dig through the import code!). You’ve probably already figured it out, but yep. The importer looks for them by filename, to keep the process efficient. That’s also why there’s only one object per file, for TimelineItems and Places. > [@AvianLyric](#): > > Just as a heads up, I would love to dive into this and contribute back to the project. No worries. Such is life 😉 ---
**Author:** ![AvianLyric](https://avatars.discourse-cdn.com/v4/letter/a/b487fb/32.png) [@AvianLyric](https://support.bigpaua.com/u/AvianLyric)\ **Post date:** [18 January 2022 12:04 UTC](https://support.bigpaua.com/t/json-import-format/219/13 "2022-01-18T12:04:56Z")
> [@matt](#): > > You’ve probably already figured it out, but yep. The importer looks for them by filename, to keep the process efficient. That’s also why there’s only one object per file, for TimelineItems and Places. Yeah figured that one out. It was the capitalisation that was throwing me off. I was generating UUIDs in lowercase and using that in the JSON and file names. Turns out the import code is parsing the UUID, then converting it back to string for the file lookup, but Swift uses uppercase UUID strings rather than lowercase (which Python does by default), so Arc wasn’t finding my files. Pretty easy to fix on my part. But would be nice if the file search code could be made case insensitive. Also one other thought, would be nice if the import code could import from a compressed file. iCloud is really slow at syncing thousands of small files, and the whole downloading stuff on iOS is a bit dodgy. Being able to create a single compressed file with right internal structure would make it much easier to push data over iCloud. ---
**Author:** ![AvianLyric](https://avatars.discourse-cdn.com/v4/letter/a/b487fb/32.png) [@AvianLyric](https://support.bigpaua.com/u/AvianLyric)\ **Post date:** [18 January 2022 18:17 UTC](https://support.bigpaua.com/t/json-import-format/219/14 "2022-01-18T18:17:51Z")
Am I right in saying the current import code doesn’t support the Arc JSON export format? To import Arc JSON, I would need to split the JSON into a `LocomotionSample/samples.json` file and a bunch of `TimelineItem/.json` files? ---
**Author:** ![matt](https://dub1.discourse-cdn.com/flex013/user_avatar/support.bigpaua.com/matt/32/3_2.png) [@matt](https://support.bigpaua.com/u/matt)\ **Post date:** [19 January 2022 07:30 UTC](https://support.bigpaua.com/t/json-import-format/219/15 "2022-01-19T07:30:29Z")
> [@AvianLyric](#): > > Turns out the import code is parsing the UUID, then converting it back to string for the file lookup, but Swift uses uppercase UUID strings rather than lowercase (which Python does by default), so Arc wasn’t finding my files. Argh. That’d be the most annoying mystery bug to find 😂 > [@AvianLyric](#): > > But would be nice if the file search code could be made case insensitive. Agree. > [@AvianLyric](#): > > Also one other thought, would be nice if the import code could import from a compressed file. Oh it doesn’t?? Wait, yes, the LocomotionSample files are gzipped. Ah, but you mean zipping up a bunch of single object files, like TimelineItem files, into a tar.gz or similar? Yeah, that’d make sense. iCloud syncing has been one the biggest headaches of testing and debugging the backup system. ---
**Author:** ![matt](https://dub1.discourse-cdn.com/flex013/user_avatar/support.bigpaua.com/matt/32/3_2.png) [@matt](https://support.bigpaua.com/u/matt)\ **Post date:** [19 January 2022 07:34 UTC](https://support.bigpaua.com/t/json-import-format/219/16 "2022-01-19T07:34:20Z")
> [@AvianLyric](#): > > Am I right in saying the current import code doesn’t support the Arc JSON export format? Yep 😞 To my shame. I was writing all this new backup/restore system during Covid lockdowns, and got super burnt out on it (along with everything else in this currently very dull world). It was a struggle to even get it shipped in its current state. So even though “support Arc JSON” was literally a core initial requirement, I didn’t get to it before burning out completely. Theoretically it should be trivial to make it happen. I mean, Arc JSON is literally just a serialised TimelineSegment, that could be decoded straight back into a tree of TimelineObjects. But yeah, need to muster up the energy to actually get it done. ---
**Author:** ![AvianLyric](https://avatars.discourse-cdn.com/v4/letter/a/b487fb/32.png) [@AvianLyric](https://support.bigpaua.com/u/AvianLyric)\ **Post date:** [20 January 2022 22:55 UTC](https://support.bigpaua.com/t/json-import-format/219/17 "2022-01-20T22:55:47Z")
Here a little bit of Python that takes an Arc JSON export and converts it into the separate `TimelineItem` `LocomotionSample` and `Place` files. Anyone running this will need to create the three output folders (`TimelineItem` `LocomotionSample` `Place`) before they run the script on their data. ```python import json with open('.json') as file: data = json.load(file) places = {} timelineItem = {} samples = [] for item in data['timelineItems']: if item.get('place'): places[item['place']['placeId']] = item['place'] if item.get('samples'): samples += item['samples'] del item['samples'] timelineItem[item['itemId']] = item with open('LocomotionSample/samples.json', 'w') as file: json.dump(samples, file, indent=2) for id, item in timelineItem.items(): with open(f'TimelineItem/{id}.json', 'w') as file: json.dump(item, file) for id, item in places.items(): with open(f'Place/{id}.json', 'w') as file: json.dump(item, file) ``` ---
**Author:** ![matt](https://dub1.discourse-cdn.com/flex013/user_avatar/support.bigpaua.com/matt/32/3_2.png) [@matt](https://support.bigpaua.com/u/matt)\ **Post date:** [21 January 2022 09:08 UTC](https://support.bigpaua.com/t/json-import-format/219/18 "2022-01-21T09:08:46Z")
That’s brilliant! Thanks so much for this 😄 Also goes to show how much more concise and efficient Python can be. ---
**Author:** ![AvianLyric](https://avatars.discourse-cdn.com/v4/letter/a/b487fb/32.png) [@AvianLyric](https://support.bigpaua.com/u/AvianLyric)\ **Post date:** [21 January 2022 11:02 UTC](https://support.bigpaua.com/t/json-import-format/219/19 "2022-01-21T11:02:12Z")
I’ve also got one that will take Arc exported GPX files and produce mostly ok JSON for import. I’ll throw that up here once I’ve given it a bit more of a clean. ---
**Author:** ![agastya](https://dub1.discourse-cdn.com/flex013/user_avatar/support.bigpaua.com/agastya/32/832_2.png) [@agastya](https://support.bigpaua.com/u/agastya)\ **Post date:** [3 October 2022 01:48 UTC](https://support.bigpaua.com/t/json-import-format/219/20 "2022-10-03T01:48:05Z")
Thanks for the script. Once before I had temporarily reverted to an older iOS version on another device due to some issues. Because of such, and various circumstances, my current Arc app data diverged from before with years of data. I have that backup and am ready to try a merging of such. Any updates or precautions? From what I can tell, I can take the older backup with necessary folders, run this script, and then place the transformed files in the Import folder? [Next page](https://support.bigpaua.com/t/json-import-format/219.md?page=2)