Editing data and adding new entries using R/Shiny

[gbragaalves]

Hi Matt!

I’ve built an open source tool for manually constructing and editing Arc timeline data, and I’d love to get your guidance on importing into Arc Editor.

The tool is here: GitHub - gbragaalves/arc_timeline_editor: A collection of R Shiny applications and scripts for processing and importing location data into Arc Timeline app. · GitHub

It’s a Shiny/R web app with a Leaflet map where you can:

• Click waypoints and calculate routes via a local OSRM server (Docker containers for car/foot/bike/bus)
• Snap-to-road messy GPS paths
• Fix metro/subway routes where GPS was poor underground
• Create visits with named places and timestamps
• Import flight tracks from FlightRadar24 KML files (with altitude, speed, heading)
• Load existing weekly sample .json.gz files and edit them — drag markers, ignore bad points, lasso-select, re-snap entire paths
• Edit route waypoints (drag, delete, insert) preserving average speed

With Arc Timeline, this worked great — I’d generate the Import/TimelineItem/{UUID}.json + Import/LocomotionSample/samples.json structure, and Arc would pick them up. I’ve been using it regularly to clean up my data.

Now I’m migrating to Arc Editor. I studied the JSON export format (schema 2.2.0) and rebuilt the exporter to match:

• items/{YYYY-MM}.json — arrays of base+trip/visit objects
• samples/{YYYY-Www}.json.gz — flat LocoKit2 samples (id, latitude, longitude at root, integer movingState/recordingState, classifiedActivityType, etc.)
• metadata.json

I also tried to infer the integer activity type codes by cross-referencing speeds and distances in my exports (e.g. code 6 averaging 700+ km/h → likely airplane, code 2 at ~4 km/h → likely walking, and so on), though I don’t know if these mappings are correct.

The generated data looks structurally identical to what Arc Editor exports, but importing the zip into Arc Editor doesn’t pick up any data.

My questions:

1. Does Arc Editor’s JSON import expect the same structure as its export? Or is there a different import format/mechanism?
2. Are there required fields or validation rules that would silently reject data?
3. Is there a way to see import errors/logs in Arc Editor (maybe in the Settings debug views)?
4. Is the activity type integer mapping documented anywhere?

Happy to share sample files or adapt the tool based on your feedback. The goal is to make it useful to the community — anyone who needs to manually fix or build Arc data.

Thanks!

[matt]

Wow! This is very cool! Love it!

@gbragaalves

Is the activity type integer mapping documented anywhere?

For these, you can see in the LocoKit2 source: LocoKit2/Sources/LocoKit2/ActivityTypes/ActivityType.swift at main · sobri909/LocoKit2 · GitHub

@gbragaalves

Does Arc Editor’s JSON import expect the same structure as its export?

Same. Arc Editor’s automatic backups, manual exports, and imports are all the same JSON schema.

@gbragaalves

Are there required fields or validation rules that would silently reject data?

I hope not silently, but reject yes. LocoKit2’s schema is much more rigidly enforced than old LocoKit’s. There’s SQL triggers and constraints that will reject certain malformed arrangements, so it’s enforced at db level. LocoKit2 is built to keep a strictly clean house, whereas old LocoKit was a lot more laid back (which led to many sleepless nights for me. heh).

Though for all of that you can again inspect the LocoKit2 source on Github. If you’re working with an AI buddy they should be able to quickly surface the various restrictions. Most of them are in the SQL schema and triggers, which are spread across several Database files: LocoKit2/Sources/LocoKit2/Database at main · sobri909/LocoKit2 · GitHub

There’s possibly also some more that are enforced at app code level - not suitable for constraints or triggers. But those wouldn’t get in the way of imports. If you’re curious, you could have your AI buddy poke about in TimelineProcessor and friends.

@gbragaalves

Is there a way to see import errors/logs in Arc Editor (maybe in the Settings debug views)?

Yeah, if there’s an error during import it should land in the debug logs. If something goes weird and the logs don’t tell you what happened, please let me know!

For the import process itself, unfortunately at this stage it’s whole database only. I don’t think the importer is allowing for updating existing data. If it sees the same row ids already in the db it’ll skip rather than update.

Though… you can double check that too. The importer code is… where is it… Ah where we go: LocoKit2/Sources/LocoKit2/ImportExport/ImportManager.swift at main · sobri909/LocoKit2 · GitHub

ImportManager is the JSON importer. It’s currently built to serve full restores from backup, and full restores from manual exports (which are the same thing as the backups). I haven’t skilled it up yet on partial imports, or updating of existing data, both of which bring various complications into the picture.

But I do want the importer to be capable of all those smarter things! So definitely let me know what would be most valuable to you, and I’ll prioritise.

[gbragaalves]

Thanks Matt! Really appreciate the quick response and all the pointers.

Good news first: importing new data into blank days works perfectly now. After your reply I went through the LocoKit2 source with my AI buddy Claude and we sorted out the issues that were blocking my initial imports.

The exportType was the main culprit — I was using "partial", then "backup", both of which caused silent decode failures. From ExportMetadata.swift we found the enum only accepts "full" or "incremental". Switched to "incremental" and things started working. We also found the activity type codes in ActivityType.swift as you pointed out, and fixed a bug where my “Edit Samples” export was only writing samples but no timeline items — ImportManager expects all three directories (items/, samples/, places/).

So the tool now successfully creates visits, trips, and routes from scratch and imports them into Arc Editor. That part is solid.

The problem I’m stuck on now is replacing or correcting existing routes. With Arc Timeline this worked because the import would overwrite existing data.

With Arc Editor, Claude and I went deep into ImportManager.swift and SampleImportProcessor.swift and confirmed what you mentioned: the importer uses INSERT OR IGNORE (GRDB’s .ignore) for everything — places, timeline items (base, visit, trip), and samples. Records with existing IDs are silently skipped.

We tried pretty much everything we could think of to work around it:

First we tried reusing the original timeline item ID and setting deleted: true, hoping Arc would see the flag and remove it. But since the item already exists in the DB, the import record is ignored entirely — the deleted: true never reaches the database.

Then we tried sending samples with deleted: true to mark the originals for removal, which is when we discovered that LocomotionSample in LocoKit2 doesn’t even have a deleted field (unlike old LocoKit). It uses disabled instead. The deleted field just gets silently ignored during JSON decode.

So we tried disabled: true instead — same problem. The samples already exist with the same IDs, so INSERT OR IGNORE skips them and the flag never gets written.

We also tried generating new sample IDs but reusing the original item ID, hoping the item would get updated with a newer lastSaved. Nope — the item is skipped (same ID exists), and the new samples end up orphaned in the DB.

Finally, generating completely new IDs for everything does work, but the corrected route shows up as a new separate item alongside the original. You end up with two overlapping routes and no way to remove the old one via import.

We looked at lastSaved too — it’s used by the ExportManager to decide which buckets to re-export during incremental backups, but the ImportManager doesn’t use it for conflict resolution at all.

So right now the situation is: adding new data to empty time periods works great, but replacing or correcting existing routes isn’t possible through the current import pipeline.

Since you mentioned wanting the importer to handle smarter scenarios, here’s what would be most valuable for my use case. The simplest option would be supporting INSERT OR REPLACE for items that come in with deleted: true — if an imported item has the same ID as an existing one and is marked deleted, just delete the existing one. That alone would let external tools mark bad items for removal. A step further would be upserting based on lastSaved — if the imported record is newer, update it. That’s the most natural behavior for “I edited this outside Arc and want to push the fix back.” And the most complete option would be an explicit edit import mode, maybe a new exportType value like "edit" that switches ImportManager to INSERT OR REPLACE for all record types.

Any of these would unlock the route correction workflow. The first option is probably the least risky since it only touches records explicitly marked for deletion.

The tool and all the investigation details (including the LocoKit2 source code findings) are in the repo README: GitHub - gbragaalves/arc_timeline_editor: A collection of R Shiny applications and scripts for processing and importing location data into Arc Timeline app. · GitHub

Happy to test any changes you make to the importer — I have a solid test setup with weekly backups and can verify edge cases quickly.

[matt]

@gbragaalves

The exportType was the main culprit — I was using "partial", then "backup", both of which caused silent decode failures.

Ah I wonder if the export schema docs are out of date. Probably are. I’ll get daily-archivist to have a look at that! That agent is in charge of keeping docs up to date, but that one’s just outside its scope, being public docs rather than internal. It’s probably been missing it.

@gbragaalves

First we tried reusing the original timeline item ID and setting deleted: true, hoping Arc would see the flag and remove it.

Ah yeah, nothing in LocoKit2 is ever hard deleted, only ever soft deleted. Well that’s not entirely true - sample pruning can hard delete samples, doing path simplification on Trips and reduction of excessive stationary samples in long visits. But other than that, everything’s soft deletes. And samples don’t even soft delete (no use for it, yet).

@gbragaalves

It uses disabled instead.

Disabled is a special one for dealing with overlapping imported data. Like when importing a workout, or importing GPX (once that feature arrives back in the app). If there’s existing items/samples that are overlapped by the importing range then they get marked disabled. That allows them to be preserved, so that if the user decides to scrap the imported data via the UI, the LocoKit2 data is still there to be resurrected and linked back up in the timeline.

@gbragaalves

We also tried generating new sample IDs but reusing the original item ID, hoping the item would get updated with a newer lastSaved.

This is actually my preferred approach, and probably the one I’ll go with once updates are possible in imports. With that, all you’d need to do to update existing data is set a newer lastSaved, and the importer will know to update instead of skipping. This is also what the old app / old LocoKit does.

And if you wanted to delete existing data, you could touch lastSaved and also soft delete. Though if you want to delete a sample, no such luck. But I’d be curious to know what the utility would be for deleting samples?

Anyway I’ll make sure this is all filed and prioritised! It feels chunky enough / risky enough that it might have to slip until v1.0 rather than sneaking into v1.0.1. But I’m not good at sticking to sensible priorities - if something feels fun and appealing I’ll often sneak it in early. And given this one is required for your tool, that makes it pretty high priority already!

@gbragaalves

Happy to test any changes you make to the importer — I have a solid test setup with weekly backups and can verify edge cases quickly.

Excellent Thanks! Hopefully not too long until I can get some test builds to you.

[matt]

Oh random aside: I’m also quite keen to use your tool myself! I really want flight data importing, because flights on 787s get ~zero location data. Which is quite disappointing.

I’ve had flight import as my top personal wishlist feature request for a few years now. I was about to start in on it when I started the whole LocoKit2 and Arc Editor projects (realising that I was building on increasingly creaking old architecture, that made everything harder than it should be due to age).

[gbragaalves]

Great to hear you’re excited about the tool! I understand it’ll take some time to deal with, so I’ll just take the opportunity to import some data I have from trips pre-2024 (Google Maps and Google Photos can provide a pretty decent picture of those times). Looking forward to testing it whenever you have a build ready.

On deleting samples — the main use case is replacing them with geographically correct ones. The most common scenario for me is metro/subway trips. Arc records samples at the boarding and alighting stations (where there’s GPS signal), but everything in between is either missing or wildly inaccurate. What I do in the tool is load those samples, snap the route to the actual metro line via OSRM or Google Maps, and generate new samples along the correct path with interpolated timestamps. So ideally I’d delete (or disable) the original bad samples and import the corrected ones in their place. Right now I can only add the new ones alongside the originals, which creates a mess.

It’s better for me than marking those records as bogus in Arc, because in the tool I can just draw the metro line from station A to station B, match the timestamps from the real samples recorded at each station, and get a clean path. The whole thing takes about 30 seconds.

Regarding the disabled field — really interesting to know that’s the mechanism for preserving overlapped data during imports.

On a tangential note about airplane recordings — I actually had a flight on a 787 where absolutely nothing was recorded, but I was able to recover the full track by importing the FlightRadar24 KML into my tool and pushing it to Arc Timeline. The tool parses the KML placemarks and extracts coordinates, altitude, speed, and heading from each point. On a 777 I’ve found that GPS sometimes works if the phone is pressed right up against the window, basically touching it. And on a 737 MAX 8 I had a flight where the entire track was recorded with great precision — location, altitude, speed — even though I was sitting in an aisle seat. Seems to vary a lot by aircraft type.

Thanks again for the support and for being so open about the codebase. Really looking forward to the updated importer!

[matt]

@gbragaalves

On deleting samples — the main use case is replacing them with geographically correct ones. The most common scenario for me is metro/subway trips. Arc records samples at the boarding and alighting stations (where there’s GPS signal), but everything in between is either missing or wildly inaccurate. What I do in the tool is load those samples, snap the route to the actual metro line via OSRM or Google Maps, and generate new samples along the correct path with interpolated timestamps. So ideally I’d delete (or disable) the original bad samples and import the corrected ones in their place. Right now I can only add the new ones alongside the originals, which creates a mess.

Ah this is a clear case for sample soft delete. Yeah I’m sold on it now. K I’ll make sure that’s filed - will need a separate Linear issue.

@gbragaalves

Regarding the disabled field — really interesting to know that’s the mechanism for preserving overlapped data during imports.

Just thinking aloud - not properly formed ideas yet: I wonder if some of these data fix ups could be treated as imports. I’m not sure if that makes sense though.

But, like, something I frequently do is record walking workouts on my watch, to get absurdly high frequency (1Hz) location data for the walk, with reasonable Kalman filtering (the Watch / HealthKit / whatever appears to use a remarkably similar Kalman tuning to LocoKit2’s, but at higher sampling rate, due to not having Arc’s harsher energy use constraints).

I can then import those walking workouts to replace what LocoKit2 recorded in the occasional case where LocoKit2’s data was weak. The most common case being where recording was too slow to restart from sleep when leaving somewhere. (Though build 49 massively reduces the risk of that, and 50 will go further).

Anyway, that’s a case of using externally generated location data to replace LocoKit2’s, and that’s where the `disabled` system comes into play. Oh you can see the logic for all that in… here.

@gbragaalves

On a tangential note about airplane recordings — I actually had a flight on a 787 where absolutely nothing was recorded, but I was able to recover the full track by importing the FlightRadar24 KML into my tool and pushing it to Arc Timeline.

Yeah 787s are a menace. Whenever I’m staring out one of their fancy fading windows I’m spending the whole time grumbling. “No location data! Insufferable!”

And that experience is also why I’ve had FlightRadar24 import on the todos as high priority for a bunch of years. You beat me to it!

And again, when I build that feature into Arc proper it’ll be using the disabled/enabled system. So I wonder if that also makes the case for using that in your tool. I’m still feeling like I haven’t thought this fully through yet though. So it’s just the germ of an idea at this point.

@gbragaalves

On a 777 I’ve found that GPS sometimes works if the phone is pressed right up against the window, basically touching it. And on a 737 MAX 8 I had a flight where the entire track was recorded with great precision — location, altitude, speed — even though I was sitting in an aisle seat. Seems to vary a lot by aircraft type.

Yeah, a lot of variance. Very wide range of outcomes.

I find A320s get great data if you’re in the window seat and you have your phone anywhere above your legs. Though I’ve also had my test phones in my carry-on get decent recordings while stowed in the overhead on A320s. But that’s been more hit and miss. Best is on the tray table.

Can’t remember how my last A350 flight went. And haven’t flown in an A380 in years I really miss that plane.