Arc Editor Failed LocoKit Import

[DotKern]

The previous build of Arc Editor failed to import anything from LocoKit besides places. It has no actual location history. This new build does successfully import, however it doesn’t actually import anything from beyond three days ago. I presume this is because it thinks it already has, on account of previous attempted imports.

Now this doesn’t bother me tha much, since I’ve only had the app for a week and spare. However, this is a bit of an interesting case, I actually have data spanning back to 2020. This is because via a large amount of python wizardry and batch files, I’ve managed to convert thousands of files from Strava and PolarSteps to the proper format, batch them into pieces under Arc Timelines import cap (about 40 tracks from trial and error, more crash the system).

This very well could be part of the issue, these were very shabbily and jangly converted using dubious coding skills. Arc Timeline does show them, so it was successful to some degree, but it’s possible that formatting errors in them are causing the crashes.

[11:56:11.650] [ERROR] [IMPORTING] Database import failed: SQLite error 19: Sample disabled state must match parent item disabled state - while executing `INSERT OR IGNORE INTO "locomotionSample" ("id", "lastSaved", "date", "secondsFromGMT", "source", "sourceVersion", "movingState", "recordingState", "disabled", "timelineItemId", "latitude", "longitude", "altitude", "horizontalAccuracy", "verticalAccuracy", "speed", "course", "classifiedActivityType", "confirmedActivityType", "stepHz", "xyAcceleration", "zAcceleration", "heartRate", "rtreeId") VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)`

[11:56:11.651] [ERROR] [MISC] SQLite error 19: Sample disabled state must match parent item disabled state - while executing `INSERT OR IGNORE INTO "locomotionSample" ("id", "lastSaved", "date", "secondsFromGMT", "source", "sourceVersion", "movingState", "recordingState", "disabled", "timelineItemId", "latitude", "longitude", "altitude", "horizontalAccuracy", "verticalAccuracy", "speed", "course", "classifiedActivityType", "confirmedActivityType", "stepHz", "xyAcceleration", "zAcceleration", "heartRate", "rtreeId") VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)`

Thank you! Looking forward to your response.

[DotKern]

For those curious what a year abroad looks like as a single import;

Yes it shows my average Rem sleep per night for the whole year. Yes it shows a very dubious heart rate graph. Where did it get this data? Who knows.

[matt]

@DotKern

This new build does successfully import, however it doesn’t actually import anything from beyond three days ago. I presume this is because it thinks it already has, on account of previous attempted imports.

Ah yep, will need to do a fresh install unfortunately! The current importers don’t allow for specifying an import date range. Oh wait! There’s a button lower on the settings/debug view, “Delete all imported data”. Hmm, wait, no, that one’s no good. It’ll delete more than I intended.

K yeah, will need to do a fresh install, then fresh old LocoKit import. Sorry about that!

@DotKern

This is because via a large amount of python wizardry and batch files, I’ve managed to convert thousands of files from Strava and PolarSteps to the proper format, batch them into pieces under Arc Timelines import cap (about 40 tracks from trial and error, more crash the system).

Nice work! And yeah the GPX importer in old Arc Timeline doesn’t cope well with large imports. Once I get to building the equivalent feature in Arc Editor I’ll make sure it copes better with large ones.

@DotKern

Arc Timeline does show them, so it was successful to some degree, but it’s possible that formatting errors in them are causing the crashes.

Ah those errors about disabled state are from the previous TestFlight build. You shouldn’t ever see them in the most recent TestFlight build, from a day or two ago. If you’re still seeing them I missed something somewhere!

So yeah, definitely do a fresh Arc Editor install then fresh old LocoKit import. As far as I’m aware, that should be 100% error free now. If there’s any errors in that flow from fresh install there’s something I missed and didn’t realise!

[matt]

@DotKern

Yes it shows my average Rem sleep per night for the whole year. Yes it shows a very dubious heart rate graph. Where did it get this data? Who knows.

I think there’s some weird race condition with old Arc Timeline’s heart rate data fetching sometimes. Like, it fetches it for the item’s date range, but then… actually I’m really not sure. It’s odd. But yeah when I’ve seen similarly weird heart rate graphs like that they’ve always disappeared on their own shortly after, when Arc updates the fetched heart rate data.

Though for a 6 month long item, that might be a big ask! Arc isn’t really built to cope with items that long

[DotKern]

No worries, I’m reinstalling it now. Will report back.

I wasn’t exactly disappointed in it for failing to get accurate data across the six months. I didn’t expect anything to come from it, it was a fun experiment though, and a good refresher on the gpskit python library. I’m curious how it got six months out of ten months. Maybe thats the maximum? It displays it in an interesting way, where it’s the same item copied across every day. Not really an actually useful tool, but also not something that someone would ever really need per se. Would have just been cool for seeing where I was on a particular day.

I almost wonder if I should take advantage of being new to the app and NOT import from LocoKit. Would the random data I imported have any negative effect? Is there anything good that might come from it? I was hoping it would help the computer learn how I move with some already pre verified (kind of) data.

It does look like it’s importing great now. Chugging along.

[DotKern]

@DotKern

I almost wonder if I should take advantage of being new to the app and NOT import from LocoKit. Would the random data I imported have any negative effect? Is there anything good that might come from it? I was hoping it would help the computer learn how I move with some already pre verified (kind of) data

Kind of answered my own question by reading this very useful thread. Does it still do very little processing on imports?

[matt]

@DotKern

I almost wonder if I should take advantage of being new to the app and NOT import from LocoKit. Would the random data I imported have any negative effect?

No significant negative effect. The activity classifiers will adapt to the old LocoKit and new LocoKit2 data being subtly different shapes (different underlying processing/filtering algorithms). But the ML of the classifiers ultimately won’t care about the difference in any noticeable way.

Though what I might personally want to do is to do those GPX imports fresh, but incrementally split up each one before doing the next import. So that instead of getting stuck with a mega 6 month item, you get it split up into usable timeline items.

But that said, Arc Editor doesn’t yet have GPX import. So… that’s not really an option yet. Damn.

[matt]

@DotKern

Kind of answered my own question by reading this very useful thread. Does it still do very little processing on imports?

I think so yeah. Well, it’ll merge and edge heal / edge steal imported GPX data as normal. But the activity type classifiers I think can do next to nothing with it, because it’s missing too much metadata (accelerometer, pedometer, maybe some others depending on import source). So the activity type classifiers can’t really get a sensible read on the imported data I think.

I’m just in the process of answering those kinds of questions more explicitly, while building the workout importer in Arc Editor. This time around I’m going to try to button down all the edge cases in the importers and imported data, so that things are much more “policy” instead of informal and organic.

Old Arc Timeline has too many cases where things happen a certain way because it didn’t cause any major problems and I didn’t have time to be more intentional about it. This time around I’d like to have much less of that!

[DotKern]

I like this, and I’m a big fan of how you’ve explained it too. One of the most important things about apps is that when they can’t do something the user understands why. It’s important for rules to make sense. Especially if apps do nine things one way and one thing the other… it better be very clear why that 10% case is special. It better be logical.

Thats one of the issues with algorithmic apps or “AI” apps (para-AI I guess?). They’re black boxes, even the developers don’t know why the system does some of the things it does, so figuring out why and ironing out those edge cases is so important. You seem to be doing a great job of that, and the approach of rebuilding a new app from (kind of) the ground up isn’t a bad idea at all, it lets you learn from previous mid-steps and mishandlings.

[matt]

@DotKern

One of the most important things about apps is that when they can’t do something the user understands why. It’s important for rules to make sense. Especially if apps do nine things one way and one thing the other… it better be very clear why that 10% case is special. It better be logical.

And this is a big reason why Arc Editor and LocoKit2 had to happen. With old LocoKit and old Arc Timeline, being quite old apps and libs by now, it’d got to the point where too many things were just old and mysterious and unexplainable. “It does this weird thing, and I’ve never been able to figure out in the code / archeological record quite why that’s happening”. That’s not a good situation! But over the years it became more and more common.

There’s a lot of old apps on the App Store now that never get better, because the teams working on them don’t understand how half the old code works, and fixing it up to more modern and clean approaches would be too big a job. Arc Editor is that drawing of the line, saying “Actually, we can’t continue like that. This has to be done all again, and done clean, modern, proper. No more archaeological mysteries.”

@DotKern

Thats one of the issues with algorithmic apps or “AI” apps (para-AI I guess?). They’re black boxes, even the developers don’t know why the system does some of the things it does, so figuring out why and ironing out those edge cases is so important.

True. There’s that added layer of ML / Machine Learning, and nascent AI involved. Which as an aside, I want to go deeper into at some point. Since iOS 26 we’ve got mini LLMs on our phones, that are easy to integrate into apps and give more “human intelligence” jobs to. I want to eventually find some ways to work that local LLM/brain into LocoKit2’s timeline processing, classification, who knows what else.

But that’ll be a fun project for another day. I don’t have any clearly formed ideas there yet, and no urgent need for it. It’s just one of those things that’s now possible and seems really cool and having a lot of potential.

Like, the current ML and timeline processing rules can hit a wall. They get only so far, then there’s still a bit of mess left for us humans to clean up ourselves. What if the LLM/AI on the phone could take it one or two steps further, applying a little bit of its own human-like intelligence, to shrink even more of that cleanup job left over for us.

[DotKern]

@matt

But over the years it became more and more common.

Yeah especially when you have lots of dependencies on various libraries and other projects. Each with their own scope creep and interfacing with hardware and APIs in their own ways. Lots of things that can go wrong in strange and inexplicable ways.

@matt

There’s a lot of old apps on the App Store now that never get better, because the teams working on them don’t understand how half the old code works, and fixing it up to more modern and clean approaches would be too big a job.

Which is too bad, because they really could be improved quite easily if only the code and mechanics behind it weren’t spaghetti. This is one of the rare AI uses I’m totally fine with, making sense of and simplifying or refining code. Hopefully it lets people revive some creaky old projects.

@matt

I want to eventually find some ways to work that local LLM/brain into LocoKit2’s timeline processing, classification, who knows what else.

This is one of the benefits of bring a solely Apple based app. You can fully lean into the ecosystems perks. ‎Having a deep understanding of iOS location tracking is one benefit, the health integration is one, and this will probably be another one. The inbuilt “journal” system would be a great place to start. I assume there is some sort of API for it, because the stoic. Journal suggestions and the iOS journal suggestions are the same. They’re based on various parts of your day and photos, so they’re already locked in time. If you ever do decide to lean fully into the journaling aspect, this would be a good second step to do so (after adding markdown support).

Another idea is having some sort of LLM or even AI with internet access figure out what a location is and what you’re doing at a location. Swarm knows that The Home Depot is a hardware store, an AI could tell that if I’m going there every day… I’m probably not shopping there. This gets you 80% of the way to automatically updating Life Balance with location information, which is essentially the whole functionality of Life Cycle, in addition to some pretty graphics.

I should really submit some feature requests at some point.

[matt]

@DotKern

The inbuilt “journal” system would be a great place to start. I assume there is some sort of API for it

Unfortunately it’s almost entirely useless. Or at least it was in iOS 18, and there was nothing about improvements to it on the WWDC sessions for iOS 26.

It’s only in one direction: apps can suggest journal entries to the Journal app, but nothing in the other direction at all. Arc can’t show journal entries, can’t read or import them, can’t learn anything from your Journal at all.

@DotKern

Another idea is having some sort of LLM or even AI with internet access figure out what a location is and what you’re doing at a location.

Yeah this is the kind of “human intelligence” task that I’d like to experiment with. There’s certain patterns that look obviously wrong to a human, but to the current ML algorithms and timeline processing heuristics are entirely value neutral. But an LLM on the phone could look at it and say “Well clearly they didn’t walk in and out of the convenience store every few seconds for ten minutes. That must be drifting location data. And the paths out of the store are all already classified as stationary. I should just convert them to Visits to clean up that obvious mess.”

Another case might be something like “Ah so they walked to a place, stopped for 5 minutes, then took a train ride. The Visit has been assigned a pharmacy Place, but obviously it’s meant to be the train station, that’s second in the results list. I’ll change it to that.”

Though in that second case I’m actually going to deal with that in heuristics / old school ML, by integrating in “previous place” and “previous activity” model features for the activity type and place classifiers. That one I can deal with the old school way. But that kind of thing, where human intelligence can spot things that old school ML can’t… that’s where the on device LLM could come in handy!

@DotKern

I should really submit some feature requests at some point.

Hah. For sure! Always good to sift through the Changemap site and vote on all the things you like, and submit any missing. I do use it for deciding next major direction in feature development. Though at the moment I’m still very much in “get Arc Editor back to feature parity with old Arc Timeline”. But when it comes to new features I’ll often skim the Changemap site to refresh my memory on the most popular ideas. I’ll often spot things that’ve been sitting there for years with heaps of votes but that had completely slipped my mind.

[DotKern]

@matt

Arc can’t show journal entries, can’t read or import them, can’t learn anything from your Journal at all.

I meant more in the regard of grabbing just the prompts. I assume there is soooome way of doing it. I’m just not certain how round about it must be.

For context here is what I mean in the Stoic. app. Since all of there are timestamped and dated they would fit really nicely into the timeline. Some are based on songs (which we already have), some on workouts (which we already have). So just letting the user attach notes to things other than segments is the main functionality if you pick it apart. The only thing we don’t have is the cute (probably AI generated) journal prompts. But thats not as important for Arc, and wouldn’t be hard to add if you decided it was.

@matt

that’s where the on device LLM could come in handy!

I’m definitely looking forward to seeing where you go with this! Lots of freedom and lots of possibilities.

@matt

I’ll often spot things that’ve been sitting there for years with heaps of votes but that had completely slipped my mind.

Well that isn’t ‎‎very reassuring! Although you seem to be barreling away pretty consistently at development so I guess it’s okay.

[matt]

@DotKern

I meant more in the regard of grabbing just the prompts.

Hmm. Just looking into the API for this now… it does look like there’s a "Journaling Suggestions” view that Arc could open up, that then allows you to select a suggestion, which will be provided to Arc.

So it wouldn’t be anything automatic, that Arc can implicitly import and make use of. But it could be a manual thing, where you get the suggestions view and can pick something from it, that will then be provided to Arc.

That’s actually better than I remembered. Though still not great. Still limited to a manual process and UI. But… it does have some feature potential. I shall ponder…

[matt]

Looking at the suggestions on my phone, it’s only showing suggestions based on workouts, photo memories (“1 Nov, Over the Years”), or place + photos suggestions, which is what Arc already does automatically (annotating Visits with the photos taken during that Visit).

I’m not seeing any music suggestions yet either. Even though I listen to a whole lot of music every day through Apple Music. Curious.

I’m also seeing standard writing prompt suggestions, “Describe something you’ve learnt about yourself”, that sort of thing. Not sure those add any value either.

Feels like this has potential, but I’m not seeing any suggestions that … realise that potential. Yet.

[DotKern]

Yeah now that I’ve looked over it, Timeline essentially has all of the functionality already without the integration. It just needs to let us add notes on other pieces of information and we’ve got a fully functioning chronological journalling app.

[matt]

Yeah the Notes system in both Arc Timeline and Arc Editor is still very basic. Just notes on items, and that’s it. There’s a lot of potential there for much cooler things.

[agastya]

matt,

I have no opinion about notes yet about suggestions in general, eg someone asking about stand time.

Perhaps my impression: sure as the app in a sense is a journal of all movement, visits, etc. maybe such offers unique benefits. Yet is it perhaps better to focus, think, and meditate on what might be most commonly wanted if such perhaps might broaden appeal or even make use of such addresses, place names, travel log, etc.

Not that I use such but maybe : auto mileage calculation, place list as a sort of dynamic address book possibly integrating Yelp or not, ease of search like searching a address book and viewing details perhaps automatically filled, place details maybe reorganized for someone that makes use of such, if an addrsss book of sorts maybe marking places as favorites to easily lookup to get directions, and I don’t know what else.

[matt]

@agastya A lot of fun ideas there! I recommend filing your favourites in the Changemap site. Otherwise we’ll end up chatting about them here then they’ll get forgotten and never implemented. On the Changemap site they can’t be forgotten, and can get votes from other people who want the same things.

[agastya]

i don’t know. Maybe I have some ideal of what the app might be but I don’t think about it much. As it’s your app perhaps it’s up to you to mediate on what is termed location visit travel movement log journal diary history time memory recollection etc and then imagine.