Best Workflow to fill Arc data gaps before moving to Arc 4?

[dagguet]

Hi, I have been using Arc for a couple of years now and I also got the new Arc 4 app.

First I realised, that Arc must have stopped recording when the new app or the last Update was released and I have a gap of 12 days I really would wish it would notify me (more throughly??) when this happens. Maybe this could be a feature request for the future, as this is something that frustrates me a lot.

And it’s also the reason why I’m writing today: I have a lot of data gaps in my Arc timeline, especially 2019-2020-2021, I remember the App sometimes turned itself off, without any notifications. Sometimes for days, sometimes even weeks. So when I updated to Arc 4, I saw there is no import feature. So now I’m wondering, if you can recommend a workflow, how to fill gaps in my timeline?

Also, the lastfm feature never seemed to have worked for me. I never really pushed to fix this, but I wonder if I should try to do this now. I have a lot of Scrobbles on lastfm, but they never showed up anywhere in my Arc timeline, although the username is correct.

Question 1:

Should I first work out all the gaps in my (old) Arc Timeline, try to fix it, before moving everything to Arc 4?

I know there are no Arc backups of these gaps. But I have recordings of the Apps MoveX, Pado and some GPS loggers with .csv files. I haven’t tried to import many of these, but I know there are some .gpx made by MoveX and Pado I can import to Arc timeline. In addition, it would be possible to reconstruct some GPS coordinates through my photos I have taken.

Question 2:

Will there be any kind of import feature for GPS logs / csv data, or through photos with GPS, or any kind of manually adding (historic) data to the timeline (through GPS data or simply by clicking on the map/Google locations + manual data stamp)?

I would really appreciate any kind of tips to this matte, so I can decide if I should put effort into fixing my (historic) timeline now, of postpone it to later (one day in the far future )

Thank you!

[matt]

@dagguet

First I realised, that Arc must have stopped recording when the new app or the last Update was released and I have a gap of 12 days I really would wish it would notify me (more throughly??) when this happens. Maybe this could be a feature request for the future, as this is something that frustrates me a lot.

Hi @dagguet!

For this do you mean the old app or new?

For the old app, there’s the “deadman switch” notifications, that would alert you if the app had been terminated and not automatically restarted. Though I suspect many people had notifications turned off, so didn’t see those.

There’s also Arc Recorder, the buddy app that takes over recording if the main app is terminated, thus preventing data gaps. At some point in the future Arc Recorder will be updated to act as fallback buddy for the new app rather than the old.

Though the new app is much more resilient to iOS’s moods, and tends to run for days or weeks uninterrupted, without termination. So there’s less urgent need for Arc Recorder. And for the notifications, my impression was that most people turned those off or ignored them. Though given the new app goes much longer without being terminated (iOS seems to treat it much more kindly) perhaps the notifications could make a comeback, given they’d be far less frequent, and perhaps people would be more inclined to leave them turned on, and pay attention when they show up.

@dagguet

Also, the lastfm feature

This is making a comeback in the new app! Planned for a feature update at some point. The ticket number is BIG-395, in case you want to search for mentions on the forum or look out for it in release notes for updates.

@dagguet

Will there be any kind of import feature for GPS logs / csv data

GPX import is coming to the new app! The ticket number is BIG-204, filed as high priority, for one of the upcoming updates.

@dagguet

I would really appreciate any kind of tips to this matte, so I can decide if I should put effort into fixing my (historic) timeline now, of postpone it to later

I think the cleanest and easiest path is to just migrate to the new app now, and wait for various import features to arrive there, rather than trying to do importing and cleanup in the old app then doing a fresh migration to the new app.

The new app has all sorts of import features planned, and some of them arriving soon. And doing cleanup or importing first in the old app would then require you to delete the old app, reinstall, and do fresh migration. Quite a bit of fuss. So as long as you’re not in a rush, easiest to just wait on the import features in the new app!

Cheers
Matt

[dagguet]

Hi Matt, thanks for your answer!

I actually do have the notifications of Arc timeline (old), and it does often tells me, that recording has stopped. However, I suppose I didn’t see it or swipe it away when I got a ton of notifications at the same time. I would also continue using notifications in future Maybe there could be some sort of visual hint, if recording is OK or not OK. For example, the Duolingo app changes its icon, when you haven’t logged in for a while (the own is “melting” like a cartoon character, it’s hilarious and got many positive feedback). Or, the widget could be enhanced (although widgets aren’t in my main view point). I even use the Home Screen widget, but I guess one gets so used to it, you don’t see the informations anymore, if it stays visually similar. I have never realised there was a second App - Arc Recorder! Maybe it would have saved me some trouble in the past.

About the import: this is great news! and I will wait for it once it update was made. Will it also be possible to import csv out of GPS loggers? because I tried to convert them to gpx and import them to moves, it never worked. and I don’t really know how to work around. I don’t mind if it is a perfect path, I just would like to have my historic data in the same spot.

However I got a bit into a map nerd mode yesterday and adjusted some timelines here and there in the old Arc timeline. I realised my old Moves imports were not complete and did integrate the moves-export-zip again, it somehow added to my timeline… parallel I have already used the new app since two days should I deinstall / delete Arc 4 and then install it again and start the import again?

Thank you!

[matt]

@dagguet

However, I suppose I didn’t see it or swipe it away when I got a ton of notifications at the same time.

Fair. Notifications are easy to miss and ignore. I have a pattern of only noticing critical ones 2-3 days later, which is rarely a good outcome!

@dagguet

Maybe there could be some sort of visual hint, if recording is OK or not OK.

The old app’s widget actually has this! There’s red/green dots on it, which indicate Arc Timeline and Arc Recorder states. If red it means the app’s heartbeat has stopped, so it’s not there anymore to record or act as fallback recorder for the other app.

That’s been my main way of keeping an eye on them, for the old apps. For Arc Timeline 4 there’s no widget yet, but we’ve got a ticket filed for that (BIG-140), so that’ll be making a comeback hopefully soonish.

We’ve also got a ticket filed (BIG-217) for a Live Activity, which is what DuoLingo uses when it wants to get right in your face about doing a lesson so you don’t lose your streak. We could do something similar for data gap prevention, perhaps. Though… we might be technically limited there, in that the Live Activity might require the app to be alive, which is the exact opposite of the situation we’d be in. Hmm… Anyway I’ll make a note to look into it.

@dagguet

I have never realised there was a second App - Arc Recorder! Maybe it would have saved me some trouble in the past.

Ah, yeah! Arc Recorder is the main lifesaver there, preventing data gaps instead of just warning about them.

Though for the new app we have much less need for Arc Recorder. The new app stays alive without interruption for days or even weeks, while the old app got terminated by iOS almost daily. But still, Arc Recorder is planned to make a comeback, as fallback buddy for the new app.

@dagguet

Will it also be possible to import csv out of GPS loggers?

Not impossible! Though I’m guessing that each app has its own CSV schema. So we’d need to figure out what’s out there and which schemas are most popular, and then add support for those.

GPX is the more universal solution, so we’ll be getting to that first. I think a bunch of people have been asking for that, so it’s high up the todos. Ticket number is BIG-204.

@dagguet

should I deinstall / delete Arc 4 and then install it again and start the import again?

Up to you on that one. There’s pros and cons. If you do fresh install and reimport you’ll get the full data from the old app as it is now. But you’ll lose any data the new app had recorded since previous install.

The new app records data with higher accuracy and detail - it’s generally better data overall. But if the old app’s got the more complete timeline now, then a fresh migration will get you on balance to a better place in the new app. Go with what makes the most sense to you there, I think!

[Fly]

I definitely rely on the old Arc notifications. I’m an app-zero and notifications-zero type. most apps have their notifications off, so I always see when Arc pokes me when it can’t get back open. My first go at Arc 4 resulted in empty timeline time when it didn’t warn me that it was dead, and that was the first time I deleted it! (It’s back again, and I’m excited for the App Store to show me the latest version with the image export, hooray!)

[dagguet]

Thank you matt. I have imported everything into Arc 4 and I’m cleaning up my timeline a bit in there, and waiting for the GPS / CSV integration possibilities in future.

However with the new Arc 4 sum of the visited countries and places, I realise everything needs to be cleaned up better. It seems only places I have actively recorded (e.g. a home or work) are shown in the overview, the automatic places (e.g. screet name given by Arc) don’t seem to be enough. Or I need to revisit all my data in history, so it gets a refresh? This will definitely take a while (since 2015 with my moves data )

[matt]

@Fly

I definitely rely on the old Arc notifications. I’m an app-zero and notifications-zero type. most apps have their notifications off, so I always see when Arc pokes me when it can’t get back open.

Noted! We’ve filed a ticket for adding those notifications back now. Ticket number is BIG-541.

Hopefully in the new app you’ll see less of them and have less need of them. But yep, they’re definitely filed now for making a comeback in an update.

[matt]

@dagguet

Or I need to revisit all my data in history, so it gets a refresh?

Ah, for this one it’ll get there on its own! You’ve probably already noticed a loading spinner under the top section on Places tab. That’s Arc doing the lookups for city and country on all the places in that view that are missing those values.

This is actually semi related to the street addresses thing mentioned in the other thread. It’s the same Apple reverse geocode lookup service, though in this case being used for Places rather than Visits.

That lookup service is rate limited, so it’ll backfill only gradually. It might take a few days to finish filling back in. But each time you see that loading spinner on Places tab, that’s the queue working its way through the backlog. And eventually you should see the numbers converge towards sensible values.

[dagguet]

Hi Matt, I actually haven’t seen the spinner wheel, but it seems to make a great difference if I have revisited each day in my timeline, or not. I tested it with December 2015 and before was only 1 city and a few places, after going through each day (without cleaning up the timeline manually), it added up the cities and places after. I guess it needs this kind of push. it’s ok, because it makes me revisit a bit of the places where I have been. And to be honest, I haven’t always checked on my timeline in arc in the past, sometimes it’s quite a mess, so I’m rearranging and cleaning up everything now. But just fyi, here are two screenshots before and after visiting each day. And there are still missing places and cities, which add up after cleaning I suppose.

After import of data (a few days ago) and a few time having opened the monthly and yearly activity tab, without diving into the month:

After revisiting each day without cleaning:

After manual timeline cleaning:

[matt]

@dagguet

but it seems to make a great difference if I have revisited each day in my timeline

What you’re likely seeing there isn’t backfilling of reverse geocode, but backfilling of place assignments.

Viewing timeline days doesn’t do anything different than viewing the same data in Places tab in terms of country and city lookups. It does however do automatic place finding for visits that have no place assigned. So most likely what you’re seeing is old visits that had no place assigned being filled in with automatic place assignments. Then when you come back to Places tab the report has updated to include those new places.

The Places tab only collates numbers on places, not visits. So that’ll be the disconnect you’ve been seeing. Once the visits have places assigned, Places tab’s reports will update to include those.

I had a bit of a ponder on this, thinking about whether Places tab could also do those place auto assignment lookups. But I think that would be a step too far for Places tab, going outside of its reasonable scope of responsibilities. Timeline cleanup (either manual or automatic) should I think stay in Timeline tab, while Places tab remains limited to merely building reports on the timeline shape as it stands at report building time.