@matt if the constant background processing is making Arc going in the shitlist, how about making a night processing mode like Amazon Photos does for its backup system?
Use those low accuracy stuff for basic display during the day, and give the users the ability to postpone all processing to times they know they can let the phone 100% dedicated and plugged in to it, like at night.
would it work?
@Amido, Arc already does this. All not immediately essential processing tasks are deferred until the phone is plugged in to power, and additionally are filed with iOS’s background tasks system, flagged as requiring power.
That “requires power” flag opts the task out of iOS’s resource use watchdogs. Those watchdogs are things like “sustained 80% CPU for more than 60 seconds”, which going over will result in the app being terminated. So while the scheduled tasks are running, they’re not at risk of going over one of the watchdog thresholds and getting the app terminated.
That however doesn’t opt Arc out of iOS’s overall friendliness metrics it maintains for each app, on daily / multi-day durations. All apps are graded (by undocumented metrics and systems) based on their usage, both by the user and in terms of system resources, and iOS will treat the app more or less kindly depending on those unknown metrics.
All of Arc’s deferred / scheduled background tasks ultimately still need to be done. Backups, ML model updates, database housekeeping, etc. So while they don’t result in the app being immediately terminated, due to being run inside iOS’s background tasks system and flagged as requiring power, they still count against the app’s overall resource consumption metrics over time, and affect how kind or cruel iOS will be towards the app.
Lowering recording accuracy won’t improve anything, as Arc’s recording engine (LocoKit) is already extremely energy use optimised. Reduced accuracy wouldn’t result in energy use reductions of any worth.
I think I understand the situation, but there are things that you very kindly explained that seem at odds with each other.
You state that lowering the accuracy won’t change the situation, but then say that the other apps have easier life because of their lower accuracy and use of other data sources.
I, for one, would love having the ability to lower the accuracy, use “standard and approved” sources and have better reliability of tracking, given that the raw data still sucks ass so I constantly end up with stuff like airplane 5 meters → walking 2000 km → stationary at the store next to my house 9 hours overnight. It’s not like this is dedicated hardware for accurate tracking so I don’t think you’re exactly considering all of the app’s strengths by saying you’re in the super high accuracy niche. I value Arc mostly for its constant export to JSON and its decent UI, for example. But I’m just one guy.
Another thing you say is that all other apps have this termination problem, but I can attest that Tracer and SilentLog have been running on my phone for months, almost a year now and never have skipped one single second of tracking. Whatever they do, I wish I had it in Arc, so I’m not locked into any app if shit hits the fan and you can’t develop this app anymore (and thanks from the bottom of my heart for the open source version for this reason, it truly is a gift to the world).
For the deferred/scheduled, I strongly encourage to take a look to Amazon Photo’s system, for real. Their app also never gets booted from memory, uploads the pictures pronto and has this beautiful “do all the heavy lifting during the night” mode that dims the screen while it uploads all your camera roll. I’d rather have Arc tell me weekly “yo man, time to let me do a night thinking about all of your crap” than having holes in my data/having it never fully analyzed recent stuff because it tries not to get booted from memory. But that’s me.
Regarding this naughty list, this is the first time I read anything about it ever, and I develop for iOS, too. Do you have some evidence/data/documentation about it, or is it just a theory you developed from all your fights with getting booted from the OS?
It strikes me Arc is especially terminated when my phone is stationary for a long time. This would suggest that tracking activity has nothing to do with the termination problem
You won’t get it. Lowering Arc’s detail or accuracy won’t do anything to reduce terminations.
Before reading further: If you want a solution to the terminations problem, install Arc Mini, and add one of Mini’s widgets to one of your home screens. Having multiple recorders will almost completely solve the problem.
There are two specific niches that Arc caters to: 1) highest possible accuracy and detail achievable with current hardware, and 2) best possible data privacy achievable with current technologies.
To reduce the chances of iOS terminating the app in the background, niche 2 would need to be given up on. It’s not recording detail level that causes the terminations, it’s the fact that Arc does all of its work on device, with no server side processing or storage required.
The reason why I highlight differences in the way Arc records versus other apps, is to emphasise how they are fundamentally different classes of apps under the hood. But that doesn’t mean that regressing Arc’s recording functionality to that level will solve any problems - it wouldn’t - it would leave the terminations problem exactly at it is, but with less detail and accuracy, for no actual benefit.
To highlight this: I run Arc v3, Arc Mini, and Arc v4 on several test devices that I take everywhere with me, for testing purposes. All of the apps use the same underlying recording engine, and record at the same accuracy and detail, but v4 doesn’t yet do many/any of the daily background tasks. Arc v4 has typical uptimes measured in multiple days or even weeks, because it only has the job of recording, at this pre-release stage. That makes it similar to competitor apps, that rely on server side processing, so that the app on the phone only has to record and nothing else.
Having Arc v3 and Mini sharing the job of doing the daily background tasks also reduces the risk of each of those apps being terminated, such that each of them often has multi-day uptimes, terminated only when I relaunch the apps during development and testing. Which is why I say installing Arc Mini will almost completely solve the problem for you.
But back to the idea of an option to reduce Arc’s detail or accuracy: As I’ve made clear, it won’t achieve anything useful in terms or increased reliability. But also it’s simply not the point of Arc. If you want less detail, you can use another app. I’m not interested in making that app. Building an optional, fundamentally different recording engine, designed to produce less detail and accuracy, simply isn’t what I’m here for. If that’s what you’re interested in, I recommend petitioning the developers of one of the competing app to add the export functionality you want.
It’s indicated in some device logs that mention scores when terminating apps, and has been discussed (in vague terms) in WWDC sessions in the past. The device logs are hard to find, and I forget exactly which ones they are, but think in terms of lists of running apps/processes, and indication of which ones were terminated, for what reason, and with vague scoring notes attached.
I also know it as a certainty due to 5+ years of testing, on a bag full of test devices. As an example, if a specific test device is used for heavy debugging during dev of a new feature, perhaps with high resource usage in foreground or background, the app on that device will be punished, sometimes for a period of days, while the exact same build installed on the other test devices will run reliably without termination for the same days. Or if for example one test device is running a new daily background task, that the other devices aren’t, and that task isn’t yet well optimised, the app will be punished on that device. Not necessarily at the time the task runs, but at other times during the day.
From my experience and experimentation there has in the past also been a factor of foreground vs background resource use time. For example a freshly installed app, that then gets almost no foreground time (ie the user installs it, opens it for a few seconds, then gets bored and moves on) will be punished more, with more frequent terminations, based on the same background resource use as the app on another phone that has had more foreground time. Perhaps some sort of “user interest” metric, that permits the app more tolerance in background resource use.
I did, and it definitely helped. It still feels super weird and most users won’t know about it. And it will solve the symptom, not the cause, meaning I still have to reload Arc for the data to make sense otherwise when I need it, all I get is “thinking” and super sluggish UI.
Yes, but that’s why I asked what about deferring everything to a mode where the screen is on, like Amazon Photo’s system. You don’t want to add a low accuracy mode that doesn’t require this much processing - fine, but what about making sure that this processing is done in a moment where it doesn’t hurt the core functionality of the app?
I mean, I understand you don’t have the time to build a new tracking engine, but you still went to the extremes of making an entire other app partly in a desperate move (if I understand correctly) to fix this problem, something that I’ve never seen done by anybody else in the history of iOS, so it’s not like you are not willing to sink time into solving this, even with extreme decisions.
I’m super grateful that Arc Mini came out of this endeavour, don’t get me wrong! I still think, from my humble dev experience and from my mere user point of view, that requiring your users to find out that they need another app for the first app to work properly should be regarded as a super last ditch resort.
Like, you literally said that if you remove the processing from the app it doesn’t get kicked.
Then for real, don’t process this stuff in the background, and make it clear it needs foreground time. At least give us the option. Yeah, of course I’m pestering the devs of the other apps to add the features I need from them so not to depend on any single app/developer, but this is Arc’s support forum, not the other devs 
You wrote this in another thread:
This is literally what I’m saying, but with extra steps, and it needs the user to keep the app alive by tapping on it from time to time. Way better to have a dedicated “processing” mode.
I could even make an automation to open the app before going to sleep!
Maybe one day all of these swiftui efforts will pay off and you can let us install the app on Mac and use the M1 to plow through the data in seconds… (;
Arc’s periodic/daily processing tasks have nothing to do with Arc’s accuracy or detail.
All location recording must be done in real time, due to phones not storing location data for later retrieval - you either get the detail at the time or you don’t get it at all. While recording, Arc records the most detail as possible (but with similar energy efficiency to less detailed apps, due to careful optimisation).
The background tasks that are performed later are things that are related to Arc’s privacy features, not detail and accuracy features, such as updating on-device ML models, which other apps would instead do server side.
This is basically the same as how Apple manage ML (machine learning) in their own apps, such as iOS and macOS Photos app. ML classification and model updates are done on the device, which is why things like face identification are slower and more energy intensive than with Google products (which do it all server side - something that isn’t possible with Apple’s and Arc’s privacy commitments).
Arc Mini has not been built to solve this problem - it’s simply a convenient extra benefit. Arc Mini is a ground up rewrite of Arc’s UI layer, as a way to cleanly transition to Apple’s SwiftUI layout engine, and as part of Arc’s ongoing transition to open sourcing of as much as possible. Arc Mini also serves as fully functional and complete example code and example app for third parties making use of Arc’s open source recording engine (LocoKit).
Arc v3 is essentially in maintenance mode, with no new major features planned for it. New features are instead intended to be added to Arc Mini and Arc v4 (as yet unreleased), with Mini serving most of the purposes that v3 currently does, while v4 goes in a different direction, thus necessitating being a separate app. That transition has been slower than hoped, due to SwiftUI being less capable and ready than I’d hoped, but remains the intended goal.
Please understand that I’ve spent over 5 years full time on this single project, been developing for iOS for over a decade, and programming for over 35 years. Arc’s underlying open source recording engine (LocoKit) is used in a range of third party apps and university research projects. This is my zone - I know what I’m doing.
Matt, I don’t think we are on the same page. I’ll try to explain myself again, hopefully I’ll suck less at writing this time.
Whatever heavy lifting Arc does, for whatever reason, would you consider implementing this:
- bundle all the heavy lifting coroutines in a big, single alternative ProcessingMode function
- put this function behind a button, say in the settings or in the … menu, or in a top bar warning that you have stuff to do
- the button will also set the app to prevent screen sleep, dim the screen and put a nice modal pane on the lines of “processing mode has been activated, it will stay on until all processing has been done”, possibly with a progress bar
In order to give us a reliable way of giving Arc its time to process all the stuff? I’m the UX guy at my company and I really like the idea of nailing down the defaults but give the users the ability to choose how the app should behave. On this note, the non-optional warning about low battery is… not my favorite
This won’t work. It doesn’t matter whether the heavy lifting is done in foreground or background, the app will be punished the same. The only benefit of doing it in the foreground is ensuring that it gets done at all (which is why Arc triggers some processing tasks in the foreground, under some conditions, if iOS hasn’t allowed that task to complete in the background recently).
Foreground processing is a workaround for iOS not allowing enough background processing time for scheduled background tasks, but it is not a workaround for resource usage thresholds and scorecards.
Please understand that you’re not going to think up anything new here that I haven’t already spent literal years considering.
It is necessary. Foreground time is the most energy expensive time for apps, by orders of magnitude. The phone’s screen is the largest energy consumer next to the CPU, and UI updates the most energy expensive parts of most apps. Especially the map view can drain battery rapidly, exhausting remaining battery in minutes. In order to preserve remaining recording time, the full interface can’t be loaded. Arc’s Low Power UI is the difference between 60 minutes remaining recording time and 5 minutes.
The Low Power UI is optional under some conditions (ie there’s an option in the menu to load the full interface), but non-optional in cases where loading the full UI would either drain the remaining battery in a matter of minutes, leading to device shutdown, or would push an already critical thermal state over the edge and lead to OS system services shutting down and mass app terminations. (The Low Power UI is used for both low battery situations and high thermal state situations).
It does matter as you can shoot your usage at 100% on all cores, take one hundredth (or less) of the time to do all the heavy lifting, and then stop most, if not all the processing for maybe days. Since this scorecard is all a conjecture (there’s no official documentation about it) you can’t rule out it will work until you actually test it, as you said nobody knows how the scoring is actually done. I, for one, can’t see Apple deciding that foreground and background processing are weighted the same.
Even beyond the booting from memory, you have plenty of threads of people with uncompleted backups, uploads, processing and all kinds of things that would be solved or hugely helped by such a system. I would reconsider it, even if you already thought about it in the past and discarded the idea.
As a default, yes, it is necessary.
To not give the user the ability to decide if they want opening your app to be the last thing their phone does before dying, and not pester them anymore about it? not necessary. Let the users decide, please.
Scheduled background tasks use a specific iOS API that allows for flagging the task as requiring power, which opts it out of resource usage watchdog thresholds during task execution, which is identical to (or even better than) foreground processing. It allows multi core high CPU use, without risk of immediate termination.
However neither those scheduled background tasks nor foreground processing are exempt from the scorecards, based on years of testing has proven.
As I’ve said, I’ve been working on this full time for over 5 years. For the first 3 years I carried around a bag of 6+ test devices everywhere I went, testing every mode of transport, in multiple countries, many different cities, dense environments, underground, overground, etc etc. I collected extensive data and established a bunch of proven patterns (whilst also ensuring that Arc was conclusively both more accurate and more efficient than competing apps).
I now only carry around 2-3 test devices most days, but still do regular data collection and comparison between devices, and occasionally competing apps (though none have popped up in a long time that are worth doing consistent daily testing against).
Again, I have tested this, and many many more things, over a period of years. I don’t know how to make this more clear to you: I know this inside and out, in absurd levels of detail and certainty, and there’s nothing you’re going to think of that I haven’t already spent potentially years assessing already.
It is optional in cases where permitting user choice is viable. You can tap the menu button, and choose to open the full UI. If doing so would lead to battery exhaustion within minutes, or critical thermal state, it is not optional.
If you would like to continue this discussion, it would be best to take it up in a new thread, or in private messages. At this point we’re just going in circles, and it’s likely not of any benefit to other readers.
Agree, this discussion is leading nowhere…
The last few weeks Arc has crashed repeatedly when I click on anything - for example, to confirm a location, a mode of transport, or check yesterday’s data. Arc mini keeps on plugging away, seems to never crash. Hopefully when the main app gets back to working again, Arc mini data will save the day. Both apps and iOS are up to date, too.
@ezslides Can you reproduce the problem every time? Or is it somewhat random?
There’s no common crashes at the moment, so it will be something that’s happening specific to your phone. Which means the first step will be to reboot your phone.
Same here. Reboot doesn’t help. No relations in disconnections to discover. Even to activate again is unstable, sometimes directly after closing and opening app, sometimes several times closing and opening and sometimes by clicking several times.
Conclusion: Arc doesn’t do what it should do. Time to say goodbye or a good solution?!
Based on the descriptions so far, this doesn’t sound related to other known termination issues. And there’s no recent increase in crashes for this release (it’s got the best crash rate of releases over the past many months). So @ezslides or @sanderb it’ll be worth posting a description of the problem in a new, separate discussion (ie the New Topic button), so that we can track it separately.
Useful details will be any patterns you can spot, but also things like device model, whether plugged in or not, whether in Low Power Mode or Airplane mode, basically any and all context you can think of. Thanks!
OK, I hoped data gaps would be a thing of the past, but despite Arc Mini there are still data gaps. Today recording is only resumed in the middle of an activity. Not surprising, really since Arc Mini is only symptom fighting. And again the circumstances are as always: being in the same place for a longer period (say 24hrs) will stop the recording process. Moving to another location will restart recording reluctantly
All in all nothing has changed
@Dinky, if you want the core problem solved, talk to Apple. It’s their decision, and under their power to fix, not mine.
This is absolutely not true. I have put in massive amounts of effort over many years, to work around iOS’s limitations. It’s a constant, unending battle, and when people say “nothing has changed” (when it’s absolutely not true) it makes me wish I never bothered. So if that’s going to be your attitude, cancel your subscription and move on.
I’m closing this thread. People who want to work through specific termination issues can create new threads, describing the symptoms on their phone, and I’ll deal with it on an individual basis.