There’s something about This and That which I think will make it massively more useful:
Please offer a button to “calculate everything right now” so that I can click it and then go do something else. When I’m back I’ll be able to quickly browse everything without waiting.
Right now I have to wait 3 minutes to look at any 1 correlation (I have lots of data since 2014) and it’s super inefficient. If I want to look at 10 things I need to tap and sit while I wait for 3 minutes to look at one. Then tap and sit for 3 minutes… at the end I’ll have spent 30 minutes. Instead, I’d prefer to leave it running for whatever hours are required and I can just instantly browse everything later (even the next day, if necessary).
I don’t mind having to keep my phone unlocked all day with the app on the foreground. But I do mind having to spend 30+ minutes to look at something that would take me 4 or so minutes if everything had already been processed.
I’m opening this feature request now after realizing I’ve just spent the past 65 minutes looking at things in the app when I could have looked at everything in at most 15 minutes had it already been processed.
IMO this feature is a must have because every time you get new data, you have to wait the same time for it to update again, so it’s not like you only have to wait for it once (though that would still warrant the feature).
Having one button that takes as long as necessary to update everything will make it much better and faster to use. Right now, by the time a new one finishes processing I don’t even remember what I was looking at before
Hi @usernamehere! Thanks for the kind words, and I’m glad you’re liking the app!
What you describe is actually the core problem I’ve been trying to solve with the app for a while now. (When I get a chance to spend time on it - This & That isn’t a big money maker, so I can’t justify spending as much time on it as I’d like yet).
The problem is that there’s just too much to calculate - calculating every correlation for every data pair would take a massive amount of time, and then need to be all redone again the next time the data from Health app updates. To put it in context, there’s about 150 supported data types, so that’s 150 * 150 combinations, ie 22,500. And then for every data types pair there’s 121 lagged correlations to calculate (lags of -60 to +60), so in total that’s around 2.7 million correlations to update!
The main problem at the moment isn’t the actual updating of the correlations (that maths can be optimised to run very fast, and to be done in parallel), it’s the saving of the updated values to the database. Reading from the database can be done in parallel, which means multiple reads can happen very fast, but writing to it has to be done in a serial queue, which means writes are slow. And when you’re trying to write potentially hundreds of thousands of changes… Yeah, it’s a problem.
That problem is basically why I changed the updating to be “on demand”, as it is now. In earlier versions of This & That I just did as you suggest, updating them all at once whenever the Health app data updated. That worked fine when the app was only calculating and storing three lags for each data types pair (previous day, same day, and next day). But once I bumped up the calculated lags to -60 to +60 the problem ballooned into something unmanageable, and I had to change the app to only do the updates at time of viewing, ie on demand, otherwise it’d be sitting there for hours trying to catch up, every time the Health app data updated.
I do have plans though! In each recent update I’ve improved the speed of the updating incrementally, and I’ve still got a few tricks to try to speed it up even more. Right now I think the biggest gain will be in finding out why the updates take much longer if they’re happening at the same time as the app is importing updated Health data. Like, if you see on the main view “Syncing Health data”, that syncing/updating process is for some reason blocking (or massively slowing down) the subsequent updating of each correlation.
So I suspect that if I find out why those two tasks (“Syncing Health data” and “Updating correlations”) are clashing, and fix that, the situation will improve significantly.
What I’m tending to do in the mornings now is open the app then wait until “Syncing Health data” finishes before going in to look at any updated correlations. Because once that sync is done the correlations tend to update very fast, such that it’s not slow enough to be annoying anymore. Of course that’s still not ideal, because syncing the Health data can take quite a while each morning anyway, so yeah, I’ve still got to figure out why they’re clashing.
Anyway, short story: I’m on it! Hopefully will be able to speed it up soon
Thank you very much for the detailed response, I find it very interesting (and I’m an iOS developer too so I can relate) and I’ll try the tip to wait for HealthKit to sync first, though I think that will be short-lived because the Apple Watch frequently uploads new heart rate measurements/etc to HealthKit. I’m not sure the process it will ever finish, but I’ll try (maybe take off the Apple Watch before opening the app…). Either way I’m happy to know you’re on it.
But I think I didn’t express myself clearly. I didn’t mean to change the current app behavior; always updating on demand by default is fine. I just wish there was a button that, when tapped, would force a one-time full processing of all data (even if it takes several hours). After the one-time completion, the app would still process only on-demand (but the user can tap it again after it finishes processing if they want to trigger another full processing).
I imagine this button could work as a stop-gap feature that very patient users can use (like myself, who can just leave the iPhone running all night updating it when I’m sleeping) while you work on improving the speed. I imagine it would be simpler/faster to add such a button than to troubleshoot the slowdown right now, in particular considering this was the default behavior before, and as you mentioned you have limited time to work on This & That.
So in conclusion, my feature request is to keep the app exactly as is (process on demand), but add a “Force full processing now” button which, when tapped, runs a one-time full processing of all data and all correlations, warning the user “it may take several hours to complete” or something like that. The button can be tapped again once it completes the one time full processing, so it would be possible to e.g. tap it once every day at night so it runs while the user sleeps (I can leave it running in Guided Access mode to force the app to stay on foreground, screen doesn’t lock, and iOS automatically reopens the app if it crashes).
By the way, I know this could be processed remotely on a server, but I’m very glad you did not go that route. We need apps that run exclusively on-device, with no account creation and no cloud sharing of data, even encrypted… privacy is very important, all data should stay locally on-device! And you do a great job at that.
An idea for the future: since iPadOS 17 now has HealthKit too, you could also port the app to iPad and run the processing on the faster M1/M2 iPad chips and sync the already processed database back to the iPhone with iCloud.
Fortunately those updates throughout the day are much faster to process. I’m finding that it’s only the first time I open the app each day that’s overly slow to sync. Largely because that first sync includes all the data types that might change overnight, eg sleep and sleep related, while the other syncs during the day are usually limited to things like Active Energy Burned, Steps, Distance, etc.
I suspect the sleep stage data types are also a bottleneck at the moment too. Because that sync has to determine sleep session boundaries (ie when a sleep session started/ended) it currently has to do a full sync every time HealthKit indicates the data has changed, because there’s no way to safely determine which date ranges to fetch without risk of accidentally truncating a sleep session at the boundary.
But I’m hopeful there’ll be a clever way to get around that, and make those sleep stage data type syncs incremental. I suspect it might be as simple as extending the fetch date range 24 hours out in either direction beyond the range HealthKit reported as changed, then using that as buffer. It’ll be fiddly code to get right, but unless I’m not thinking of something, that approach should work, and could give significant speed ups in that first sync of the day.
This was also something I wanted (though I’d planned on it being an automatic thing in the background once a day, rather than an explicit button). But what I quickly realised after extending the lags to -60 to +60 was that it’s actually a massive amount of database bloat, for data types that the user might potentially never be interested in and never look at.
The first stage at the moment, when you navigate into a data types pairing for the first time, is it creates all the Correlation database rows for that pairing. So that’s roughly 150 * 121 = 18,150 Correlation rows created on first view. But if the user never even cares to look at for example “Running Power vs Appetite Changes”, the database is better off never having those Correlation rows created in the first place.
Though the background full sync could still be done for all data type pairs that have been viewed at least once. So that’s still a plausible idea. Perhaps if I can twist HealthKit’s arm to reliably do background syncs, getting that stage done before the app is foregrounded could be a big improvement. But that’s a big if - iOS typically isn’t reliable with those kinds of background updates, especially if it’s a big workload. But still, worth a try.
Strong agree! When I was first building my other app - Arc App - it was around the time Facebook bought Moves app (Arc’s original competitor), and suddenly everyone’s sensitive all-day location data was being stored and used by an incredibly untrustworthy company, and that set off alarm bells for me. And even if you do trust the company storing the data, health and location data can be extremely sensitive, so it’s just too great a risk to take.
This is already in the works I even wanted to have the iPad app ready for v1.0. Though it ran into the same problem as above: it’s just too much data to sync. The CloudKit syncs just took too long. (Aside: It will use a private CloudKit container, so is fully encrypted with user’s private iCloud key, and not accessible by me or even by Apple).
But as you say, with iPadOS 17 having HealthKit now, that takes away a big chunk of data sync necessity. And I’m also quite keen to try the new CKSyncEngine too. Doing very large CloudKit syncs is quite a handful to manage, so if the queuing, rate limit handling, retrying, optimised background syncing, etc can now be reliably handed off to a system service, that might improve things considerably. So I’m keen to get back to working on the iPad app as soon as I can get a chance!
Notably, there’s several feature ideas I’ve been holding back on because they’re data visualisations that won’t fit well on an iPhone screen, but would work great on an iPad. So the iPad app is quite exciting in that sense too.
