Hi! After using This&That for a while now, I’ve noticed that the sleep length isn’t present in the Data types list, even though my Health app has it, and This&That has the permission to read it.
Is it a bug, or is there some other reason why it’s not available? I feel like sleep is closely related to lost of other health metrics, so not having it feels wrong…
You should see a “Total Sleep” data type, which is the duration of actual time asleep during the sleep session, regardless of sleep stage (so it’s light + core + deep + REM).
I also wanted to have the total duration of the sleep session, ie light + core + deep + REM + awake, but for some reason that was really fiddly to get done.
I think it might’ve been something to do with how HealthKit’s data types changed once they introduced sleep stage tracking, but they also kept the old types because people’s old data still exists, and that created some ambiguity. But I think I could still see a way to get it done, it was just going to be more fiddly than I’d hoped, and I had a bunch of other data types that also needed custom handling, so I marked the sleep types task as “close enough for now; move on!” 
I’m happy to be pushed to get back to that though! If there’s more than one person who cares about it (ie more than just me) then it’s easier to justify spending time on it.
Yeah, then I think something went wrong in my case 
Though it’s a bit weird, some of the sleep data I have is still of the old “just length” type, and yet it also wasn’t detected by the app, so maybe there’s also some other problem?
Oh wow, yeah, that does look like a bug! Very odd.
Though I’m drawing a blank on how that could happen… Weeeiiiird.
You could just delete and reinstall the app, to get it to recreate the database structure and data types. Though that would mean you’d lose any “Favourites” you’ve pinned. But otherwise wouldn’t do any harm, and would rebuild the data types so that hopefully nothing is missing this time. I regularly do fresh installs on my main phone, while testing and debugging, and it’s not too much of a hassle (apart from the first big data sync from HealthKit taking a while).
Though if the problem isn’t in This & That’s database but instead in iOS failing to return any sleep data, then it won’t help. Or… perhaps doing a fresh install would trigger iOS to reevaluate the permissions and start returning the data.
Would be nice to know which is the cause of the problem though, so that I can try to fix it or work around it in an app update. But… don’t think there’s any way we’ll be able to figure that out. Oh well. Let’s just hope the fresh install fixes it
For that, I’ve had that sometimes in my data too. Usually it’s something like sleeping on an airplane, so presumably the accelerometer data the watch is getting is too noisy and it doesn’t meet the threshold for determining sleep stages.
I think in those cases This & That will still pick up the sleep, but it’ll only store a “Total Sleep” value, by combining the … actually, I’ll have a peek at the code:
// MARK: - Sleep
private let mindfulnessAndSleepSampleTypes: Set<SampleTypeConfig> = [
SampleTypeConfig(.mindfulSession, grouping: .mindfulnessAndSleep),
SampleTypeConfig(.sleepAnalysis, categoryValues: HKCategoryValueSleepAnalysis.allAsleepValues.map { $0.rawValue }, grouping: .mindfulnessAndSleep, customName: "Total Sleep"),
SampleTypeConfig(.sleepAnalysis, categoryValues: [HKCategoryValueSleepAnalysis.asleepDeep.rawValue], grouping: .mindfulnessAndSleep, customName: "Deep Sleep"),
SampleTypeConfig(.sleepAnalysis, categoryValues: [HKCategoryValueSleepAnalysis.asleepCore.rawValue], grouping: .mindfulnessAndSleep, customName: "Core Sleep"),
SampleTypeConfig(.sleepAnalysis, categoryValues: [HKCategoryValueSleepAnalysis.asleepREM.rawValue], grouping: .mindfulnessAndSleep, customName: "REM Sleep")
// SampleTypeConfig(.sleepAnalysis, categoryValues: [HKCategoryValueSleepAnalysis.inBed.rawValue], grouping: .mindfulnessAndSleep, customName: "Time In Bed")
]
Ok, so it’s using the allSleepValues. And Apple’s docs say:
This value includes asleep, asleepCore, asleepDeep, asleepREM, and asleepUnspecified.
Which means it also includes the deprecated old asleep type, which they replaced with asleepUnspecified in iOS 16. So yeah, looks like those nights where it doesn’t detect actual sleep stages should still show up correctly in This & That’s “Total Sleep” data. And old sleep data from before iOS 16 should also show up correctly in “Total Sleep”, because that old data will be using the generic asleep category too.
Welp, I have bad news, reinstalling didn’t help…
Is there any way I can help in diagnosing that? Maybe using a TestFlight version of the app with debug logging or something similar to that?
Well that’s super weird.
Yeah I think it might have come to that! I’ll have a think what logging we’ll need, then send you an email invite to the TestFlight group. Thanks!
Actually, before I send the build, could you try one thing for me? Poking about the code, I’m coming to think it must be a problem with HealthKit permissions - iOS not providing the sleep data. (Or if it’s not that, it’s something very weird that I haven’t thought of yet).
Sometimes when you delete an app and reinstall it iOS will just use the cached HealthKit permissions it had for the previous install. To force it into definitely re-requesting the permissions you have to do things like turning the phone off and on again (or in the olden days it was something like waiting 24 hours then trying again, which was not fun).
So it might be worth trying this: Delete the app, turn the phone off and on again, then reinstall the app.
A bit of a fuss, I know! But I’ve got a hunch that’ll shake out the problem.
Tried that as well, didn’t work unfortunately…
I guess then I’ll be waiting for your email! Just in case, my main address is me@astrra.space (not sure which one I used when registering here, others may have issues with receiving mails)
Thanks for all the good work you’re doing, hopefully I’ll be able to help you with this
Ah damn. Really hoped that’d work.
Ok, I’ve sent the invite and a build! You can see the debug logging by tapping the settings button at top left, then “Debug logs”. A new log file is generated on each fresh app launch.
For testing, you’ll be best to do a new fresh install with the beta build, so that you can get logging of the first run after fresh install, where the app initialises the data types table.
The lines to look out for in the logs will be:
INSERTING hkTypeIdentifier: HKCategoryTypeIdentifierSleepAnalysis, categoryValues: 4
INSERTING hkTypeIdentifier: HKCategoryTypeIdentifierSleepAnalysis, categoryValues: 5
INSERTING hkTypeIdentifier: HKCategoryTypeIdentifierSleepAnalysis, categoryValues: 1,3,4,5
INSERTING hkTypeIdentifier: HKCategoryTypeIdentifierSleepAnalysis, categoryValues: 3
Those will show that the correct data type rows have been inserted. Which eliminates potential bug 1, ie that the data types aren’t even being added to the app. That bug seems incredibly unlikely, but we’ve got to at least eliminate the possibility.
The next thing to look for in the logs will be lines like:
INSERTED/UPDATED DaySamples type: Total Sleep, dayDurations: 1306
INSERTED/UPDATED DaySamples type: Core Sleep, dayDurations: 439
INSERTED/UPDATED DaySamples type: REM Sleep, dayDurations: 439
INSERTED/UPDATED DaySamples type: Deep Sleep, dayDurations: 439
If all is working correctly, you should see those lines by the time the first HealthKit sync has finished. If the dayDurations are 0, or the lines are missing, then… HealthKit hasn’t provided the data. Which is my current best guess for what’s going on.
Although frustratingly, if that’s what’s happening, there’s no way for the app to detect it. HealthKit is designed defensively for privacy, in that it won’t tell an app whether the user has granted permission to a data type, it will only say whether the data type permission has been requested. And we already know that it’s been requested, because your Settings app shows that it’s been granted. But at least that would narrow down the problem, so we know what we’re dealing with.
Oh, one other thing to try would be toggling off the sleep data permission in Settings app, along with fresh launches of the app, restarts of phone, turning the toggle back on, etc. Basically switching everything on and off again repeatedly until the lights start working
Though I’m guessing you’ve already tried that. But yeah, that would be the next thing I’d try myself.
Huh, I guess iOS just doesn’t like me 
The INSERTING logs show up as you’ve described, but the INSERTED/UPDATED ones show 0 dayDurations, as expected…
There’s also a bunch of lines similar to this:
[14:42:27.114] [ERROR] [HEALTHKIT] Error Domain=com.apple.healthkit Code=5 "Authorization status is not determined for all types provided." UserInfo={NSLocalizedDescription=Authorization status is not determined for all types provided.}
[14:42:27.131] [ERROR] [HEALTHKIT] Error Domain=com.apple.healthkit Code=5 "Authorization not determined" UserInfo={NSLocalizedDescription=Authorization not determined}
There’s a ton of them, 99 lines of the first one and 142 lines of the second one (which are all the same except for the timestamp), which I guess is the thing about health not telling apps whether the permission has been requested or granted.
I’ve also tried reinstalling the app and granting only the sleep permission, which resulted in the same amount of “Authorization status…” logs, but also added 141 lines of:
[15:05:31.662] [ERROR] [HEALTHKIT] Error Domain=com.apple.healthkit Code=0 "(null)"
Which I guess is the app trying to read the data it actually doesn’t have access to?
But as for the “allowed” data it’s the same story:
[15:05:31.916] [ERROR] [HEALTHKIT] Error Domain=com.apple.healthkit Code=0 "(null)"
[15:05:31.985] INSERTED/UPDATED DaySamples type: REM Sleep, dayDurations: 0
[15:05:31.987] INSERTED/UPDATED DaySamples type: Deep Sleep, dayDurations: 0
[15:05:31.989] INSERTED/UPDATED DaySamples type: Core Sleep, dayDurations: 0
[15:05:31.991] INSERTED/UPDATED DaySamples type: Total Sleep, dayDurations: 0
[15:06:31.992] --1--
[15:07:31.996] --2--
[15:10:36.854] --3--
[15:10:37.239] [ERROR] [HEALTHKIT] Error Domain=com.apple.healthkit Code=0 "(null)"
[15:10:37.240] [ERROR] [HEALTHKIT] Error Domain=com.apple.healthkit Code=0 "(null)"
The thing that stands out to me is that we maybe actually can differentiate between which permissions have been granted and which haven’t, seeing that the sleep logs were the only specific ones that showed up in the iNSERTED/UPDATED section (but the INSERTED section contained all types).
And I also tried doing various other shenanigans with the toggles, reinstalls, restarts, and reboots, though to no avail…
Ugh. Well that’s very unsatisfying! I mean, it tells us that it’s a problem with iOS not providing the data, but that’s also something we can do nothing about 
Yeah, given the different error lines in your logs depending on granted or not, that seems like a hole in the privacy layer that HealthKit is supposed to provide. It’s supposed to act identically when requesting data, regardless of whether the permission has been granted or not. It’s meant to only say “you haven’t asked for permission yet”, and “there’s no data” (which can mean either there really is no data, or the user hasn’t granted permission, but the developer/app isn’t supposed to be able to know which).
Aside: All those lines of “Authorization not determined” are because on every launch the app tries to sync HealthKit data, even on first launch, but on first launch it hasn’t got that far in the onboarding yet, so the first attempt will fail. That’s just me being lazy, not trying to stop that first pointless sync attempt on first launch. Harmless, but noisy in the logs.
So anyway… where does that leave us… Actually I’m stumped. I literally can’t think of anything else to try. At this point it seems like it’s something that we’re going to get nowhere on without escalating it to Apple Support.
Unless there’s some detailed logging that can be turned on in iOS Settings app, perhaps. Having a poke around in Settings → Developer… Ugh. Nothing about Health / HealthKit at all.
Do you have any other apps installed that make use of sleep data from HealthKit? I guess we should find out whether it’s just This & That that’s getting nothing or whether it’s all apps.
Yup, I’m using RISE for sleep analysis and it gets the data correctly, even says which app was the initial source (Zepp life, the companion app for my smart bracelet… wait, hold on)
Actually, now that I think about it, can the source of the data be a problem? Maybe there’s some specific request for data from 3rd party apps or something like that?
It can be in some cases. Though third party apps don’t have to do (or shouldn’t have to do) anything specific in requesting the data, only in potentially processing it.
When apps query HealthKit they request a sample type (in this case .sleepAnalysis), possibly a date range, and whatever other conditions they want to put on the query. In This & That’s case it simply asks for samples matching the sample type, with no other conditions. So it should be getting given all samples of .sleepAnalysis type, regardless of source app, date range, etc.
The cases where apps might have to be more careful / explicit in processing the fetched data is where there’s things like overlaps from different sources. For example if two separate apps both recorded sleep data for the same sleep session, you can’t just add up the totals of all the samples within that session - you’d end up with roughly double values. You instead need to account for overlaps. (Which is what This & That already does.)
But what we’re seeing is This & That being given no data at all. Which doesn’t fit with any of the expectations
Though that’s the one step that the app’s current debug logging isn’t showing - the raw responses from the HealthKit queries (or at least some summary of). So that’ll have to be the next step I guess - adding logging for that!
Though I suspect it’s still going to tell us that the app is getting given nothing at all. I can’t imagine any way the the processing logic after that point could conclude zero values, if it’s given any data at all. It only adds, never subtracts. But yeah, we’ll have to see to be sure. I’ll get another TestFlight build ready!