Greetings from Australia! I have a new idea for the next app release. I read elsewhere on this forum that you are developing a new app to replace Arc Timeline and there will be major rehaul associated with the new app. Thus, I would like to propose the following:
Utilising Apple’s CloudKit database. This will be specially helpful as we seek to store redundant copies of the user’s data for recovery. Currently I can see that Arc Timeline is using iCloud Drive, but the data is stored under file-and-folder-based structures that contains so many individual files. Usually, it will clutter users’ iCloud Drive and flood their “Recent” interface. I suggest that the app use CloudKit to store the data as a single, non-user-accessible database, similar to how many other apps are currently using. Examples include apps like Rond, Pado, etc.
An iPad app. Yes, I know that this is behind your to-dos, but an iPad app would be extremely useful to review the data thanks to the device’s big screens. I think the ideal way is to use iPhone as the main data recorder and iPad as the viewer of the timeline data. And as far as I know, an iPad app would pave the way for a corresponding Mac app as well as Macs are now ARM-based.
I should not forget to mention that a CloudKit database would mean better, more realiable sync between iPhone and iPad/Mac rather than thousands of individual files stored in iCloud Drive. Hopefully you would consider this idea!
Just take a look at the old posts on this forum. And it seems that you have once moved away from CloudKit due to its inefficient storage management. But the CloudKit has been improved a lot since 2020 (as a developer I can tell that). Is it worth to take a look at the current CloudKit?
I noticed this clutter as well so I turned off iCloud Drive backups and automatic daily exports from the Arc app. The app also says if you have iCloud backup turned on the data is already included in the backup. I don’t see much value in these extra backups. I’d rather think there’s more value in having extra backups to my own Mac.
This is actually how Arc initially did its iCloud backups, and it was a disaster
A key difference between CloudKit and iCloud Drive is that CloudKit must send the data to the remote server immediately, while iCloud Drive saves the data to the local file system then syncs it later, at a time that iOS thinks is most efficient and opportune.
Arc’s database is made up of millions of small objects (eg potentially hundreds or thousands of LocomotionSamples per TimelineItem). Storing those in CloudKit requires a send request per object, and the wifi connection has to be available and reliable at that time. The requests can be batched together, which reduces some of the problem, but CloudKit rate limiting still caused significant problems, as did the requirement that wifi be available and reliable at backup time.
Doing it the iCloud Drive way means that Arc can push the data into the right shape and store it on the local file system quickly and efficiently, regardless of whether a viable wifi connection is present. iOS can then sync those files to the remote server either immediately or some time later, as it sees fit, with that syncing having no impact on Arc’s ability to complete the backups quickly and efficiently.
That distinction also matters in what energy use is counted towards Arc. With CloudKit, all of the syncing to the remote server is energy use that gets counted towards Arc, and Arc gets punished for it. With iCloud Drive, the delayed syncing is iOS’s responsibility and isn’t counted towards Arc’s energy use.
The CloudKit way simply didn’t work for the scale of Arc’s databases, nor within Arc’s strict energy use requirements. The iCloud Drive however way works very effectively! Though as you say, there is a trade off of flooding iCloud Drive’s “Recent” view with noise.
Examples include apps like Rond, Pado, etc.
Those kinds of apps aren’t comparable to Arc. They store orders of magnitude less data - they’re a completely different class of animal.
For an iPad (and Mac) app, the problem unfortunately is similar, in that Arc stores really, really large amounts of data, so moving it around between devices is prohibitively difficult.
An example of this problem in Apple’s own apps is the Health app. If you’ve ever tried to view your data in the Health app on your iPad or a second iPhone you’ll have noticed that it’s usually days or even weeks out of date. It’s so much data that they can’t get it synced fast enough to be useful most of the time.
Arc would have that same problem, but worse. Worse because Arc’s data is far more “mutable”, in that it often gets updated after the initial save. HealthKit’s data on the other hand is mostly “immutable”, in that it can’t be edited after being saved - it can only be deleted. So HeathKit only has to sync initial saves and maybe some infrequent deletes, while Arc would have to sync initial saves as well as frequent updates. More data, and data that also constantly changes.
The potential for getting out of sync would be extreme. Or rather, the possibility of the devices ever actually catching up and finishing syncing would be very slim. If Apple’s Health app can’t do it, with data that never changes after initial save, then Arc’s chances are even slimmer.
That’s not to say it’s impossible! I still have my dreams! There are ways to make it happen. They’re just not easy. So it remains in the basket of things that will get done once the AIs finish building my robot twin, who can get far more work done each day than fleshy old me can
There are ways to make it happen. They’re just not easy. So it remains in the basket of things that will get done once the AIs finish building my robot twin, who can get far more work done each day than fleshy old me can 