Hi and welcome to the October 2024 monthly update of Dawarich!

News​

The biggest news of October for me is that I'm looking for a job! If you have a position for a Ruby on Rails developer with 10+ years of experience in Germany, drop me a line on LinkedIn or email me.

As for Dawarich, there is a pretty big list of changes and improvements in this release. Let's dive into them!

Important features and changes​

The Watcher release​

This feature will be useful for those who want to automate the import process. You can now put your GPX and GeoJSON files to the /tmp/imports/watched/[email protected] directory and Dawarich will automatically import them. This is useful if you have a service that can put files to the directory automatically. The directory is being watched every 60 minutes for new files.

For example, if you want to import a file for the user with the email address "[email protected]", you would put it in the tmp/imports/watched/[email protected] directory. The file will be imported into the database and the user will receive a notification in the app.

For now, only GeoJSON and GPX files are supported.

The up-to-date Photon in Docker​

@rtuszik was kind enough to build a new Docker image with the latest version of the Photon API. The source code with the usage instructions can be found on GitHub.

Set up your backup​

This is a kind reminder for you to set up a backup for your Dawarich instance. You can use the backup tutorial to do this. It's important to keep your data safe and secure.

Added​

• linux/arm/v7 is added to the list of supported architectures to support Raspberry Pi 4 and other ARMv7 devices
• Owntracks' .rec files now can be imported to Dawarich. The import process is the same as for other kinds of files, just select the .rec file and choose "owntracks" as a source.
• User can now select between "Raw" and "Simplified" mode in the map controls. "Simplified" mode will show less points, improving the map performance. "Raw" mode will show all points.
• Importing Immich data on the Imports page now will trigger an attempt to write raw json file with the data from Immich to tmp/imports/immich_raw_data_CURRENT_TIME_USER_EMAIL.json file. This is useful to debug the problem with the import if it fails.
• You can now put your GPX and GeoJSON files to tmp/imports/watched/[email protected] directory and Dawarich will automatically import them. This is useful if you have a service that can put files to the directory automatically. The directory is being watched every 60 minutes for new files.
• To expose the watcher functionality to the user, a new directory /tmp/imports/watched/ was created. Add new volume to the docker-compose.yml file to expose this directory to the host machine.

  ...

  dawarich_app:
    image: freikin/dawarich:latest
    container_name: dawarich_app
    volumes:
      - gem_cache:/usr/local/bundle/gems
      - public:/var/app/public
+     - watched:/var/app/tmp/watched

  ...

  dawarich_sidekiq:
      image: freikin/dawarich:latest
      container_name: dawarich_sidekiq
      volumes:
        - gem_cache:/usr/local/bundle/gems
        - public:/var/app/public
+       - watched:/var/app/tmp/watched

    ...

volumes:
  db_data:
  gem_cache:
  shared_data:
  public:
+ watched:

Changed​

• GET /api/v1/points can now accept optional ?order=asc query parameter to return points in ascending order by timestamp. ?order=desc is still available to return points in descending order by timestamp
• GET /api/v1/points now returns id attribute for each point
• Retries disabled for some background jobs
• Use static version of geocoder library that supports http and https for Photon API host. This is a temporary solution until the change is available in a stable release of geocoder.
• The Map page now by default uses timeframe based on last point tracked instead of the today's points. If there are no points, the map will use the today's timeframe.
• The map on the Map page can no longer be infinitely scrolled horizontally.
• Refactored the stats calculation process to make it more efficient.
• Hostname definition for Sidekiq healtcheck to solve #344. See the diff:

  dawarich_sidekiq:
    image: freikin/dawarich:latest
    container_name: dawarich_sidekiq
    healthcheck:
-     test: [ "CMD-SHELL", "bundle exec sidekiqmon processes | grep $(hostname)" ]
+     test: [ "CMD-SHELL", "bundle exec sidekiqmon processes | grep ${HOSTNAME}" ]

• Renamed directories used by app and sidekiq containers for gems cache to fix #339:

  dawarich_app:
    image: freikin/dawarich:latest
    container_name: dawarich_sidekiq
    volumes:
-     - gem_cache:/usr/local/bundle/gems
+     - gem_cache:/usr/local/bundle/gems_app

...

  dawarich_sidekiq:
    image: freikin/dawarich:latest
    container_name: dawarich_sidekiq
    volumes:
-     - gem_cache:/usr/local/bundle/gems
+     - gem_cache:/usr/local/bundle/gems_sidekiq

Fixed​

• Now you can use both http and https protocols for the Photon API host. You now need to explicitly provide PHOTON_API_USE_HTTPS to be true or false depending on what protocol you want to use. Example is in the docker-compose.yml file.
• For stats, total distance per month might have been not equal to the sum of distances per day. Now it's fixed and values are equal.
• Mobile view of the map looks better now.
• Stats update is now being correctly triggered every 6 hours.
• A bug where "RuntimeError: failed to get urandom" was being raised upon importing attempt on Synology.
• Fixed a bug where Google Takeout import was failing due to unsupported date format with milliseconds in the file.
• Stats distance calculation now correctly calculates the daily distances.
• New app version is now being checked every 6 hours instead of 1 day and the check is being performed in the background.

Removed​

• Owntracks' .json files are no longer supported for import as Owntracks itself does not export to this format anymore.

Conclusion​

That's it for the October 2024 monthly update! I hope you enjoyed the news and changes. If you have any questions or suggestions, feel free to reach out to me on Twitter or Mastodon.

You can also support the project by sponsoring it on Patreon, Ko-Fi or Github Sponsors ✨

Thank you and see you in the next monthly update!

What's going on?​

As you probably already know, Google discontinuing the web version of its Location History / Timeline service by December 1st, 2024. Here are some dry facts:

• Starting December 1st, 2024, the web version of Timeline will no longer be working
• Timeline data will now be stored on the user's device instead
• Users’ last 90 days of location data will be transferred to the first device they sign in with after December 1, but any data older than this will be deleted unless saved
• Users can manually backup Timeline data to Google’s servers for device restoration, but it will not be the default

The transition offered by Google is far from perfect: some users already have reported they have lost years and years of their location history data.

The least you can do is to export your data from Google. You can do it either using Google's Takeout feature or by navigating to Timeline -> Backup in your Google Maps application. Either way, in the end, you will receive an archive with your location history, hopefully, for the whole time you have been tracking it.

What's next?​

That's where Dawarich comes in very handy. Although Google lacks consistency when it comes to the export file format, Dawarich already supports all three (3) kinds of files that you can get from Google.

Google Takeout​

If you requested your data using Google Takeout, you might end up with two different types of files: Semantic Location History, and Records.

Semantic Location History is a bunch of files, split into yearly directories, and named after a year and a month, for example: Semantic Location History/2022/2022_APRIL.json. There will be a file for each month of location history data. This kind of file is the detailed one, and aside from coordinates and timestamps it usually might also contain addresses, suggested by Google, means of transportation, and so on.

To import Google Semantic Location History files, you just need to navigate to the Imports page of Dawarich, select "Semantic Location History" as a Source, select your files (you can select multiple files at once), and hit "Create import". Dawarich will upload the files, save their contents in the database, and then start creating points in the background. After a few seconds (or minutes, if you have lots of files) the data will be imported and you will receive a notification in Dawarich UI.

Records.json is the second type of file you might receive with your Google Takeout Archive. It usually only contains coordinates of geopoints and timestamps of where you've been there, but it also holds a lot more geodata than Semantic Location History files. Because of this, these kinds of files tend to be a lot heavier and should not be imported via the web interface. Instead, you'll have to complete a series of steps.

1. Upload your Records.json to your server, where you host Dawarich
2. Copy the file to the tmp folder of Dawarich:

docker cp Records.json dawarich_app:/var/app/tmp/imports/Records.json

3. ssh into Dawarich container: `docker exec -it dawarich_app sh
4. Run the importing command:

bundle exec rake import:big_file['tmp/imports/Records.json','[email protected]']

The process will take some time, depending on the size of your file, so let it work. Once the file is processed, all points from it will be queued for background import, which will also take some time. You can monitor the stats of the importing process on the Sidekiq page of your Dawarich instance: https://your.host/sidekiq.

Location history on your device​

If you can't retrieve your location history via Google Takeout, you'll probably be able to do that directly on your mobile device. The instructions on how to do that are available on Google Answers.

When you receive the file with your exported data, you can import it using the Imports page of Dawarich. The process is similar to Semantic Location History kind of files, the only thing you have to do differently is select "Google Phone Takeout" as the source of data before uploading it.

In the end​

There, you have it! If you still haven't exported your location history data, do it ASAP, before it's completely gone, and import it to your instance of Dawarich.

Hi and welcome to the September 2024 monthly update of Dawarich!

This is the first monthly update of the project. It will be published in the end(-ish) of each month and will contain the most interesting news and updates of the project.

News​

Believe it or not, in the beginning of the month, there was less than 900 stars on the GitHub repository. But now, there are almost 2000 stars! The post on the 0.12.0 release on Reddit was very popular and it helped to attract more attention to the project. Shortly after that post, someone posted a link to Dawarich repo on HackerNews and lots of people visited the project page from there. The project is growing and it's very exciting!

Important features and changes​

Miles​

Dawarich now supports miles! To switch to miles, provide DISTANCE_UNIT environment variable with value mi in the docker-compose.yml file. Default value is km.

It's recommended to update your stats manually after changing the DISTANCE_UNIT environment variable. You can do this by clicking the "Update stats" button on the Stats page.

⚠️IMPORTANT⚠️: All settings are still should be provided in meters. All calculations though will be converted to feets and miles if DISTANCE_UNIT is set to mi.

  dawarich_app:
    image: freikin/dawarich:latest
    container_name: dawarich_app
    environment:
      ...
+     DISTANCE_UNIT: "mi"
  dawarich_sidekiq:
    image: freikin/dawarich:latest
    container_name: dawarich_sidekiq
    environment:
      ...
+     DISTANCE_UNIT: "mi"

Default time range on the map​

The default time range on the map is now 1 day instead of 1 month. It will help you with performance issues if you have a lot of points in the database.

The GPX and GeoJSON export release​

⚠️ BREAKING CHANGES: ⚠️

Default exporting format is now GeoJSON instead of Owntracks-like JSON. This will allow you to use the exported data in other applications that support GeoJSON format. It's also important to highlight, that GeoJSON format does not describe a way to store any time-related data. Dawarich relies on the timestamp field in the GeoJSON format to determine the time of the point. The value of the timestamp field should be a Unix timestamp in seconds. If you import GeoJSON data that does not have a timestamp field, the point will not be imported.

Example of a valid point in GeoJSON format:

{
  "type": "Feature",
  "geometry": {
    "type": "Point",
    "coordinates": [13.350110811262352, 52.51450815]
  },
  "properties": {
    "timestamp": 1725310036
  }
}

Changelog​

Since I already posted about the 0.12.0 release, I will not go into details about it. Instead, I'll tell you about the changes that were made after that release.

Since Dawarich is being actively developed, there are always lots of moving parts. I encourage you to check the releases page regularely to see all the changes that were made. All breaking changes and ways to mitigate them are always described in the release notes.

Added​

• GET /api/v1/health endpoint to check the health of the application with swagger docs
• PATCH /api/v1/settings endpoint to update user settings with swagger docs
• GET /api/v1/settings endpoint to get user settings with swagger docs
• GET /api/v1/points response now will include X-Total-Pages and X-Current-Page headers to make it easier to work with the endpoint
• "Slim" version of GET /api/v1/points: pass optional param ?slim=true to it and it will return only latitude, longitude and timestamp
• 7 new tile layers to choose from. Now you can select the tile layer that suits you the best. You can find the list of available tile layers in the map controls in the top right corner of the map under the layers icon
• The Pages point now shows total number of points found for provided date range
• Links to view import points on the map and on the Points page on the Imports page
• Support for miles. To switch to miles, provide DISTANCE_UNIT environment variable with value mi in the docker-compose.yml file. Default value is km
• GeoJSON format is now available for exporting data
• GPX format is now available for exporting data
• Importing GeoJSON is now available
• Missing page and per_page query parameters to the GET /api/v1/points endpoint swagger docs
• Resource limits to docke-compose.yml file to prevent server overload. Feel free to adjust the limits to your needs

deploy:
  resources:
    limits:
      cpus: '0.50'    # Limit CPU usage to 50% of one core
      memory: '2G'    # Limit memory usage to 2GB

Changed​

• GET /api/v1/points will no longer return raw_data attribute for each point as it's a bit too much
• GPX export now has time and elevation elements for each point
• Default value for RAILS_MAX_THREADS was changed to 10
• Visit suggestions background job was moved to its own low priority queue to prevent it from blocking other jobs.
• Default time range on the map is now 1 day instead of 1 month. It will help you with performance issues if you have a lot of points in the database
• The Points page now have number of points found for provided date range
• Ruby version updated to 3.3.4
• Visits suggestion process now will try to merge consecutive visits to the same place into one visit
• Default exporting format is now GeoJSON instead of Owntracks-like JSON. This will allow you to use the exported data in other applications that support GeoJSON format
• A notification about an existing import with the same name will now show the import name
• Export file now also will contain raw_data field for each point. This field contains the original data that was imported to the application
• Map settings moved to the map itself and are available in the top right corner of the map under the gear icon

Fixed​

• Deleting points from the Points page now preserves start_at and end_at values for the routes
• Visits map now being rendered correctly in the Visits page
• Fixed issue with timezones for negative UTC offsets
• Point page is no longer reloads losing provided timestamps when searching for points on Points page
• Optimize order of the dockerfiles to leverage layer caching by @JoeyEamigh
• Add support for alternate postgres ports and db names in docker by @JoeyEamigh
• Creating exports directory if it doesn't exist by @tetebueno
• Link to Visits page in notification informing about new visit suggestion
• The Imports page now loading faster
• Fixed a bug preventing the application from starting, when there is no users in the database but a data migration tries to update one
• Fixed a bug where the confirmation alert was shown more than once when deleting a point
• Importing geodata from Immich will now not throw an error in the end of the process

Conclusion​

Phew, that was a lot of changes! With every release it always feels like not much was added or changed, but when you look back at the changelog, it's always a surprise how much was done in a month.

I'd also like to thank everyone who contributed to the project in any way, whether it was a bug report, a feature request, a pull request, or just a kind word. You are the ones who make this project better every day. Thank you!

If you're feeling generous, you can support the project by sponsoring it on Patreon, Ko-Fi or Github Sponsors ✨

Thank you and see you in the next monthly update! More exciting stuff is coming soon! 🚀

Okay, this is the very first blog post on Dawarich blog. If you want to add a tutorial or some docs, feel free to open a pull request to https://github.com/dawarich-app/site. Cheers! 🎉