Hi and welcome to the November 2024 monthly update of Dawarich!

November was a good month for Dawarich. I've added a lot of new features and fixed a some bugs, and, among other things, I've added a new Patreon tier. It's called "Hosted Dawarich" and basically offers my service of hosting and updating Dawarich for you. Take a look at the Patreon page for more details and feel free to ask me any questions.

Let's dive into the details!

Important features and changes​

The Trips release​

You can now create, edit and delete trips. To create a trip, click on the "New Trip" button on the Trips page. Provide a name, date and time for start and end of the trip. You can add your own notes to the trip as well.

If you have points tracked during provided timeframe, they will be automatically added to the trip and will be shown on the trip map.

Also, if you have Immich integrated, you will see photos from the trip on the trip page, along with a link to look at them on Immich.

The Immich Photos release​

With this release, Dawarich can now show photos from your Immich instance on the map.

To enable this feature, you need to provide your Immich instance URL and API key in the Settings page. Then you need to enable "Photos" layer on the map (top right corner).

An important note to add here is that photos are heavy and hence generate a lot of traffic. The response from Immich for specific dates is being cached in Redis for 1 day, and that may lead to Redis taking a lot more space than previously. But since the cache is being expired after 24 hours, you'll get your space back pretty soon.

The other thing worth mentioning is how Dawarich gets data from Immich. It goes like this:

1. When you click on the "Photos" layer, Dawarich will make a request to GET /api/v1/photos endpoint to get photos for the selected timeframe.
2. This endpoint will make a request to POST /search/metadata endpoint of your Immich instance to get photos for the selected timeframe.
3. The response from Immich is being cached in Redis for 1 day.
4. Dawarich's frontend will make a request to GET /api/v1/photos/:id/thumbnail.jpg endpoint to get photo thumbnail from Immich. The number of requests to this endpoint will depend on how many photos you have in the selected timeframe.
5. For each photo, Dawarich's frontend will make a request to GET /api/v1/photos/:id/thumbnail.jpg endpoint to get photo thumbnail from Immich. This thumbnail request is also cached in Redis for 1 day.

The Websockets release​

This one is more about QoL, making your UI more alive. Notifications, new points on the map, imports now won't require page reloading to see an update, they will just be on your page dynamically.

Scratch map release​

Yeah, you can now enable the "Scratch Map" layer on the map. Have a look:

Changelog​

Added​

• The Trips feature.
• If you have provided your Immich instance URL and API key, the map will now show photos from your Immich instance when Photos layer is enabled.
• GET /api/v1/photos endpoint added to get photos from Immich.
• GET /api/v1/photos/:id/thumbnail.jpg endpoint added to get photo thumbnail from Immich.
• Admins can now see all users in the system on the Users page. The path is /settings/users.
• New notifications are now being indicated with a blue-ish dot in the top right corner of the screen. Hovering over the bell icon will show you last 10 notifications.
• New points on the map will now be shown in real-time. No need to reload the map to see new points.
• User can now enable or disable Live Mode in the map controls. When Live Mode is enabled, the map will automatically move to the new points as they are being added to the map.
• Scratch map. You can enable it in the map controls. The scratch map highlight countries you've visited. The scratch map is working properly only if you have your points reverse geocoded.
• GET /api/v1/countries/borders endpoint to get countries for scratch map feature
• Dawarich now can export metrics to Prometheus. You can find the metrics at your.host:9394/metrics endpoint. The metrics are being exported in the Prometheus format and can be scraped by Prometheus server. To enable exporting, set the PROMETHEUS_EXPORTER_ENABLED env var in your docker-compose.yml to true. Example:

  dawarich_app:
    image: freikin/dawarich:latest
    container_name: dawarich_app
    environment:
      ...
+     PROMETHEUS_EXPORTER_ENABLED: "true"

Fixed​

• Retrieving photos from Immich now using takenAfter and takenBefore instead of createdAfter and createdBefore. With createdAfter and createdBefore Immich was returning no items some years.
• Retrieving photos from Immich now correctly handles cases when Immich returns no items. It also logs the response from Immich for debugging purposes.
• Exported files will now always have an extension when downloaded. Previously, the extension was missing in case of GPX export.
• Deleting and sorting points on the Points page will now preserve filtering and sorting params when points are deleted or sorted. Previously, the page was being reloaded and filtering and sorting params were lost.
• Exported GPX file now being correctly recognized as valid by Garmin Connect, Adobe Lightroom and (probably) other services. Previously, the exported GPX file was not being recognized as valid by these services.
• Speed is now being recorded into points when a GPX file is being imported. Previously, the speed was not being recorded.
• GeoJSON file from GPSLogger now can be imported to Dawarich. Previously, the import was failing due to incorrect parsing of the file.

Changed​

• Maps are now not so rough on the edges.
• Rate limit for the Photon API is now 1 request per second. If you host your own Photon API instance, reverse geocoding requests will not be limited.
• Requests to the Photon API are now have User-Agent header set to "Dawarich #{APP_VERSION} (https://dawarich.app)"
• Default number of Puma workers is now 2 instead of 1. This should improve the performance of the application. If you have a lot of users, you might want to increase the number of workers. You can do this by setting the WEB_CONCURRENCY env var in your docker-compose.yml file. Example:

  dawarich_app:
    image: freikin/dawarich:latest
    container_name: dawarich_app
    environment:
      ...
      WEB_CONCURRENCY: "2"

• PROMETHEUS_EXPORTER_HOST and PROMETHEUS_EXPORTER_PORT env vars were added to the docker-compose.yml file to allow you to set the host and port for the Prometheus exporter. They should be added to both dawarich_app and dawarich_sidekiq services Example:

  dawarich_app:
    image: freikin/dawarich:latest
    container_name: dawarich_app
    environment:
      ...
      PROMETHEUS_EXPORTER_ENABLED: "true"
+     PROMETHEUS_EXPORTER_HOST: 0.0.0.0
+     PROMETHEUS_EXPORTER_PORT: "9394"

  dawarich_sidekiq:
    image: freikin/dawarich:latest
    container_name: dawarich_sidekiq
    environment:
      ...
      PROMETHEUS_EXPORTER_ENABLED: "true"
+     PROMETHEUS_EXPORTER_HOST: dawarich_app
+     PROMETHEUS_EXPORTER_PORT: "9394"

• Dawarich now uses POST /api/search/metadata endpoint to get geodata from Immich.
• Admins can now provide custom password for new users and update passwords for existing users on the Users page.
• The bin/dev file will no longer run bin/rails tailwindcss:watch command. It's useful only for development and doesn't really make sense to run it in production.
• The Vists suggestion job is disabled. It will be re-enabled in the future with a new approach to the visit suggestion process.

Conclusion​

What a month huh? Feel free to reach out to me on X, Mastodon or Discord server if you have any questions or suggestions.

You can also support the project by sponsoring it on Patreon, Ko-Fi or Github Sponsors ✨

Thank you and see you in the next monthly update!

Hi and welcome to the October 2024 monthly update of Dawarich!

News​

The biggest news of October for me is that I'm looking for a job! If you have a position for a Ruby on Rails developer with 10+ years of experience in Germany, drop me a line on LinkedIn or email me.

As for Dawarich, there is a pretty big list of changes and improvements in this release. Let's dive into them!

Important features and changes​

The Watcher release​

This feature will be useful for those who want to automate the import process. You can now put your GPX and GeoJSON files to the /tmp/imports/watched/[email protected] directory and Dawarich will automatically import them. This is useful if you have a service that can put files to the directory automatically. The directory is being watched every 60 minutes for new files.

For example, if you want to import a file for the user with the email address "[email protected]", you would put it in the tmp/imports/watched/[email protected] directory. The file will be imported into the database and the user will receive a notification in the app.

For now, only GeoJSON and GPX files are supported.

The up-to-date Photon in Docker​

@rtuszik was kind enough to build a new Docker image with the latest version of the Photon API. The source code with the usage instructions can be found on GitHub.

Set up your backup​

This is a kind reminder for you to set up a backup for your Dawarich instance. You can use the backup tutorial to do this. It's important to keep your data safe and secure.

Added​

• linux/arm/v7 is added to the list of supported architectures to support Raspberry Pi 4 and other ARMv7 devices
• Owntracks' .rec files now can be imported to Dawarich. The import process is the same as for other kinds of files, just select the .rec file and choose "owntracks" as a source.
• User can now select between "Raw" and "Simplified" mode in the map controls. "Simplified" mode will show less points, improving the map performance. "Raw" mode will show all points.
• Importing Immich data on the Imports page now will trigger an attempt to write raw json file with the data from Immich to tmp/imports/immich_raw_data_CURRENT_TIME_USER_EMAIL.json file. This is useful to debug the problem with the import if it fails.
• You can now put your GPX and GeoJSON files to tmp/imports/watched/[email protected] directory and Dawarich will automatically import them. This is useful if you have a service that can put files to the directory automatically. The directory is being watched every 60 minutes for new files.
• To expose the watcher functionality to the user, a new directory /tmp/imports/watched/ was created. Add new volume to the docker-compose.yml file to expose this directory to the host machine.

  ...

  dawarich_app:
    image: freikin/dawarich:latest
    container_name: dawarich_app
    volumes:
      - gem_cache:/usr/local/bundle/gems
      - public:/var/app/public
+     - watched:/var/app/tmp/watched

  ...

  dawarich_sidekiq:
      image: freikin/dawarich:latest
      container_name: dawarich_sidekiq
      volumes:
        - gem_cache:/usr/local/bundle/gems
        - public:/var/app/public
+       - watched:/var/app/tmp/watched

    ...

volumes:
  db_data:
  gem_cache:
  shared_data:
  public:
+ watched:

Changed​

• GET /api/v1/points can now accept optional ?order=asc query parameter to return points in ascending order by timestamp. ?order=desc is still available to return points in descending order by timestamp
• GET /api/v1/points now returns id attribute for each point
• Retries disabled for some background jobs
• Use static version of geocoder library that supports http and https for Photon API host. This is a temporary solution until the change is available in a stable release of geocoder.
• The Map page now by default uses timeframe based on last point tracked instead of the today's points. If there are no points, the map will use the today's timeframe.
• The map on the Map page can no longer be infinitely scrolled horizontally.
• Refactored the stats calculation process to make it more efficient.
• Hostname definition for Sidekiq healtcheck to solve #344. See the diff:

  dawarich_sidekiq:
    image: freikin/dawarich:latest
    container_name: dawarich_sidekiq
    healthcheck:
-     test: [ "CMD-SHELL", "bundle exec sidekiqmon processes | grep $(hostname)" ]
+     test: [ "CMD-SHELL", "bundle exec sidekiqmon processes | grep ${HOSTNAME}" ]

• Renamed directories used by app and sidekiq containers for gems cache to fix #339:

  dawarich_app:
    image: freikin/dawarich:latest
    container_name: dawarich_sidekiq
    volumes:
-     - gem_cache:/usr/local/bundle/gems
+     - gem_cache:/usr/local/bundle/gems_app

...

  dawarich_sidekiq:
    image: freikin/dawarich:latest
    container_name: dawarich_sidekiq
    volumes:
-     - gem_cache:/usr/local/bundle/gems
+     - gem_cache:/usr/local/bundle/gems_sidekiq

Fixed​

• Now you can use both http and https protocols for the Photon API host. You now need to explicitly provide PHOTON_API_USE_HTTPS to be true or false depending on what protocol you want to use. Example is in the docker-compose.yml file.
• For stats, total distance per month might have been not equal to the sum of distances per day. Now it's fixed and values are equal.
• Mobile view of the map looks better now.
• Stats update is now being correctly triggered every 6 hours.
• A bug where "RuntimeError: failed to get urandom" was being raised upon importing attempt on Synology.
• Fixed a bug where Google Takeout import was failing due to unsupported date format with milliseconds in the file.
• Stats distance calculation now correctly calculates the daily distances.
• New app version is now being checked every 6 hours instead of 1 day and the check is being performed in the background.

Removed​

• Owntracks' .json files are no longer supported for import as Owntracks itself does not export to this format anymore.

Conclusion​

That's it for the October 2024 monthly update! I hope you enjoyed the news and changes. If you have any questions or suggestions, feel free to reach out to me on Twitter or Mastodon.

You can also support the project by sponsoring it on Patreon, Ko-Fi or Github Sponsors ✨

Thank you and see you in the next monthly update!

What's going on?​

As you probably already know, Google discontinuing the web version of its Location History / Timeline service by December 1st, 2024. Here are some dry facts:

• Starting December 1st, 2024, the web version of Timeline will no longer be working
• Timeline data will now be stored on the user's device instead
• Users’ last 90 days of location data will be transferred to the first device they sign in with after December 1, but any data older than this will be deleted unless saved
• Users can manually backup Timeline data to Google’s servers for device restoration, but it will not be the default

The transition offered by Google is far from perfect: some users already have reported they have lost years and years of their location history data.

The least you can do is to export your data from Google. You can do it either using Google's Takeout feature or by navigating to Timeline -> Backup in your Google Maps application. Either way, in the end, you will receive an archive with your location history, hopefully, for the whole time you have been tracking it.

What's next?​

That's where Dawarich comes in very handy. Although Google lacks consistency when it comes to the export file format, Dawarich already supports all three (3) kinds of files that you can get from Google.

Google Takeout​

If you requested your data using Google Takeout, you might end up with two different types of files: Semantic Location History, and Records.

Semantic Location History is a bunch of files, split into yearly directories, and named after a year and a month, for example: Semantic Location History/2022/2022_APRIL.json. There will be a file for each month of location history data. This kind of file is the detailed one, and aside from coordinates and timestamps it usually might also contain addresses, suggested by Google, means of transportation, and so on.

To import Google Semantic Location History files, you just need to navigate to the Imports page of Dawarich, select "Semantic Location History" as a Source, select your files (you can select multiple files at once), and hit "Create import". Dawarich will upload the files, save their contents in the database, and then start creating points in the background. After a few seconds (or minutes, if you have lots of files) the data will be imported and you will receive a notification in Dawarich UI.

Records.json is the second type of file you might receive with your Google Takeout Archive. It usually only contains coordinates of geopoints and timestamps of where you've been there, but it also holds a lot more geodata than Semantic Location History files. Because of this, these kinds of files tend to be a lot heavier and should not be imported via the web interface. Instead, you'll have to complete a series of steps.

1. Upload your Records.json to your server, where you host Dawarich
2. Copy the file to the tmp folder of Dawarich:

docker cp Records.json dawarich_app:/var/app/tmp/imports/Records.json

3. ssh into Dawarich container: `docker exec -it dawarich_app sh
4. Run the importing command:

bundle exec rake import:big_file['tmp/imports/Records.json','[email protected]']

The process will take some time, depending on the size of your file, so let it work. Once the file is processed, all points from it will be queued for background import, which will also take some time. You can monitor the stats of the importing process on the Sidekiq page of your Dawarich instance: https://your.host/sidekiq.

Location history on your device​

If you can't retrieve your location history via Google Takeout, you'll probably be able to do that directly on your mobile device. The instructions on how to do that are available on Google Answers.

When you receive the file with your exported data, you can import it using the Imports page of Dawarich. The process is similar to Semantic Location History kind of files, the only thing you have to do differently is select "Google Phone Takeout" as the source of data before uploading it.

In the end​

There, you have it! If you still haven't exported your location history data, do it ASAP, before it's completely gone, and import it to your instance of Dawarich.

Hi and welcome to the September 2024 monthly update of Dawarich!

This is the first monthly update of the project. It will be published in the end(-ish) of each month and will contain the most interesting news and updates of the project.

News​

Believe it or not, in the beginning of the month, there was less than 900 stars on the GitHub repository. But now, there are almost 2000 stars! The post on the 0.12.0 release on Reddit was very popular and it helped to attract more attention to the project. Shortly after that post, someone posted a link to Dawarich repo on HackerNews and lots of people visited the project page from there. The project is growing and it's very exciting!

Important features and changes​

Miles​

Dawarich now supports miles! To switch to miles, provide DISTANCE_UNIT environment variable with value mi in the docker-compose.yml file. Default value is km.

It's recommended to update your stats manually after changing the DISTANCE_UNIT environment variable. You can do this by clicking the "Update stats" button on the Stats page.

⚠️IMPORTANT⚠️: All settings are still should be provided in meters. All calculations though will be converted to feets and miles if DISTANCE_UNIT is set to mi.

  dawarich_app:
    image: freikin/dawarich:latest
    container_name: dawarich_app
    environment:
      ...
+     DISTANCE_UNIT: "mi"
  dawarich_sidekiq:
    image: freikin/dawarich:latest
    container_name: dawarich_sidekiq
    environment:
      ...
+     DISTANCE_UNIT: "mi"

Default time range on the map​

The default time range on the map is now 1 day instead of 1 month. It will help you with performance issues if you have a lot of points in the database.

The GPX and GeoJSON export release​

⚠️ BREAKING CHANGES: ⚠️

Default exporting format is now GeoJSON instead of Owntracks-like JSON. This will allow you to use the exported data in other applications that support GeoJSON format. It's also important to highlight, that GeoJSON format does not describe a way to store any time-related data. Dawarich relies on the timestamp field in the GeoJSON format to determine the time of the point. The value of the timestamp field should be a Unix timestamp in seconds. If you import GeoJSON data that does not have a timestamp field, the point will not be imported.

Example of a valid point in GeoJSON format:

{
  "type": "Feature",
  "geometry": {
    "type": "Point",
    "coordinates": [13.350110811262352, 52.51450815]
  },
  "properties": {
    "timestamp": 1725310036
  }
}

Changelog​

Since I already posted about the 0.12.0 release, I will not go into details about it. Instead, I'll tell you about the changes that were made after that release.

Since Dawarich is being actively developed, there are always lots of moving parts. I encourage you to check the releases page regularely to see all the changes that were made. All breaking changes and ways to mitigate them are always described in the release notes.

Added​

• GET /api/v1/health endpoint to check the health of the application with swagger docs
• PATCH /api/v1/settings endpoint to update user settings with swagger docs
• GET /api/v1/settings endpoint to get user settings with swagger docs
• GET /api/v1/points response now will include X-Total-Pages and X-Current-Page headers to make it easier to work with the endpoint
• "Slim" version of GET /api/v1/points: pass optional param ?slim=true to it and it will return only latitude, longitude and timestamp
• 7 new tile layers to choose from. Now you can select the tile layer that suits you the best. You can find the list of available tile layers in the map controls in the top right corner of the map under the layers icon
• The Pages point now shows total number of points found for provided date range
• Links to view import points on the map and on the Points page on the Imports page
• Support for miles. To switch to miles, provide DISTANCE_UNIT environment variable with value mi in the docker-compose.yml file. Default value is km
• GeoJSON format is now available for exporting data
• GPX format is now available for exporting data
• Importing GeoJSON is now available
• Missing page and per_page query parameters to the GET /api/v1/points endpoint swagger docs
• Resource limits to docke-compose.yml file to prevent server overload. Feel free to adjust the limits to your needs

deploy:
  resources:
    limits:
      cpus: '0.50'    # Limit CPU usage to 50% of one core
      memory: '2G'    # Limit memory usage to 2GB

Changed​

• GET /api/v1/points will no longer return raw_data attribute for each point as it's a bit too much
• GPX export now has time and elevation elements for each point
• Default value for RAILS_MAX_THREADS was changed to 10
• Visit suggestions background job was moved to its own low priority queue to prevent it from blocking other jobs.
• Default time range on the map is now 1 day instead of 1 month. It will help you with performance issues if you have a lot of points in the database
• The Points page now have number of points found for provided date range
• Ruby version updated to 3.3.4
• Visits suggestion process now will try to merge consecutive visits to the same place into one visit
• Default exporting format is now GeoJSON instead of Owntracks-like JSON. This will allow you to use the exported data in other applications that support GeoJSON format
• A notification about an existing import with the same name will now show the import name
• Export file now also will contain raw_data field for each point. This field contains the original data that was imported to the application
• Map settings moved to the map itself and are available in the top right corner of the map under the gear icon

Fixed​

• Deleting points from the Points page now preserves start_at and end_at values for the routes
• Visits map now being rendered correctly in the Visits page
• Fixed issue with timezones for negative UTC offsets
• Point page is no longer reloads losing provided timestamps when searching for points on Points page
• Optimize order of the dockerfiles to leverage layer caching by @JoeyEamigh
• Add support for alternate postgres ports and db names in docker by @JoeyEamigh
• Creating exports directory if it doesn't exist by @tetebueno
• Link to Visits page in notification informing about new visit suggestion
• The Imports page now loading faster
• Fixed a bug preventing the application from starting, when there is no users in the database but a data migration tries to update one
• Fixed a bug where the confirmation alert was shown more than once when deleting a point
• Importing geodata from Immich will now not throw an error in the end of the process

Conclusion​

Phew, that was a lot of changes! With every release it always feels like not much was added or changed, but when you look back at the changelog, it's always a surprise how much was done in a month.

I'd also like to thank everyone who contributed to the project in any way, whether it was a bug report, a feature request, a pull request, or just a kind word. You are the ones who make this project better every day. Thank you!

If you're feeling generous, you can support the project by sponsoring it on Patreon, Ko-Fi or Github Sponsors ✨

Thank you and see you in the next monthly update! More exciting stuff is coming soon! 🚀

Okay, this is the very first blog post on Dawarich blog. If you want to add a tutorial or some docs, feel free to open a pull request to https://github.com/dawarich-app/site. Cheers! 🎉