Changelog
0.17.1
Changelog
⛰️ Features
- (api) Bump express-typed-router and add global validation failure logging
🐛 Bug Fixes
- (api) Make optional query params optional in route schemas #697 thanks @MFA-G
- (lastfm) Source missing options when initing client #699
- (webscrobbler) Loosen scrobble schema for real-world usage #700
- (No Category) Fix route mounting for /api/callback #699
📚 Documentation
- (No Category) Update BrainzMash language
0.17.0
What’s New?
Preview UI Feature Parity
The new UI has feature parity with the existing UI, plus so many more improvements. :tada:
0.17.0 will be the last release before the new UI replaces the existing one. If you have design feedback or things aren’t working as expected please report them in this issue. Also would be nice if you left a comment letting me know that everything is working as expected and you like the new design! Feedback either way is good.
Listenbrainz User-Agent Changes
Listenbrainz has updated their User-Agent requirements to require a point of contact, similar to how Musicbrainz works.
The 0.16.5 release that addressed Listenbrainz User-Agent updates is no longer sufficient. Use the updated Listenbrainz for multi-scrobbler to update your configs to include a contact.
Use the 0.17.2 release. Your listenbrainz configuration does not need to be modified.
Improved Source Play Processing
The base Source class has been restructured to use a queue workflow, similar to the base Client class, that enables new functionality and improves all existing Play processing. Some highlights:
- Backlog play processing no longer blocks Source startup
- Ingress based Sources can now send batch scrobbles and get an immediate response without needing to wait for all Plays to process
- Source plays that fail to process can now be retried (same way as Client plays)
- Sources can process existing, queued Plays even if polling has failed
Optimized Existing Play Checks
When polling Listening History based Sources MS parses all returned history, every time, to look for changes in the list.
Previously, if you had preCompare transforms these needed to run for all history before MS could check if each Play already existed.
Now, MS stores a hash of the original data, before transforms, and uses this to check for existing Plays before transforming. This optimization can potentially save hundreds of API calls to musicbrainz and wasted CPU cycles transforming Plays that would then be thrown away immediately.
Play Events
The events shown on the Timeline for a Play in the Preview UI were previously hacked together from many different locations in a Play’s data.
This data has now been restructured as a first-class data structure persisted to the database. What this means for you is that the Timeline will now show a more complete, user-friendly list of events and that future event types are trivial to create.
Api Validation and Docs
Thanks to @Mini-Sylar ’s excellent express-typed-router library MS now has fully-typed api routes.
Ingress-based Sources like Webscrobbler and Listenbrainz/Last.fm Endpoints get exhaustive request body/query validation with same kind of descriptive errors configuration reports.
And all api endpoints, including ingress, are now self-documenting and added to the main docs website. If you’ve ever wanted to build on top of the data MS creates, you now can!
Note: Other than ingress-based endpoints, the APIs are documented but not fully stable yet. They will be stabilized once the preview ui is released.
BrainzMash Musicbrainz Server
For users who are using Musicbrainz for enhancing their scrobbles: a new, public Musicbrainz mirror is detailed in the docs under the BrainzMash Musicbrainz Server expandable section.
BrainzMash is a community-run pool of self-hosted, read-only hybrid Musicbrainz/Lidarr metadata servers. Requests are made to one, public endpoint that load-balances traffic out to all participating, private servers.
Supplementing your config with this additional Musicbrainz server can increase your transformer speed and throughput.
Security Documentation
A Security section has been added to the docs that details the risks of public access to your Multi-Scrobbler and what you can do to secure it.
Thank to @gnosticJade for the Caddy example and corrections on protected routes!
Other Changelog Highlights
⛰️ Features
- (lastfm) Use p-retry for request retries
- (librefm) Implement rate limiting
- (No Category) Use better rate limiting implementation for musicbrainz
🐛 Bug Fixes
- (atproto) Guard against null rate limit header parsing #689
- (atproto) Detect dupe rkey in batch before persist #689
- (atproto) Implement proper error handling for xrpc calls #689
- (source) Check for player repeat using closed listening session fallback #691
- (tests) Improve network mocking to compensate for libraries with frozen imports
- (ui) Log settings route schema and middleware #693
- (transform): Catch and log errors during console diff generation #684
📚 Documentation
- (No Category) Add security guidance
- (No Category) Adjust caddy snippet with better guidance
- (No Category) Cleanup atproto admonitions
- (No Category) Add BrainzMash musicbrainz guidance
- (No Category) Add openapi docs
🚜 Refactor
- (lastfm) Simplify api error reporting
🧪 Testing
- (lastfm) Test rate limiting and error handling
0.16.5
This is a small release due to needing to get Listenbrainz’s new request requirements out to stable! A larger release with new UI functionality will be coming soon.
What’s New?
Listenbrainz User-Agent
A recent botnet attack has forced Listenbrainz to require all traffic to include a User-Agent and implement retry backoffs.
Multi-Scrobbler already implements backoffs but did not have a proper User-Agent in place. This release introduces a UA for all requests that should restore Listenbrainz connectivity.
teal.fm production lexicon
teal.fm has moved their lexicon to the production namespace and so this release of multi-scrobbler updates the repository records to use the new namespace.
Other Changes
- @halkeye fixed the code for generating “Edit this page” links in the docs :tada:
0.16.4
The jumble of patches released this week in the aftermath of 0.16.0 were indicative of a poor release.
I take pride in making sure Multi-scrobbler is not one of those projects you are hesitant to upgrade because you don’t know if “this will be the one that breaks unexpectedly”. 0.16.0 was not up to that quality and for that, I apologize.
This release should hopefully make things right.
What’s New?
ENV Testing and Stability
The ENV improvements made in 0.16.0 are here to stay but to address the quality issues mentioned above this release focused on building a proper test suite to validate and test all ENV keys with real, valid data:
- tests were added for all sources/clients that
- generate all ENVs for each source/client with real data derived from the ENV schema shape, using zocker
- assign these ENVs to the actual environment and then build the actual source/client, verifying no config errors and that exactly one source/client of that type exist
This now puts ENV testing on the same level of testing as file/aio config and should catch any errors, like those fixed in 0.16.1 - 0.16.3, before they get to the release stage.
Full Changelog
⛰️ Features
- (No Category) Relax configureAs requirements for AIO configs
- (No Category) Strongly-type internal events #666
- (No Category) Add ENV config test suite
- (No Category) Update json schema to draft-2020-12
- (No Category) Normalize domain:port url input
- (No Category) Better validation for normalized config urls
🐛 Bug Fixes
- (discord) Fix discord ENV parsing validation
- (No Category) Check for ENV prefix with startsWith instead of includes #668
- (No Category) Narrow env config key collection to include prefix underscore #668
- (No Category) Hand-rolled env data fixtures for some edge cases
- (No Category) Hand-roll applemusic env fixture data
📚 Documentation
- (No Category) Add more guidance for File config example/editor
- (No Category) Hand-roll discord ENV updates
0.16.3
Changelog
🐛 Bug Fixes
- Fix incorrect prefix not being used for env key collection #668
- (apple music) include library-songs in recent tracks #667
📚 Documentation
- Improve scenario guidance for lfm/lz endpoints
0.16.2
Changelog
🐛 Bug Fixes
- (source): Fix missing id/uid on cached recent plays #662 #665
0.16.1
Changelog
🐛 Bug Fixes
- (No Category) Add missing
configureAsfor ENV parsed components #661
0.16.0
What’s New?
Reduced Memory Usage
Moving to Zod in 0.15.0 provided a big reduction in memory footprint that has been further improved in this release by moving individual Source/Client validation to be dynamically loaded. This means MS will not load validation/types for Source/Clients you do not use.
Users with a minimal number of Sources/Clients should see the largest impact from this.
Benchmarks for MS with no Sources/Clients
Startup
| Version | Runtime | Validation Context | % Memory Usage Change |
|---|---|---|---|
| 0.14.2 | tsx | ts gen + ajv | - |
| 0.15.0 | node | zod + eager validation | -13% |
| 0.16.0 | node | zod + dynamic validation | -14% |
Stabilized (after 4 minutes of running)
| Version | Runtime | Validation Context | % Memory Usage Change |
|---|---|---|---|
| 0.14.2 | tsx | ts gen + ajv | - |
| 0.15.0 | node | zod + eager validation | -22% |
| 0.16.0 | node | zod + dynamic validation | -27% |
ENV Config UX Improvements
The move to Zod in 0.15.0 also opened the door for big improvements to ENV Config experience for the end-user:
Self-Documenting
ENV variables now share, or inherit, schema from file/aio config so ENV descriptions match those found elsewhere. The ENV config for Source/Clients in the docs is now self-documenting (generated from schema) to prevent typos and make common ENV easier to extend.
Improved Validation
ENV validation at startup is now more straightforward and useful. Instead of mapping variables to file (KOITO_TOKEN => data.token) and then printing an error
✖ Invalid input: expected string, received undefined → at data.tokenVariables are now directly validated to give you a clearer idea of what is missing or wrong
✖ Invalid input: expected string, received undefined → at KOITO_TOKENSource/Client ID Enforcement
To further improve DX/UX the ID requirements first introduced in 0.14.0 are now mandatory.
Previously, MS would log a warning and use your Source/Client name as the ID, as noted in the 0.14.0 upgrade guide. Now, it reports a configuration error if no ID is present.
To migrate, fully complete the Configuring IDs section from the 0.14.0 migration guide.
Other Changelog Highlights
⛰️ Features
- (lfm endpoint) Implement better api emulation #656
- (source) Add Ingress SOT type
- (now playing) Buffer valid updates for endpoint sources #659
- (ui) Improve UX for theme switcher
- (ui) Implement basic settings page
- (No Category) Improve location wording for config errors
🐛 Bug Fixes
- (source): Fix player cleanup for ingress sources #652 #591
📚 Documentation
- (No Category) Use italicize-bold text style for required env properties instead of separate column
- (No Category) Update env descriptions for id, name, and enable
- (No Category) Update now playing for clients
- (No Category) Fix config.json example
- (No Category) Add lastfm endpoint alternative guidance
- (No Category) Update configuration types language
0.15.0
What’s New?
Apple Music Source
Thanks to the efforts of @Exerra , Apple Music is now implemented as a Source for Multi-Scrobbler!
Exerra, the primary developer of node-musickit-api, discovered that the API access required for MS to monitor listening activity only needs credentials that can be acquired from the browser of a logged in user. Critically, this bypasses the long-standing assumption that a paid apple developer account was required to access these APIs.
Exerra also contributed the entire Apple Music Source implementation to Multi-Scrobbler :crown: If you end up using this Source please consider sponsoring them as a thank you for all this hard work!
Improved Subsonic Playback Reporting
The OpenSubsonic API recently finalized a new spec for Playback Reporting that, when implemented by both the client and server, enables realtime player position reporting/states. This allows MS to track Subsonic players with the same level of accuracy as Spotfy, Jellyfin, and other first-class Sources!
Huge thanks to @JannisPohle for the initial PR on cleaning up subsonic players ( #640 ) but then stepping up to also implement this playback functionality ( #590 ) :crown:
As emphasized above, this new functionality requires that both the Subsonic server and the client you use to play music from that server both implement Playback Reporting. An informational tooltip in the UI Preview will indicate if enhanced reporting is available.
Data Directories Standardization
Previously, all persistent data was stored in the directory defined by the CONFIG_DIR ENV, or defaulted to {projectDir}/config.
Now, these locations are configurable by data type and fallback to OS-standard locations:
- Configuration (
config.json,jellyfin.json, etc…) usesCONFIG_DIRENV or OS-standard locations like$XDG_CONFIG_HOME/multi-scrobbler - Data (
ms.db, logs folder, stored credentials, etc…) usesDATA_DIRENV or OS-standard locations like$XDG_DATA_HOME/multi-scrobbler
Upgrading
If you are using a Docker installation with default volume mounting to /config then no change is required. For backwards compatibility, the docker image installation defaults both of these ENVs to /config.
For users using the nodejs/local installation, you will likely want to explicitly set both ENVs for your start command. For instance, if you are currently storing everything in the config folder within the cloned repository and the working directory is also the repository folder then add these ENVs to the beginning of your command:
CONFIG_DIR=./config DATA_DIR=./config ... npm run startSee the Config and Data Directories section in the installation docs to find the OS-standard locations MS will use when these ENVs are not set.
Monitoring Behavior
In preparation for wider usage, the feature known as “Should Scrobble Behavior” has been renamed to “Monitoring Behavior”. The configuration naming has been changed to reflect this.
Upgrading
If you are an Icecast or Azuracast user who has systemScrobble in your File/AIO config options, rename this property to autoMonitor. The behavior for this property remains the same.
Example
Old:
[ { "name": "Station 1", "id": "myIceStation1", "data": { "url": "http://myIcecast.com/radio" }, "options": { "systemScrobble": true } }]New:
[ { "name": "Station 1", "id": "myIceStation1", "data": { "url": "http://myIcecast.com/radio" }, "options": { "autoMonitor": true } }]Config Improvements
Internally, config has undergone a complete overhaul and is now entirely built using Zod. This is a huge improvement over the hacky typescript-types-to-json-schema generator that was being used because the config shape now fully dictates validation, rather than there being a reflection middle step. It additionally removes the ajv library and saves almost 30MB in the docker image when uncompressed.
With the overhaul, a simplification of the config reduced development complexity and made the config explorer in docs clearer, at the expensive of a few, potential breaking changes.
Upgrading
User Stage Transforms
If you have User Stage transforms and they do not already have a "type": "user" property, you need to now add this property or validation will throw an error.
Example
Previous User Transform config could be untyped like
"options": { "playTransform": { "preCompare": [ { "title": [ "badTerm", ] } ], }Now, all user transforms must be typed like all other transforms or config parsing will throw an error:
"options": { "playTransform": { "preCompare": [ { "type": "user", "title": [ "badTerm", ] } ], }Removed Deprecated Cache Config
Cache Configuration was simplified in 0.14.0 but using pre-0.14.0 configuration only caused a deprecation warning. Now, these deprecations have been removed and using the old configuration will cause an error.
Node Native Runtime
Multi-Scrobbler has been refactored to remove the dependency on tsx as the runtime. Now, MS uses Nodejs’ type stripping so that it can directly run the source typescript files without needing the typescript runtime or any compilation steps.
This reduces packaging, tooling, and local run complexity:
- environments do not need to rely on
npmscripts for running MS- or needing to specify the tsx binary with node for commands
- Improves process monitoring readability
- Reduces memory usage since typescript does not need to be loaded into memory
- Reduces disk space since typescript does not need to be installed to run in production
- Tooling calls (mocha, vscode extensions) can directly use node without a shim
If you are currently running MS without using npm run start and have questions about how to migrate please open a discussion.
UI Preview Improvements
The New UI Preview introduced in 0.14.2 has seen several rounds of refinement thanks to user feedback. It is still in “read-only” mode for this release but the next release will begin to introduce write functionality. Check out the preview on your instance:
http://yourMsDomain/nextand share your feedback in the github issue!
Other Changelog Highlights
🐛 Bug Fixes
- (client) Correct create/update timestamps for queues and scrobbles
- (client) Correct SOT scrobble api requests to not have the same from/to value
- (listenbrainz) Fix play fetch more indicator
- (musicbrainz) Catch scenario where all search prereqs fail
- (rocksky) Fix missing albumArtist field #632
- (ui) Stabilize play timeline item order
- (mpd) Assign empty array if artists not defined #621
- (No Category) Fix play fetch options logging data for pageless functions
🧪 Testing
- (No Category) Fix db error test using transient db
Changes Since RC1
⛰️ Features
- (No Category) Reduce db write frequency when idle by buffering date updates #653
🐛 Bug Fixes
- (webscrobbler) Fix bad access of PlayerState #650
- (No Category) Fix missing logPath base dir argument
- (No Category) Guard against tracks/track in mb recording medium #649
📚 Documentation
- (No Category) Add missing user type to user transform examples
- (No Category) Clarify when activity is scrobbled
0.14.2 - UI Preview
What’s New?
Historical Scrobble Matching for teal.fm/Rocksky
Multi-Scrobbler checks your scrobble clients to see if an incoming Play was already scrobbled. This process was previously overhauled in v0.12.0 to make it work for any arbitrary time range, but this overhaul did not include teal.fm and Rocksky due to limitations in how querying atproto records works.
In v0.14.2 this has been resolved so that these clients have the same arbitrary timerange functionality matching as all other clients. This is achieved by initially downloading a full copy of your record and converting them into MS scrobbles to store in the new database. Then, the copy is kept in sync with real time queries to the atproto api. MS then queries this copy when matching scrobbles.
New UI Preview
The UI for MS has been fully rewritten and a preview is now available alongside the normal UI. Some highlights:
- mobile-first design
- live, visual updates for all activity
- search and filter all of your scrobbles
- scrobble timeline shows how MS processed your scrobble data from end-to-end
It can be used by navigating to
http(s)://yourMsDomain/nextI need your feedback on bugs, opinions on design choices, and general usage impressions! Refer to the github issue for the preview ui for more details and to provide feedback.
Some screenshots from the preview ui:
Left: Source/Client list (main page) | Right: Source detail page with Plays
Left: Scrobble duplicate check results | Right: musicbrainz transformer “no results” with used queries
visual indicator of live activity (just scrobbled) on main page