Demlo - A dynamic and extensible music library organizer (in pure Lua)

Question

Demlo organizes your music library automatically and dynamically. It runs a chain of user-defined scripts using variables such as tags and file properties. This way it yields virtually unlimited customization power to the user.
Use case
As an example, consider a music library that is a a perfect definition of chaos:
No folder structure.
Tags are not homogeneous, some entries are filled, other not, it varies from file to file.
Some files have bad audio quality.
Audio codecs are not always the same. Some file are CBR/VBR, lossless/lossy, ...
mp3's ID3 tags are a nightmare.
Covers are sometimes embedded in tags, others are of terrible quality.
Features
A few scripts are already provided and they give a good example of what you can do. You can choose to use them, to rewrite them, or to write a new script from scratch. You can preview changes, and once you are done you call the program over you music library, it will process everything in one single run!
Scripts can be chained: this feature makes Demlo extremely flexible and powerful!
Move all files according to a unique, yet dynamic folder hierarchy. For
instance, setting
output.filename = table.concat {library, '/', o.artist, '/',
                 not empty (o.album) and (empty (o.date) and o.album .. '/' or o.date .. ' - ' .. o.album .. '/') or '',
                 empty (track_padded) and '' or track_padded .. ' - ',
                 o.title}
will move "john doe - 1. intro.ogg" to
"/home/$USER/music/John Doe/2013 - Great Album/01 - Intro.ogg" if there is an album and a date,
"/home/$USER/music/John Doe/Great Album/01 - Intro.ogg" if there is an album and no date,
"/home/$USER/music/John Doe/01 - Intro.ogg" if there is no album.
Case checking: a powerful script that will turn everything to title case. It supports special cases such as Roman numerals, McDonald, etc. It also supports a list of exceptions (e.g. AC-DC, DJ, feat., etc.).
Encoding: you can conditionally re-encode you files, so you can choose to only re-encode files with some particular encoding settings. You can set the desired bitrate, etc.
Covers: automatically remove embedded covers, remove duplicates, remove files beyond/below a quality threshold, etc.
Different kinds of music: classical, OST, and band songs have usually different kinds of tags and folder structures. You can manage different audio libraries easily by setting the corresponding "library" variables.
Edit tags of a file set with your favorite text editor.
Support for cue sheets. (Both external and embedded.)
Demlo can be interfaced with any other program, both ways. (X calls Demlo or Demlo calls X.)
MusicBrainz support for online tagging and cover fetching.
Technical characteristics
This program is completely written in Lua. Its only major dependency is FFmpeg. A few Lua modules are required, which should not eat more than a few KiB on your drive. This leads to a lightweight codebase for a portable and extensible program.
Dependencies
ffmpeg
lua (>= 5.1)
lua-dkjson
lua-filesystem
lua-llthreads2
lua-penlight
lua-socket
slnunicode
chromaprint (optional)
History
This program is based on Musish by the same author.
The original program was written in POSIX shell but suffured from many limitations as well as performance issues. It was rewritten in Lua to circumvent all those issues while applying minor changes to the CLI options and to the variable names. The syntax of the script changed to Lua as well. Here follows a list of major changes:
Traversing folders recursively is no longer a security issue.
Overall performance got boosted by a factor of approx. 2.
Scripts and configuration files are sandboxed, which means they can no longer contain harmful code. However it is no longer possible to chain scripts.
All tags are processed dynamically, no more hard-coded tags.
Links
Official web page
Arch Wiki
AUR package
Issue tracker: Please file bug reports there thanks!
Last edited by Ambrevar (2015-02-20 09:22:22)

Answer 1

 I hoped the structure I presented was clear enough. No, all/majority of tags would be a little too much, but genre, different release types, file types and some other basic distinctions, yes and tagsfs will probably help with that. There are special folders like [Compilation], [High Voltage SID Collection] and probably others in the future, maybe with help of tagsfs. Compilations and Artists are prepended by the leading letter of 'sorted artists/albums' level. Then albums/artists are sorted/presented in the manner visible in the Picard script:
$if($eq(%compilation%,1),
%COMPILATION%///%FIRSTLETTER%///$if2(%ALBUM%,%NONALBUMTRACKS%,[Other]) - $if(%DATE%,[%DATE%],),
%FIRSTLETTER%///$trim($if2(%ALBUMARTIST%,%ARTIST%),.)///$if(%DATE%,%DATE% - ,)$if2(%ALBUM%,%NONALBUMTRACKS%,[Other]) )
$if(%RELEASETYPE%,[%RELEASETYPE%],)%BARCODE%$if(%INCOMPLETE%,[%INCOMPLETE%],)[%EXT%]///
$if(%DISCNUMBER%,%DISCNUMBER%-,)%TRACKNUMBER%. %ARTIST% - %TITLE%
 Picard has preview examples, so copying the whole script into file naming script under options should make this definitely clear.
Ambrevar wrote:Thinking about it, a similar feature is provided by most library managers where you can choose filters and tree structure and so on. It remains virtual though, it does not touch the filesystem. Is it really necessary to you to have that sort of view directly on your filesystem? The problem of the filesystem view seems very complex to me.
Not all devices provide that capability. It is more pleasant to work with that approach than previous structures I used, remember that I can restructure the whole library at any time and probably copy the local one elsewhere using the new structure, but I have not looked into that yet.
 We probably have very different experiences, I see this as very comfortable and convenient. Before that I had been using folders for different file types in the main directory, but it ended up with so many entries for FLAC and MP3, that parsing that directories in a player or any maintenance using a file browser/terminal was too troublesome/slow. So I reduced the number of files, by using the alphabet and additional characters as sorting mechanism. And since that was one step away from separating compilations from albums/single releases, I did that also. Now I can access the whole library very quickly, even when one or two additional characters need to be typed.
Ambrevar wrote:Furthermore, would you consider this to be an automated task or a manual one? Using your example, how can a program know that Lauge & Baba Gnohm is both Lauge and Baba Gnohm if no tags says so?
 Both, I want as always full control. MusicBrainz [database] relations.
Ambrevar wrote:Hard links do not work for folders.
Symlinks cause problems with most file browsers that do not follow them during copying.
 I do not recall writing they should link folders, the directory structure would have to be [re]build and then files [hard/sym]linked, which might pose a maintenance issue, although deleting links and empty directories not really, but "leftovers" like covers could block cleaning. *The only file browser* I would recommend is Proto from a very friendly Polish programmer named Mieszko Lassota, but it is, or at least was last time I checked, Windows only and slowly further developed, although still kills every mc, mucommander and ranger on sight, that is how I remember it. SpaceFM is bearable, but I still only use rsync to copy files and see no point using anything else.
Ambrevar wrote:Can you elaborate on this? What are the limitations of Picard beyond symlinking? It could be interesting to know what feature you would like.
 The recursive/linking aspect it the main point here. I hope this is not too big of a disappointment. Having too long names has also been a problem in few cases, classical music if I recall correctly, but those are limitations of the file system, however even that could be overcome with Picard, but the code would not be as easily readable as it is now. The question would be what should be abbreviated/priorities, what to do when after abbreviation the directory/file names are still too long and so on. I had to do it by hand these few times. Picard does not handle covers/images well in my experience, but I used something else for that, puddletag I think. I would have to continue to organize the library to remember.
 Furthermore, not all media players/organizers do have the capability to get exactly this structure with formatting, leading letters and distinction between compilations and albums being two examples. Browsing this robust file structure with ncmpcpp is very efficient, I almost never use the library view. As long as a media player has the browsing capability, I have the same view/behavior. And other hardware also works without issues. Yes, it basically comes down to having the file system do, what media players should, however with a few exceptions I had only positive results for more than almost two years now. Do you now see benefits of having that file structure?
 Again I will take a look at your tool, when I find some time and what progandy suggested also of course. How do you save your music on the file system?
Edit: Style. Clarification.
Last edited by emeres (2014-09-12 13:02:34)