FLAC yourself

NOTE : This is the first in a series of about a decades-worth of notes (not an exaggeration) I’ve been meaning to type up and store somewhere, but haven’t had the/any free time. Having just moved back to the U.K., and being theoretically a gentleman of leisure for a couple of weeks, it’s now – if ever.

DLNA is fantastically useful. The convenience of being able to play via any of your screens over the network (or just from a USB hard drive) is a giant leap in terms of convenience over carrying huge amounts of DVD/Bluray discs around (literally around the world in my case, having just moved back from Australia after 12+ years – I have crates of them still in storage in Sydney).

Format support can however be spotty, especially when dealing with lossless soundtracks. The Playstation 4, for instance, won’t even play DTS over the network, let alone DTS-MA or TrueHD. So here is a guide on how to create archive versions of your media files containing lossless versions of the soundtracks.

Why you might do this

I use FLAC for soundtracks. Why would I do this? There are three lossless formats generally in wide use at present, uncompressed Linear PCM, and compressed DTS-MA and Dolby TrueHD. I’ve converted all of these to FLAC for almost all of my rips. DTS-X and Dolby Atmos are not covered here, as they are very different, in that rather than being studio-mixed around a certain speaker layout they are mixed at the point of, and to match, the listener’s environment (with the exception of a >=7.1 channel track to provide ambience).

LPCM is fine, but larger than FLAC due to it’s uncompressed nature (and thus takes up more bandwidth when sending over a network). It also lacks some features of FLAC, including error checking.

DTS-MA has several disadvantages. Firstly, it is proprietary. Secondly, even if it’s the case that your player and receiver can deal with the lossy (“core”) DTS part of the audio stream (no sure thing – see, once again, the PS4), it is likely that the extra chunks used to restore the missing parts of the audio to recreate the lossless audio stream (added using the “XLL” extension – see section 8 of this document) will be – at best – ignored, as will extra channels beyond 6.1 (see the “XXCH” extension in section 6.4 of the same document). If by any chance lossless audio is supported, then the stream must be reconstructed, giving rise to overhead in terms of bandwidth and CPU usage. Due to a quirk of the format the output is always reconstructed at its maximum possible size – 24bit 192kHz with 7.1 discrete channels – with the unused space padded out with silence. This is too much for the many devices that cannot push out that much data over HDMI (especially over older pre-1.3 versions of the protocol), and is mostly unnecessary. Very few commercial soundtracks exploit the maximum bandwidth available (if you know of any feel free to drop me a line).

TrueHD is also proprietary. It’s less problematic than DTS-HD, in that it does not contain a core AC-3 stream to re-integrate – rather the lossless version is carried as a stream separate to the lossy version. Once again though, it takes a fair bit of CPU power to decode.

For slower machines (older Raspberry Pi’s, as an example) this means conversion is the only game in town for compressed lossless audio soundtracks (it helps that LPCM has been part of the HDMI standard since the beginning in order to allow format decoding on the player). I have had this running in the past on a model B using Raspbmc/XBMC (later renamed to OSMC/Kodi) connected to my processor and playing lossless soundtracks flawlessly (HDMI versions prior to 1.3 may have issues with 7.1 LPCM soundtracks). The CPU usage remains light enough not to be overwhelming when converting FLAC to LPCM in software and no hardware conversion is required. Should it be necessary any downmixing can also be performed here without the risk of melting your Pi’s CPU (e.g. 7.1 -> 5.1 or stereo). It may be possible to do the conversion in software at runtime using later model Pi’s (I haven’t tried), but it’s an extra stage you do not need, especially if your target player needs reconversion to a natively supported format e.g. AC-3 or AAC.

FLAC ticks all the boxes in that it is quick to decompress, fairly ubiquitous and an open-source format which is not likely to go away anytime soon, making it ideal for both archiving purposes and for playback on resource-constrained systems.

Why you might NOT want to do this

For DTS-MA and TrueHD this is, for all intents and purposes, a one-way conversion. While it may be possible to re-encode as TrueHD or DTS-MA there is no guarantee you will be able to convert back to a bit-exact copy of your original source (though TrueHD/MLP is likely to be easier than the DTS route, as it avoids the lossy compression altogether). If you are worried about bit exactness, preserving metadata and the like then ensure you keep your original disc (in fact I’d recommend you do this anyway, though as anyone who suffered through the laserdisc disc rot saga would tell you, keeping the physical media is no guarantee) and back up your original/intermediate files somewhere safe. The FLAC “–keep-foreign-metadata” option may help (unverified), but is beyond the scope of this post.

With regard to playback the situation can also get a bit difficult. While the CPU load involved in decoding FLAC is minimal, many backends will also attempt to transcode the video along with the audio, even when the video output is supported natively on the target (e.g. h264 on Chromecast), rather than simply converting the audio and re-muxing. Whilst this can be to a degree avoided by ensuring both the server and player are compatible (e.g. using Plex for both), or using Kodi/Videostream/VLC as a converter/frontend, it’s as well to check this end-to-end yourself beforehand (also check the screen size of the target device is large enough for the video, and check the video bandwidth is set to maximum). I’m currently using an ODroid XU4Q as a server – I’ll put up another post detailing the in’s and out’s of this at a later date. It’s as well to check this will not be an issue for you before irrevocably committing yourself to converting all your files, especially if you don’t fancy editing DLNA player profiles.

How to do this

There are a couple of methods of performing this conversion. Previously converting DTS-MA required using MakeMKV with an extra .DLL to be downloaded (see here). This DLL is no longer required, and – maybe not coincidentally – DTS-MA is now also supported by ffmpeg. I meant to do this myself at some point, but all things come to those who prevaricate. Ahem.

As a consequence MakeMKV no longer requires any extra software, but I’ve included instructions on how to use it as it provides a nice friendly GUI interface and integrates the AACS decryption step should it be necessary. If you’re ripping direct from Blu-ray this is the place to start, though it will accept already created Matroska files as input should you wish to convert previously created rips. If you want to convert already ripped files stored in a different container format (.mp4 and the like) jump ahead to the section detailing how to use ffmpeg at the commandline. This is also currently the only option to convert files on non-x86 machines.

The MakeMKV method

Download and install MakeMKV. It looks to be in perpetual free beta status but you can buy a license to ensure continued development.

Under the View/Preferences menu enable “Expert mode” :

makemkv-expertmode

Select and open your source file or Bluray disc :

makemkv-opensource

Now select the relevant lossless soundtrack and set the profile to FLAC :

makemkv-selected

Edit any metadata here. This is also a good opportunity to remove any unnecessary subtitles or cover pictures if you’re looking to save space. Note that this will remove the DTS core soundtrack. Re-enable it if you wish to preserve it. Now hit the start button in the “Make MKV” panel :

makemkv-complete

We now have our newly FLAC’ed output file. Due to the flatter format (and possibly better compression) we’ve also shaved around half a GB from our output file (8,489,995,617 bytes -> 7,980,226,213 bytes). Success!

The FFmpeg method

Firstly run ffprobe (or ffmpeg with no output parameters) to determine the contents of your source file.

ffprobe -i 'input.mkv'

This will list the video and audio tracks comprising the file. Find the lossless DTS-MA/TrueHD track in your language that you wish to convert. If you require only a particular stream, or wish to perform multiple conversions check here for info.

Stream #0:1(eng): Audio: dts (DTS-HD MA), 48000 Hz, 5.1(side), s32p (24 bit) (default)

The following assumes that you have a single input audio stream and don’t want the subtitles :

ffmpeg -i 'input.mp4' -vc copy -ac flac -sn output.mkv

You should now have a correct Matrsoka container with lossless audio. Verify with ffmpeg (as above) or mediainfo and you’re away!

-Dbugger_off

As someone titled a compiler switch during one of my first jobs. Been looking for a way to specify commandline switches on a CMake project under VS2017? Missing Project/Properties? This is what you’re loking for : https://blogs.msdn.microsoft.com/vcblog/2016/10/05/cmake-support-in-visual-studio/#debug-code

In other news, discovered my GPS USB stick automatically adds leap seconds to GPS time. Unfortunately it only adds them up to 2015 (bought it last year). Which is logical, but a mite irritating. Would have been better if they didn’t add them at all to a certain extent.

Dealing with your OCS in Linux

The day has finally come. The two old P4 Dell’s I’ve been using as Linux workhorses (CentOS, as we look after a fair amount of RHEL boxes and that was the closest fit – RedHat no longer providing 32-bit builds) have finally given up the ghost, to be replaced – at least in the short term – with an RPi 3. This then seems like as good a time as any to put up my notes on getting a working professional Linux desktop on ARM/x86 up somewhere (RHEL 7 makes all this practically seamless for x86-64, but unfortunately isn’t yet available for ARM[64] desktop), that somewhere being here.

First up – instant messaging. We currently use Lync in the office, and the messaging part is fairly easy to replace using Pidgin (formerly Gaim). Our first step is to export any root certificates necessary from an existing Windows install (or to procure one from your ITS department). To do so run the Microsoft Management Console (“mmc”) and add in the Certificates snap-in :

MMCConsole1

Once this is done, export the root certificate of the domain (making sure to check the expiry date to ensure you have the right one) by double-clicking and opening the “Details” tab, then hitting the “Copy to File..” button :

MMCConsole2

Export the file as PEM. For some info on the different certificate formats see here – if you export as the wrong format i.e. DER you can use openssl to convert it to the correct format :

openssl x509 -in dd-root.cer -outform pem -out dd-root.pem

MMCConsole3

Save with your preferred filename, including the extension .pem, and copy to your target Linux box. The Windows part of this install is now complete. Now amble over to your Linux machine and start installing Pidgin. This should easily be handled by your package manager. Under Debian-based systems (including Ubuntu and Mint) this involves firing up the apt package manager :

sudo apt-get install pidgin pidgin-themes pidgin-sipe

This will install the necessary Pidgin icon set (not pulled in by default for some reason), and support for the SIPE protocol (an extended SIP protocol used by Lync) alongside Pidgin itself. Under Redhat-based distributions the process is similar. The package management software in this case is “yum” :

sudo yum install pidgin pidgin-sipe bluebird-xfwm4-theme

NOTE : If you have not already installed the optional EPEL repositories you may need to do so – see here for more details.

The last item is again an icon set – I’ve chosen this one out of personal preference, but any of these themes should work (try “yum search bluebird” to obtain a list). Hit Enter and agree to the various prompts. Once Pidgin has been successfully installed you’ll need to copy in your root certificate. This is installed system-wide :

sudo mv .pem /usr/share/purple/ca-certs/

Or – as in the case of Debian 8 – per-user :

mv .pem ~/.purple/certificates/x509/

Pidgin should now be successfully installed, so fire it up. On opening it’ll ask you to set up an account – select “Office Communicator” as your protocol :

Screenshot from 2016-05-31 09_38_25

And you’re done! Haven’t tried this with Voice and Video yet, will update as, and if, necessary …

 

 

Still not dead (I think)

Been a bit slack on posting over Christmas/New Year, due to having moved to a new apartment, had the most crappy case of gastric flu ever and gotten my wallet stolen and locked out twice (I really wasn’t feeling too clever, and my shortlived attempt to drink my way through it – was, err, counterproductive).

Fun times breaking a window to find the one behind was nailed shut, heading to the other side of the City to find the real estate had gone on leave for a week. This meant swinging twenty feet off the ground trying to get through the a 2 foot window. Especially when someone apparently mistook me for a cat burglar and called the cops (to be fair I’ve only recently moved in, had ripped jeans and no shoes on and was covered in my own blood due to the window – so I can now see it from their perspective a bit). Meaning I had to sleep rough until the next morning when I could get hold of my mate Dave and use his phone+Internet to transfer money for a locksmith – he being the only person not on holiday, moved out or otherwise uncontactable.

That’s a significantly abridged version. Bloody “security keys” and single living don’t really mix. Grr!!!

So productivity slacked off a bit, and now I’m heading back to work.

A tonne of stuff in the pipeline though. It’s just that the pipeline hasn’t got much shorter lately ;). It’s not helping that I’m having VMware issues (yes – I still use Visual Studio for most dev stuff and then compile on all my different targets. Force of habit, since version 1.5 …). Need to spring for an upgrade to 12. It does get a bit wearing when the module compilation breaks on every release. I’ve been using it since v5, but I’m seriously thinking of checking out  the alternatives, e.g. VirtualBox, or Parallels (if they still make that – haven’t heard for a while ..). Enough rambling – I think I’m definitely sure I’m not dead now.

UPDATE : Damn – it looks like Parallels has indeed been discontinued, so maybe it’s VirtualBox, though their OpenGL/D3D guest support looks a bit antiquated too. Interestingly it looks like it uses a dynarec for 16 bit code. Or maybe I’ll just have wait until payday for Workstation 12, unless there’s a still a free version of Player – think this is “VMware Player Pro” now though….

BTW – lest anybody wonders – previously myself and the ex had a security key each, so the “getting locked out” never came up, except in the early days when we only had one and she used to take it home early and fall asleep. I think I’m going to have to demand a spare from the real estate. I dread to think of the effects on my sanity should this happen again …

 

Not simplex enough for me

I forked Stephen M. Cameron’s C port of Kurt Spencer’s open simplex noise while on the train to make it play nicely with Visual C++, which while it’s reasonably up to date with C++ is definitely stuck in the last century when it comes to C (it’s still shy of C99). I had a go at making of it ANSI C friendly too while I was there. The repo is here : Open Simplex Github fork.

EDIT : Changes now pushed up to main repository.