Re: [LAD] Non Session Management

On Tue, 2012-03-27 at 01:04 +0200, Emanuel Rumpf wrote:
> I'm trying to figure out different use cases:
>
> - an app loads an audio file (reference to orig file)
>
> ---possibility: file moves -> ref. has to be adjusted
>
> - the app is non-destructive, for changes, a copy is produced (where?)
>
> - the session is being duplicated - the new session keeps the refs to
> original audio files, (but creates copies, for files which have been
> modified ?? or refer.s them too ? refs. them too, I suggest. )
>
> The problem here is, that files could quickly distribute (and
> cross-link) over MANY different directories. Maybe a common directory
> for all audio-files modified by ANY session-capable application
> instead ?
> pros: For all instances, we knew at least their files location.
> Different instances could link to the same file (creating a new copy,
> only when modifying)

This exact problem came up when I set out to implement non-destructive
plugin state saving in Ardour, with support for plugins referring to
files (e.g. loaded samples).

The obvious basic solution in a non-destructive context is to save each
snapshot to its own directory. The tricky bits come when you have links
in that directory to files. What you end up with is a few directories
with specific roles.

I'll try to document the path leading to this as tersely as possible.
The plugin/host situation is analogous to app/SM, very closely with
respect to what's on disk, not so much with respect to responsibilities
and such.

There are two kinds of file:

* External files, e.g. something you loaded from the filesystem
somewhere. There are assumed to be immutable.

* Files created by the plugin, which may be different each save

As I have said a few times on this list, for various reasons I think
straightforward and transparent symlinks are the best way to refer to
external files. I'll just take this as given/obvious and not bother
rehashing the argument (erecting a bunch of protocol and/or file format
junk only *adds* problems).

So, you want to save state, and the plugin refers to a bunch of files.
As mentioned above, this can mean a lot of references to one file.
Since these could be massive, redundant copies must never happen at
all. For ease of moving things, or resolving them if they break, we
want a minimal number of actual links to an external file, i.e. 1 (this
is also necessary for archival, see below). So, we don't want every
single state snapshot directory to have a link to /media/bigfile.wav.
To avoid this, make a directory specifically for all links to external
file, and link to those links instead. This localizes all links to
files outside the session.

The other case is files written to during execution, e.g. recorded
waveforms. Here, on each save, you need to check if the file is
actually any different, and make an actual copy if so (so the original
can continue to be used) in the state snapshot directory. During run
time, all the created files live in a "scratch" directory, which you can
preserve, or just throw out when you close since copies have been made
for every snapshot.

Working with the requirement that the plugin's file namespace must not
be messed with (e.g. if you save foo.wav to your state directory, it
should actually be called foo.wav), you end up with:

1) The "file directory". This is where the plugin creates files,
whenever. It can be considered run-time scratch.

2) The "copy directory". This is where copies of things in the file
directory are made, to preserve their state at a particular instant in
time. This mirrors the structure of the file directory, but appends .2
or .3 etc. for the various snapshots of a file

3) The "save directory". This is the directory of a particular state
snapshot. It can contain whatever.

4) The "link directory". This is where all links to external resources
live. Any state snapshot that refers to an external file actually links
to a link here.

This is in the Lilv API. It might seem complicated, but it is only
optionally this powerful, if you don't care about non-destructive saving
or avoiding copies you can simply use one directory for everything.

I think I can make a reasonable argument that this is the minimal
number of directories that meet the desired requirements:

A) The file directory and copy directory can not be the same directory,
because the copies would pollute the plugin's file namespace and
possibly clash. It would also make it difficult to know what files are
actually needed by the session, since some may be scratch. The files
directory is unique in that its contents may be safely deleted.

B) The copy directory and save directory can not be the same directory,
since multiple saves may refer to the same file with the same state.
Without the copy directory this would require two copies of the same
file.

C) The link directory must be separate to avoid having a large number
of links to the same external file. This is convenient in general
(only one link to potentially fix), but required for archival. An
archival tool (e.g. tar -h) would actually copy the file to the links
location. Without the link directory this would produce an archived
session with many copies of the same (possibly massive) file.

D) The save directory is inherently where one save takes place, its
existence is a given.

Figuring this all out was a long process of trial and error, but now
that I write it down it seems clear it can't be simplified without
failing to meet a requirement. I am all ears for arguments to the
contrary, though.

This scheme would need some slight tinkering to tolerate external files
that change, but this would require keeping copies of system files
around just in case, not just when you archive. That is, ANY external
files used EVER would have to be copied in case they change. I don't
think this is acceptable, and plugins have no business modifying
external files anyway. Apps that want to get fancy and share files may
want a solution to this, but particularly since shared mutable files is
almost certainly crazy anyway, one app should simply "own" any such
files. It may tell other hosts about them if it pleases, possibly via
OSC. I don't think the session manager itself should be bloated out
with a bunch of unnecessarily shared file database stuff to support a
rather esoteric ability that can easily be achieved without doing so.
In other words, if apps want to talk to each other and share file paths,
that is their business.

-dr

_______________________________________________
Linux-audio-dev mailing list
Linux-audio-dev@email-addr-hidden
http://lists.linuxaudio.org/listinfo/linux-audio-dev
Received on Tue Mar 27 04:15:06 2012