Next generation monitor: scheduler

[emcodem]

Hey, i made my thoughts about what could be the most useful extension for ffastrans that i can - using existing features only - come up with in order to support the creator, @steinar.

My final conclusio was that coming up with an alternative "Monitor" component should be the most flexible approach that could integrate the best.

The new Monitor component would be a totally independent windows service that allows to attach custom monitor pluings in form of a .NET dll.

Features:
-) split workflow mangement off from watchfolder management
-) Webinterface with credentials that allows configuring new watchfolders with targetfolders and connected ffastrans workflow plus the name of the "start component"
-) as said, set the "output" directory, so you can possibly delete a lot of workflows that just deliver to different output folders
-) Plug-in Interface: it should be possible to dynamically find and use all dll's in the plugin folder, so it is possible to develop your own method of finding files or similar
-) Scheduler: start workflows based on a schedule
-) Possible start workflows based on filesystem notify, so no more waiting time for watcfhfolders to respond (if applicable - fallback to scheduled folder scan)
-) Brain: keeping it's own database of job history, with a possibility to remove this kind of need and functionality from the core software... by:
-) Provide it's own REST API
-) the same technolgy can also serve as advanced job montor

Any thoughts?

Like this (the values are some random example data, dont get confused by it):

ffastrans_webinterface.png (35.15 KiB) Viewed 8889 times

[admin]

Hi emcodem, I like your dedication!

I have a couple of question you might want to enlighten upon:

How many methods are there of finding new files? You have the rude "search-fo-new/changed-files-using-various-teqnikes" and the system notification which is nice but don't work on SMB-shares (as far as i know). I can see that scheduled monitoring would be a nice option and it's something I'm considering. But are there other ways of looking for files?

I know that the current job history is not very good and I'm going to work on that in time, but is there a particcular reason why you want to keep your own history database? It seems like a database on top of a database.

That said (written), FFAStrans really needs a better web-monitor but I have not got the head above water to make web-gui's....which I really is something I normally don't do. I'm hoping to provide a good enough api for other to create good monitors and other solutions. So I'm really keen to get feedback on what the api should be able to do.
Last, are you planning on just using the API or do you need to go "through the backdoor" in order to accomplish what you want?

-steinar

[emcodem]

Hey steinar,

up in front, sorry for the long thread but as you only respond weekly, i had lots of time to think about it

great to get some response on this, i hope to get some more re-questions. In the end if i really go on developing this, it should be a good planned thing that you even might want to deliver along with your builds in future.
I'd like to serve packages and maintain the code on github as usual.

How many methods are there of finding new files?
You have the rude "search-fo-new/changed-files-using-various-teqnikes" and the system notification which is nice but don't work on SMB-shares (as far as i know). I can see that scheduled monitoring would be a nice option and it's something I'm considering. But are there other ways of looking for files?

Well, it might not only be about a "File on the Filesystem" that is the event for starting a job. In my head it is just an event in programatical sense. To be honest, i might think of ffastrans in another way than you: for me it is more like just a job processor. Why dedicate to work with Files only.
On the other hand there are lots of different File events as well, just as you needed to implement the different P2 and other modes. This stuff can be pretty flexible, is hard to debug and does not really fit to any core software. It just should be outsourced into a module.

I know that the current job history is not very good and I'm going to work on that in time, but is there a particcular reason why you want to keep your own history database? It seems like a database on top of a database.

Yeah, the reason for keeping my own history DB is that the History DB needs to be disconnected from the "working" DB in order to prevent that this optional UI stuff influences the mission critical processing itself e.g. processing gets slow or stops because too many history jobs are stored in the production DB. Users typically dont really like to have a short history; what they acutally deserve is to have all history to the very start.
Using a Database for history that is not connected at all to the "productive part" of the software (in your case to the cache i guess) makes production more stable and secure: In case the History database does not work, is overloaded due too many users etc.., production is not influenced.

That said (written), FFAStrans really needs a better web-monitor but I have not got the head above water to make web-gui's....which I really is something I normally don't do. I'm hoping to provide a good enough api for other to create good monitors and other solutions. So I'm really keen to get feedback on what the api should be able to do.

No worries, that is why i offer my services here. I already opened a feature request topic here regarding what i need for start (jobid,workflowid), but to be honest i think users deserve more info about reasons for failure and so. In my head you could basically continue writing logs and ini's as you do now and i could just locate and parse them but it would of course not be very welcome if there are extremely frequent and deep logical changes in the way how the logs are written.

Last, are you planning on just using the API or do you need to go "through the backdoor" in order to accomplish what you want?

See above-- and additionally i typically try to figure out what is most stable and use that, not matter if it is a backdoor or not. In most cases with professional transcoders i connected through the backdoor to their database which turned out to be most stable. They did a lot of changes in their public API's but they never changed their basic DB design.
A comment to that: ffastrans for example is assembling ffmpeg commandlines, in my head that was always like a backdoor; using the libavcodec would be like using the API in comparison. I always kept away from scripting ffmpeg commandlines because they change so fast...

I see there is a lot of potential for discussions, hopefully we can concentrate on the topic itself. What i like to try in general is to split off coding from old techniques into a new world, allowing more flexible and modern possibilities.


2 more things:
1) FileSystemNotifiers are in deed available on most SMB and similar Servers, in difference to Linux Inotify which only workso locally, on a SMB Server there is no difference between local and remote Filesys Notify subscribers. It is just a network communication that one can listen to. To be honest, i have never seen any SMB type of server that don't support it, maybe some very old and crappy linux SAMBA implementations do not implement it.
However, there are a lot of caveats just as with any very old technology like Sockets themselves. They suffer from reliability and documentation. So whenever one uses them you either go for a "best effort" approach or you find a workaround. Over time i learned that using Filesys Notify always comes with a fallback "dirlisting" in case something was missed or the connection got lost but the program was not notified about the error. This makes it pretty hard to implement filesys notifiers.
One might think: "why use Filesys Nofify when it is not at all reliable", well that is because it is best effort and only this makes a software really good. Imagine a full scan for new files on a (from a todays perspective) very large file repository takes 15 minutes to finish. One can reduce the storage load and response time dramatically when using Filesys notify.


2) How many methods are there of finding new files?
Basically indefinite. Just as P2 and Co, most users end up in a case where they want to "send the event of a new job needs to be started" in pretty complex cases, e.g. when file A, B and C arrived or when some metadata xml contains this and that value but not those values... i faced and already implemented lots of side cases - also i found a noticeable number of requests in that direction in this very forum.

Cheers!
emcodem

[emcodem]

As a separate subject, i'd like to point my thoughts about what the API could deliver:
-) from my perspective it does not really make sense to have separate methods for getting queued, running and finished jobs but that does not really matter
-) Typically the starting Point for a Job History related Userinterface is a list of Jobs. Each job should contain all relevant information the user might be interested. E.g. The workflow Name is good but as it is not unique, only the WF ID enables me to find the connected workflow.

Here a list of Job API related infos that would help building a really useful UI:

* Unique Workflow ID - that enables me to show workflow related info to a specific job
* Unique Job ID
* All available variables in their current state - this one is most interesting, it allows basic communication from a running/finished job to the UI - also technical file information is very relevant (if applicable - not sure what about jobs that have a multi file monitor like P2)
* if you have it, the original s_source file (full path) (i really mean the "original" one, not the current one as it can change)
* Array of the currently executing processor nodes (or the last node executed) - i could use the workflow api to get the nodes description then
* Path to the log file AND/OR An api method to retrieve the job log
* Jobs should be log enabled by default (logs cannot really get big as in hundreds of megabytes so why have an option to disable logging anyway) - and there should possibly be one central and not changeable location for logs - this makes log maintenance possible and simple

[admin]

Thank you for all your suggestions on how I can improve the FFAStrans API. I really appreciate it even though I don't respond to all in detail. The reason I don't answer right away these days is that I am very busy.

The current data-modell for FFAStrans is not ideal by far. Fixing it is very much work because I need to refactor most of the modell to be much more transparent in a REST-API environment. The goal is to JSON'ify the lot. The work has allready begun even though it's not visible for the users. In that regard it's natural to expose most of the string-data being processed and used in a workflow. Regarding log files; the current state is very hard to parse into a JSON in order to present to the user. So basically, they need to be worked on in order to be usefull for anyone but myself. That is also on the todo-list. But as with the data modell in general, it will take time an effort.

I can see that there are many scenarios of what kind of files and their dependencies you can monitor but I was referring to the ways of looking for files, regardless of how they may be processed. The last time I tried notification over SMB it did not work. That might have changed now. However, if we know that it's flaky then we need to do a file search anyway because we cannot know if we miss any files when our first attempt is a (flaky) notification. That's why I think that the search for files actually is the most foolproof and predictable approach. Yes, it takes more resources but for the userit it takes even more resources if the damn thing does not work. So I don't see the point of doing notification if we anyway end up having to filesearch because we don't know if we miss something. At least, that's how I see it. If you know a way to overcome having to do both, please let me know.

Can you please elaborate on "All available variables in their current state"? Do you mean ALL or just the user variables, or maybe also all the media-variables?

-steinar

[emcodem]

Hey steinar,

i can imagine that there is a need to re-factor lots of stuff, that is always the case when a software is multiple years old
Regarding log parsing i somewhat disagree with you, from my perspective the core only has to provide logs but other software should parse it to their needs. These days we have very advanced log parsing and indexing mechanisms available and due to the nature of ffastrans, logs can always change with the used dependencies as well.
What i want to say is: if there is a defined way how to get some log, i can make something out of it, no matter how good it is structured or not.

@Filesys Notify: basically this is only needed for really huge environments that require to listen on directories recursive where multiple hundred thousand or more files are located. Watching a volume that is faster than 1second to serve the dirlisting does not at all require Filesys Notify. However there are also some pitfalls for this kind of watching, e.g. date modified, last access and created not actually behaving as usual. But those are pretty easy to overcome.

@All variables in their current state: just user variables would be a good start but to be honest i was more referring to any variable that e.g. a "Populate Variables" node would have access to.
You know, i don't know the backend so my thoughts are boilerplate, i just say what would be best without thinking on if it is possible or what it would require on your side.

Cheers,
emcodem

[admin]

Yes there is a way to get log but you wont see it in the API, and the log is not very good but it's there. You need to enable logging in the workflow properties under the "Special" tab. The log-filenames will be in the form of job_id.txt. The content does not follow a particular standard, which is something I would really like to solve to make i more readable/parsable.

I will take a look at notify again and check if it pass the tests. It will need to be just as good as or better than dirlisting.

I have to think a bit on the exposing variables states stuff. Not all variables are populated all the time, which means if you ask for a variable that's not populated per node or per job, the current job instance will have to populate it. However, with the current design the nodes are executed in a closed environment so you cannot simply ask for a variables state from the API. I will take a look at it though.

I'm very happy you ask for these features because I cannot figure out all use cases by myself So I'm more than willing to make sure the API is as good as possible within the FFAStrans framework but it takes time and there is a lot to work with here.

-steinar

[emcodem]

Hey there,

to follow up the original topic about introducting a scheduler:
It would be pretty simple for me to introduce a scheduler on webinterface side. It would run internally every second and check if there are scheduled_jobs that require exectuion.
The problem is that when a job should run e.g. every 1 minute (e.g. a job that compares 2 folders for differences and only carries on processing if there are new differences found), we would also have a job every minute in the history.

So before i continue implementing a scheduler, i would like to ask for ideas how we could handle this case for scheduled jobs that were executed but actually did not do anything. Also one does probably not really like to see a scheduled job that does nothing on the Running job UI. And above that, the actions that are considered as "nothing" should not be queued up but being executed always and immediately.

Any idea if it is possible and simple to come around that topic? Or is this just too far away from the current design?

cheers,
emcodem