FFAStrans over internet
FFAStrans over internet
Hi, can I build a farm with some friends through internet? Or Is only by LAN? Thanks
Re: FFAStrans over internet
Theoretically yes, practically no (not because you can't do it but because it's probably not worth it).
So... Let me elaborate on that.
When you create a farm, all you need is a shared drive that all hosts can access and where all hosts can read and write. Whether this is done on a LAN or over the internet through a VPN doesn't matter, so much so that it's gonna be one and the same from FFAStrans' perspective. During the pandemic many people were forced to stay at home and I did try to setup stuff this way. Needless to say, it worked. People were able to see the jobs displayed in the status_monitor.a3x and eventually stop them etc and I myself was able to open the workflow manager and modify workflows etc. There was only one problem: it was really slow. You see, every computer needs access to the db folder, either because it needs to load all the JSON of the workflows in the workflow manager or because it has to read all the job statuses in the status monitor. Now, this is ok, and you could technically use internet for the "clients" that need to just use the status monitor etc, but it's not your use case. Let's get to your use case, then, shall we? Eheheheh Well, in your use case you would have different hosts in different parts of the world accessing the same storage through the internet (perhaps behind a VPN) and this will slow everything down. So much so that if you use things like the A/V Decoder with full decoding, for instance, it will take ages to complete. Now, you might argue that you don't actually need to have a shared global media cache to have a farm and you would be right, it's entirely possible to have a farm that assigns jobs to different hosts without splitting them node by node. In this case you will have a dedicated local cache for each host and only the final files will be delivered to the central location, but still, the delivery of big broadcast files could take ages and most importantly if you have indexing or full decoding it will still need to go through the whole file and it ain't funny if it's a 1.2 TB ProRes UHD masterfile for instance... Now, even if theoretically all those things are overcome and you get on with it, the last thing I would be concerned about is latency. You see, every time you need to perform a task, a ticket is created. Those tickets can be pending, queue, running etc and are picked up and executed by the exe_manager so that the relative task is carried out and eventually a new ticket is created. You see, when a new ticket is created, the hosts in the farm will "rush" to say that they're available and that they can pick it up, but remember, we're using a master-less design, which means that there isn't a master that checks the hosts and assign them tickets, but hosts "pick up" a ticket and mark it as picked up for everyone. This is a good thing, by the way, cause having a master-less design allows us to be totally independent so that jobs can be carried out until the very last host of the farm dies, unlike in a master design in which if the master dies, all the slaves are useless and everything stops. Anyway, I'm digressing. The point is that the whole db and the ticket thing is handled with files and being a file based system the longer it takes for a host to read the tickets and assign them, the worse it is. Besides, given that everything goes through the internet there's a chance that network errors occur, packets are lost, ACKs are not received with data sent again (TCP) or lost forever (UDP) etc and all those things will make it harder for FFAStrans to synchronise everything (although Grandmaster's logic is really really really well done and should be able to sustain terrible network errors/storage disconnections without the blink of an eye).
Anyway, to recap:
- Can it be done?
Yes!
- Is it actually worth it?
Maybe not, but it will depend on you.
Cheers,
Frank
So... Let me elaborate on that.
When you create a farm, all you need is a shared drive that all hosts can access and where all hosts can read and write. Whether this is done on a LAN or over the internet through a VPN doesn't matter, so much so that it's gonna be one and the same from FFAStrans' perspective. During the pandemic many people were forced to stay at home and I did try to setup stuff this way. Needless to say, it worked. People were able to see the jobs displayed in the status_monitor.a3x and eventually stop them etc and I myself was able to open the workflow manager and modify workflows etc. There was only one problem: it was really slow. You see, every computer needs access to the db folder, either because it needs to load all the JSON of the workflows in the workflow manager or because it has to read all the job statuses in the status monitor. Now, this is ok, and you could technically use internet for the "clients" that need to just use the status monitor etc, but it's not your use case. Let's get to your use case, then, shall we? Eheheheh Well, in your use case you would have different hosts in different parts of the world accessing the same storage through the internet (perhaps behind a VPN) and this will slow everything down. So much so that if you use things like the A/V Decoder with full decoding, for instance, it will take ages to complete. Now, you might argue that you don't actually need to have a shared global media cache to have a farm and you would be right, it's entirely possible to have a farm that assigns jobs to different hosts without splitting them node by node. In this case you will have a dedicated local cache for each host and only the final files will be delivered to the central location, but still, the delivery of big broadcast files could take ages and most importantly if you have indexing or full decoding it will still need to go through the whole file and it ain't funny if it's a 1.2 TB ProRes UHD masterfile for instance... Now, even if theoretically all those things are overcome and you get on with it, the last thing I would be concerned about is latency. You see, every time you need to perform a task, a ticket is created. Those tickets can be pending, queue, running etc and are picked up and executed by the exe_manager so that the relative task is carried out and eventually a new ticket is created. You see, when a new ticket is created, the hosts in the farm will "rush" to say that they're available and that they can pick it up, but remember, we're using a master-less design, which means that there isn't a master that checks the hosts and assign them tickets, but hosts "pick up" a ticket and mark it as picked up for everyone. This is a good thing, by the way, cause having a master-less design allows us to be totally independent so that jobs can be carried out until the very last host of the farm dies, unlike in a master design in which if the master dies, all the slaves are useless and everything stops. Anyway, I'm digressing. The point is that the whole db and the ticket thing is handled with files and being a file based system the longer it takes for a host to read the tickets and assign them, the worse it is. Besides, given that everything goes through the internet there's a chance that network errors occur, packets are lost, ACKs are not received with data sent again (TCP) or lost forever (UDP) etc and all those things will make it harder for FFAStrans to synchronise everything (although Grandmaster's logic is really really really well done and should be able to sustain terrible network errors/storage disconnections without the blink of an eye).
Anyway, to recap:
- Can it be done?
Yes!
- Is it actually worth it?
Maybe not, but it will depend on you.
Cheers,
Frank
Re: FFAStrans over internet
Welcome to the forum and thanks for using FFAStrans buddhabas!
Your post kind of looks like posts from spammers we have seen in the past, could you maybe talk a little about what you like to do in detail?
Your post kind of looks like posts from spammers we have seen in the past, could you maybe talk a little about what you like to do in detail?
emcodem, wrapping since 2009 you got the rhyme?
Re: FFAStrans over internet
Thanks dear!
I'd like to know if distribuite transcoding Is possibile, maybe using Hamachi
Re: FFAStrans over internet
Very good explaination 
So yes, if you use VPN we actually don't really speak about internet anymore, so the setup will be exactly the same as in any other LAN environment - because you got your servers in the same network. The rest (all the file accessibility stuff that frank mentions above) depends basically on your workflow.
You'd probably have an easier time to write a little script that takes care about the job distribution to the different servers and install ffastrans on each server separately.
So yes, if you use VPN we actually don't really speak about internet anymore, so the setup will be exactly the same as in any other LAN environment - because you got your servers in the same network. The rest (all the file accessibility stuff that frank mentions above) depends basically on your workflow.
You'd probably have an easier time to write a little script that takes care about the job distribution to the different servers and install ffastrans on each server separately.
emcodem, wrapping since 2009 you got the rhyme?