At Netflix, we deal with millions of digital assets every day. Hours of video clips, along with audio, text and image assets are ingested for various purposes. Several workflows are then executed on them; such as inspection, transcoding, editing, logging, etc. These assets can also be used in machine learning workflows, either to train these models, or to get content insights. Not all workflows are applicable to all assets, and some workflows depend on other workflows to run. Additionally, new workflows are introduced regularly, and they need to be executed on existing assets, as well.
We implemented a workflow rule engine that allows users to define rules and conditions to specify the applicable workflows for assets, based on their types, metadata and states. In order to make this system scalable and fault tolerant, we utilize Kafka to send out events on asset state changes (on create, update, workflow completion, etc.) with minimal information in the payload (asset id and version). The rule engine then enriches this payload by fetching additional metadata, evaluates it against the workflow rules, triggers applicable workflows based on the outcome, and monitors their results by listening to the workflow events.
By using a highly available Kafka setup, we can easily scale, handle ETL cases such as migrations, replay messages if needed without impacting asset ingestions.
Presenter
Burak Bacioglu
NetflixAs a Staff Software Engineer at Netflix, I specialize in designing and building scalable distributed systems that enable seamless integrations within the Netflix studio and content ecosystem. With 15+ years of experience in the industry, I have honed my expertise in architecting and implementing high-performance, scalable and reliable systems used by dozens of other engineering teams, such as Asset Management platform, Event Routing service, Workflow Orchestrator, Distributed Audit Logger and Subscription based User Notifications service. Worked with persistent and volatile data stores such as Cassandra, MySQL and EVCache (memcached); messaging frameworks such as Kafka and SQS; search and analytics solutions such as Elasticsearch, Atlas and Tableau. Leveraging my in-depth understanding of cloud technologies, I work closely with cross-functional teams to deliver innovative solutions that meet the complex requirements of the media industry.
Presenter
Meenakshi Jindal
Netflix15+ years of experience in full software development life cycle including architecture, design, prototyping, implementation, and maintenance in various industries like banking, insurance, travel and media. Skilled in designing and implementing Scalable Distributed Cloud Micro-services infrastructure using SOLID coding and 12 factor app principles. Architect of the next generation scalable Asset Management Platform with innovative features required in the competitive media landscape. Passionate about solving cloud distributed systems and NOSQL database challenges. Master in computer application. MS in Information System and Management. Senior Member of IEEE. Associate AWS Certified Solutions Architect.