How Netflix Directs 1/3rd of Internet Traffic

This document summarizes Haitao Wang's experience working on streaming platforms at Alibaba and Microsoft. It describes Alibaba's data infrastructure challenges in handling large volumes of streaming data. It introduces Alibaba Blink, a distribution of Apache Flink that was developed to meet Alibaba's scale needs. Blink has achieved unprecedented throughput of 472 million events per second with latency of 10s of milliseconds. The document outlines improvements made in Blink's runtime, declarative SQL support, and use cases at Alibaba including real-time A/B testing, search index building, and online machine learning.

ApacheCon Las Vegas 2019 September 9-12 Beam Summit 20th anniversary of the Apache Software Foundation

18 aug2021 Continuous SQL with Apache Streaming (FLaNK and FLiP) https://emamo.com/event/worldfestival-2021/s/pro-talk-continuous-sql-with-flink-WR115a In this talk, I will walk through how someone can set up and run continuous SQL queries against Pulsar topics utilizing Apache Flink. We will walk through creating Pulsar topics, schemas and publishing data. We will then cover consuming Pulsar data, joining Pulsar topics and inserting new events into Pulsar topics as they arrive. This basic overview will show hands-on techniques, tips and examples of how to do this using Pulsar tools. https://github.com/tspannhw/FLiP-IoT https://github.com/tspannhw/SpeakerProfile/tree/main/2021/talks

This presentation was given to the engineering organization at Zendesk. In this presentation, I talk about the challenges that the Netflix API faces in supporting the 1000+ different device types, millions of users, and billions of transactions. The topics range from resiliency, scale, API design, failure injection, continuous delivery, and more.

New Features in Confluent Platform 6.0 / Apache Kafka 2.6, including REST Proxy and API, Tiered Storage for AWS S3 and GCP GCS, Cluster Linking (On-Premise, Edge, Hybrid, Multi-Cloud), Self-Balancing Clusters), ksqlDB.

This document discusses developing an advanced visualization tool for Flink and Spark jobs that provides insight into data characteristics and the physical execution plan. It aims to help developers detect issues, understand distributed systems, and guide testing of adaptive partitioning techniques. The tool enhances existing metrics and APIs to visualize input/output patterns and physical tasks/subtasks. Future plans include public beta release and integrating dynamic repartitioning to mitigate data skew.

http://flink-forward.org/kb_sessions/faster-and-furiouser-flink-drift/ Not long ago, we had the opportunity to test Apache Flink to see just how fast it would go on a moderately realistic task with fast hardware and with a good streaming transport layer underneath. Our goal was not so much careful comparison with other software, but flat-out speed, Flink against Flink. In the process, we learned a lot about what it takes to go fast. Some of the lessons were ones that we had “learned” a number of times before: – the bottleneck isn’t where you thought it was – copying data is expensive – context switches are expensive – measure twice, cut once But there were some real surprises along the way. The really important knobs weren’t quite what people say you should turn. One of the biggest surprises was the degree to which high performance libraries have threading built into them which makes the actual concurrrency much higher than the apparent concurrency. The result was that at least one cluster parameter needed to be adjusted by 30x to get real

In the era of cloud generation, the constant activity around workloads and containers create more vulnerabilities than an organization can keep up with. Using legacy security vendors doesn't set you up for success in the cloud. You’re likely spending undue hours chasing, triaging and patching a countless stream of cloud vulnerabilities with little prioritization. Join us for this live webinar as we detail how to streamline host and container vulnerability workflows for your software teams wanting to build fast in the cloud. We'll be covering how to: Get visibility into active packages and associated vulnerabilities Reduce false positives by 98% Reduce investigation time by 30% Spot a legacy vendor looking to do some cloud washing

Is your service architecture and engineering velocity constrained by organizational concerns? Does it seem impossible to give priority to key initiatives regardless of intent? Are engineers switching tasks so often that they are just treading water? Are critical projects endlessly backlogged? Has staffing up pushed the limits of your team structure? Navigating through challenges like these can be daunting and solutions fraught with uncertainty. How do you know what, where, when to change. And whatever the answer is today it will most certainly vary over time. Effective organizations evolve, at key inflection points, to support critical business and technical goals. There is not only a strong relationship between organizations and the software they produce (Conway’s Law) but many organizational solutions can be derived from analogs in the technical realm. In other words, we can treat organizational improvement as a refactoring exercise. Over the last 20 years Netflix engineering has proven time and again an ability to adapt and grow, resulting in undisputed dominance over the global internet tv market. In this talk we’ll use Netflix as a case study to illustrate how specific strategies, framed as technical analogs, have been employed to maximize engineering agility, velocity, and impact. These powerful, yet simple strategies and solutions provide a useful blueprint for organizational success.