Full-Stack & Ops

Literature

    2.
    5.
    ​Jez Humble​
      2.
      ​DevOps Handbook.​
      4.
      Lean Enterprise
    6.
    ​Clean code (java) first chapters are good
    7.
      1.
      ​Medium​
      2.
      ​git​
    8.
    10.
    ​pyVideo​

DESIGN PATTERNS

    2.
    ​
    ​

CONTINUOUS INTEGRATION

    1.
    ​Travis​
    2.
    ​Circle CI​
    3.
    ​TeamCity​
    4.
    ​Jenkins​

PACKAGE REPOSITORIES

    1.
    Pypi - public
    2.
    ​Gemfury - private

DEVOPS / SRE

​PMML ​

- an XML file that describes a ML model that is transferable between applications.
    ​PMML uses XML to represent mining models.
    The structure of the models is described by an XML Schema.
    One or more mining models can be contained in a PMML document

​Cap theorem​

Docker

GIT REPOSITORIES

Kubernetes

    For beginners:
      ​1, 2*, 3*, 4, 5*, 6,
    Advanced
      ​1, 2, 3,

Helm

Kubeflow

MiniKF

S2i

    Builds docker images out of gits

AirFlow

    ​Airflow is a platform created by the community to programmatically author, schedule and monitor workflows.

Prefect

Seldon

Tutorials

AWS Lambda

rabbitMQ

​ActiveMQ ​

- Apache ActiveMQβ„’ is the most popular open source, multi-protocol, Java-based messaging serve

Kafka

Also, the routing logic of AMQP can be fairly complicated as opposed to Apache Kafka. For instance, each consumer simply decides which messages to read in Kafka.
In addition to message routing simplicity, there are places where developers and DevOps staff prefer Apache Kafka for its high throughput, scalability, performance, and durability; although, developers still swear by all three systems for various reasons.

Zoo keeper

ELK

Logz.io

Sentry

    ​For python, Your code is telling you more than what your logs let on. Sentry’s full stack monitoring gives you full visibility into your code, so you can catch issues before they become downtime.

Kafka for DS

Redis for DS

    1.
    What is, vs memcached​
    2.
    ​Redis cluster​
    3.
    ​Redis plus spacy​
    4.
    Note: redis is a managed dictionary its strength lies when you have a lot of data that needs to be queries and managed and you don’t want to hard code it, for example.
    5.
    ​Long tutorial​

Statsd

FastAPI

SnowFlake / Redshift

    1.
    Snoflake Intro and demo​
    4.
    ​SF vs RS​
    5.
    ​RS vs BQ​
    6.
    ​SF vs BQ​

Programming Concepts

​Dependency injection - based on SOLID the class should do one thing, so we are letting other classes create 3rd party/class objects for us instead of doing it internally, either by init passing or by injecting in runtime.
​SOLID - the five principles of object oriented.

Visualization

Plotly for jupyter lab β€œjupyter labextension install @jupyterlab/plotly-extension”
    1.
    ​Venn for python​

Serving Models

    1.
    ML SYSTEM DESIGN PATTERNS, res, git​
    2.
    Seldon
    5.
    ​Dapr is a portable, serverless, event-driven runtime that makes it easy for developers to build resilient, stateless and stateful microservices that run on the cloud and edge and embraces the diversity of languages and developer frameworks.
Dapr codifies the best practices for building microservice applications into open, independent, building blocks that enable you to build portable applications with the language and framework of your choice. Each building block is independent and you can use one, some, or all of them in your application.
    1.
    ​

​

MLOPS

    1.
    ​Awesome MLOps​
    1.
    ​MLOps Roadmap​
    3.
    ​State of MLOps​

​
​

    1.
    Metaflow, medium 1 (high level review), 2 (schema), 3, **4 (amazing), 5 (extra), 6 (loading and storing data docs!)

EXPERIMENT MANAGEMENT

    1.
    2.
    Cnvrg.io -
      1.
      Manage - Easily navigate machine learning with dashboards, reproducible data science, dataset organization, experiment tracking and visualization, a model repository and more
      2.
      Build - Run and track experiments in hyperspeed with the freedom to use any compute environment, framework, programming language or tool - no configuration required
      3.
      Automate - Build more models and automate your machine learning from research to production using reusable components and drag-n-drop interface
    3.
    Comet.ml - Comet lets you track code, experiments, and results on ML projects. It’s fast, simple, and free for open source projects.
    4.
    Floyd - notebooks on the cloud, similar to colab / kaggle, etc. gpu costs 4$/h
    5.
    6.
    Missing link - RIP
    7.
    Spark
      2.
      ​What are Rdds?​
      3.
      ​keras , tf, spark​
      5.
      ​Best practices​
    8.
    Databricks
      1.
      ​Koalas - pandas API on Apache Spark
      2.
      ​Intro to DB on spark, has some basic sklearn-like tool and other custom operations such as single-vector-based aggregator for using features as an input to a model
      3.
      ​Pyspark.ml​
      6.
      ​Documentations (read me, has all libraries)
      7.
      ​Medium tutorial, explains the 3 pros of DB with examples of using with native and non native algos
        1.
        Spark sql
        2.
        Mlflow
        3.
        Streaming
        4.
        SystemML DML using keras models.
      10.
      ​Utilizing spark nodes for grid searching with sklearn
        1.
        from spark_sklearn import GridSearchCV
      11.
      ​How can we leverage our existing experience with modeling libraries like scikit-learn? We'll explore three approaches that make use of existing libraries, but still benefit from the parallelism provided by Spark.
These approaches are:
    Grid Search
    Cross Validation
    Sampling (random, chronological subsets of data across clusters)
    1.
    Github spark-sklearn (needs to be compared to what spark has internally)
      1.
      ​Ref: It's worth pausing here to note that the architecture of this approach is different than that used by MLlib in Spark. Using spark-sklearn, we're simply distributing the cross-validation run of each model (with a specific combination of hyperparameters) across each Spark executor. Spark MLlib, on the other hand, will distribute the internals of the actual learning algorithms across the cluster.
      2.
      The main advantage of spark-sklearn is that it enables leveraging the very rich set of machine learning algorithms in scikit-learn. These algorithms do not run natively on a cluster (although they can be parallelized on a single machine) and by adding Spark, we can unlock a lot more horsepower than could ordinarily be used.
      3.
      Using spark-sklearn is a straightforward way to throw more CPU at any machine learning problem you might have. We used the package to reduce the time spent searching and reduce the error for our estimator
    4.
      1.
      ​example​
    5.
      1.
      ​Aws sagemaker​
      2.
      ​Medium and sklearn random trees

A/B testing

    1.
    (highly recommended - buy) trustworthy controlled experiments by Ron, Diane, Ya
    2.
    CUPED- Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data CUPED is an acronym for Controlled experiments Using Pre-Experiment Data [1]. It is a method that aims to estimate treatment effects in A/B tests more accurately than simple differences in means. As reviewed in the previous section, we traditionally use the observed difference between the sample means" by Ron Kohavi, Alex Deng, Ya Xu, Toby Walker​
    3.
    6.
    8.
    (Paper) Top Challenges from the first Practical Online Controlled Experiments Summit - "Online controlled experiments (OCEs), also known as A/B tests, have become ubiquitous in evaluating the impact of changes made to software products and services. While the concept of online controlled experiments is simple, there are many practical challenges in running OCEs at scale. To understand the top practical challenges in running OCEs at scale and encourage further academic and industrial exploration, representatives with experience in large-scale experimentation from thirteen different organizations (Airbnb, Amazon, Booking.com, Facebook, Google, LinkedIn, Lyft, Microsoft, Netflix, Twitter, Uber, Yandex, and Stanford University) were invited to the first Practical Online Controlled Experiments Summit. All thirteen organizations sent representatives. Together these organizations have tested more than one hundred thousand experiment treatments last year. Thirty-four experts from these organizations participated in the summit in Sunnyvale, CA, USA on December 13-14, 2018. While there are papers from individual organizations on some of the challenges and pitfalls in running OCEs at scale, this is the first paper to provide the top challenges faced across the industry for running OCEs at scale and some common solutions"
    9.
    (book) modern epidemiology - has a/b testing
    10.
    ​Challenges, Best Practices and Pitfalls in Evaluating Results of Online Controlled Experiments - A/B Testing is the gold standard to estimate the causal relationship between a change in a product and its impact on key outcome measures. It is widely used in the industry to test changes ranging from simple copy change or UI change to more complex changes like using machine learning models to personalize user experience. The key aspect of A/B testing is evaluation of experiment results. Designing the right set of metrics - correct outcome measures, data quality indicators, guardrails that prevent harm to business, and a comprehensive set of supporting metrics to understand the β€œwhy” behind the key movements is the #1 challenge practitioners face when trying to scale their experimentation program. On the technical side, improving sensitivity of experiment metrics is a hard problem and an active research area, with large practical implications as more and more small and medium size businesses are trying to adopt A/B testing and suffer from insufficient power. In this tutorial we will discuss challenges, best practices, and pitfalls in evaluating experiment results, focusing on both lessons learned and practical guidelines as well as open research questions.
    Slides:
    11.
    ​Increasing experimentation accuracy and speed by using control variates - In this article, we share details about our team’s journey to bring the statistical method known as CUPED to Etsy, and how it is now helping other teams make more informed product decisions, as well as shorten the duration of their experiments by up to 20%. We offer some perspectives on what makes such a method possible, what it took us to implement it at scale, and what lessons we have learned along the way.
    13.
    (good) How to Double A/B Testing Speed with CUPED - Microsoft’s variance reduction that’s becoming industry standard. "Controlled-experiment Using Pre-Existing Data (CUPED) is a variance reduction technique created by Microsoft in 2013. Since then, it has been implemented at Netflix, Booking.com, BBC, and many others.
    In short, CUPED uses pre-experiment data to control for natural variation in an experiment’s north star metric. Be removing natural variation, we can run statistical tests that require a smaller sample size. CUPED can be added to virtually any A/B testing framework; it’s computationally efficient and fairly straightforward to code."
    16.
    (BBC) Increasing experimental power with variance reduction at the BBC - This article discusses how the Experimentation team have been accounting for pre-experiment variance in order to increase the statistical power of their experiments

A/B Testing Tools

    1.
    ​R-shiny Tool​
    2.
    ​contextualbandits.com - adapt your creative according to context and outcomes automatically without A/B-sitting your campaigns.
​
Last modified 1mo ago