Engineering blog

In this post, we look into using Spark as a way to speed up feature extraction and data preprocessing for ML models.

This post examines a straightforward way of running a parallel pipeline with the resources at hand, using a simple and established tool: Make

BenevolentAI utilizes various API technologies to enable cross-team collaboration and automation. This blog discusses those and provides general guidelines for designing well-engineered and user-friendly APIs.

Our team discusses how spark and kubernetes are used to ingest, normalize and process millions of scientific papers for NLP.

At BenevolentAI we work with a Kubernetes based infrastructure that is completely containerized. Using Kubernetes we run many sophisticated workflows for data processing, model training, model serving and metrics and evaluation.

MetalLB is a software load-balancer for bare metal Kubernetes clusters. In this post we will go through the process by which we deployed MetalLB to our production cluster at BenevolentAI.

BenevolentAI’s System Software Engineer, Rory, explores the benefits of pair working on research and decision making. In software development, this collaborative way of working can help build team support and cohesion, and ultimately help teams to excel.