Cloudknot: A Python Library to Run your Existing Code on AWS Batch

Loading

Follow to receive video recommendations   a   A


In this presentation, we introduce Cloudknot, a software library that simplifies cloud-based distributed computing by programmatically executing user-defined functions (UDFs) in AWS Batch. It takes as input a Python function, packages it as a container, creates all the necessary AWS constituent resources to submit jobs, monitors their execution and gathers the results, all from within the Python environment. Cloudknot overcomes limitations of previous similar libraries, such as pywren, that runs UDFs on AWS Lambda, because most data science workloads exceed the AWS Lambda limits on execution time, RAM, and local storage.

Editors Note:

I would like to work with open source projects to create a branch of the tree with all of the best videos for your open source project. Please send me an email if you are interested.