A "Hello, world!" Prefect flow running on an ephemeral Dask Cluster on Kubernetes.
The Prefect Core docs suggest running flows on a Dask cluster via Dask Distributed and an article from the MET Office Informatics Lab demonstrates running an adaptive Dask cluster on k8s. This example is inspired by those sources, as well as the respective docs for the technologies used.
| (() => { | |
| // Run this in Chrome's console, while in Bitbucket's website and logged in | |
| const csrftoken = document.cookie.match(/\bcsrftoken=(.*?)(?:;| |$)/)[1]; | |
| const repoName = window.__initial_state__.section.repository.currentRepository.full_name; | |
| const expiry = 1000 * 60 * 60; // Delete only files older than an hour | |
| let page = 1; | |
| function iterateNext() { | |
| fetch(`https://bitbucket.org/${repoName}/admin/lfs/file-management/?iframe=true&spa=0&page=${page}`, { |
| ''' Script for downloading all GLUE data. | |
| Note: for legal reasons, we are unable to host MRPC. | |
| You can either use the version hosted by the SentEval team, which is already tokenized, | |
| or you can download the original data from (https://download.microsoft.com/download/D/4/6/D46FF87A-F6B9-4252-AA8B-3604ED519838/MSRParaphraseCorpus.msi) and extract the data from it manually. | |
| For Windows users, you can run the .msi file. For Mac and Linux users, consider an external library such as 'cabextract' (see below for an example). | |
| You should then rename and place specific files in a folder (see below for an example). | |
| mkdir MRPC | |
| cabextract MSRParaphraseCorpus.msi -d MRPC |
| # from the guide https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html | |
| # and from other resources found, trying to achieve a good classifier based on Inveption V3 pre-trained netfork | |
| from keras.applications.inception_v3 import InceptionV3 | |
| from keras.preprocessing import image | |
| from keras.models import Model | |
| from keras.layers import Dense, GlobalAveragePooling2D | |
| from keras.preprocessing.image import ImageDataGenerator | |
| from keras import backend as K | |
| from keras.callbacks import ModelCheckpoint |
| import numpy as np | |
| from keras import backend as K | |
| from keras.models import Sequential | |
| from keras.layers.core import Dense, Dropout, Activation, Flatten | |
| from keras.layers.convolutional import Convolution2D, MaxPooling2D | |
| from keras.preprocessing.image import ImageDataGenerator | |
| from sklearn.metrics import classification_report, confusion_matrix | |
| #Start | |
| train_data_path = 'F://data//Train' |
in case of processing a very large array e.g. Promise.all(A_VERY_LARGE_ARRAY_OF_XHR_PROMISE)
which would probably blow you browser memory by trying to send all requests at the same time
solution is limit the concurrent of requests, and wrap promise in thunk
Promise.allConcurrent(2)([()=>fetch('BLAH1'), ()=>fetch('BLAH2'),...()=>fetch('BLAHN')])
| import numpy as np | |
| #input is a RGB numpy array with shape (height,width,3), can be uint,int, float or double, values expected in the range 0..255 | |
| #output is a double YUV numpy array with shape (height,width,3), values in the range 0..255 | |
| def RGB2YUV( rgb ): | |
| m = np.array([[ 0.29900, -0.16874, 0.50000], | |
| [0.58700, -0.33126, -0.41869], | |
| [ 0.11400, 0.50000, -0.08131]]) |
| /** | |
| * List unique CSS properties for all DOM elements | |
| * Initially created to list unique font stacks on a page | |
| * @see {@link http://stackoverflow.com/a/35022690/ Inspired by this StackOverflow answer} | |
| * | |
| * @see {@link https://gist.github.com/macbookandrew/f33dbbc0aa582d0515919dc5fb95c00a/ URL for this file} | |
| * | |
| * @author AndrewRMinion Design (https://andrewrminion.com) | |
| * @version 1.1 | |
| * |
| /***************************************************************** | |
| * onMessage from the extension or tab (a content script) | |
| *****************************************************************/ | |
| chrome.runtime.onMessage.addListener( | |
| function(request, sender, sendResponse) { | |
| if (request.cmd == "any command") { | |
| sendResponse({ result: "any response from background" }); | |
| } else { | |
| sendResponse({ result: "error", message: `Invalid 'cmd'` }); | |
| } |
