Keras + Hyperopt: A very simple wrapper for convenient hyperparameter optimization

This project is now archived. It's been fun working on it, but it's time for me to move on. Thank you for all the support and feedback over the last couple of years. If someone is interested in taking ownership, let's discuss. ✌️

Hyperas Build Status PyPI version

A very simple convenience wrapper around hyperopt for fast prototyping with keras models. Hyperas lets you use the power of hyperopt without having to learn the syntax of it. Instead, just define your keras model as you are used to, but use a simple template notation to define hyper-parameter ranges to tune.

Installation

pip install hyperas

Quick start

Assume you have data generated as such

def data():
    x_train = np.zeros(100)
    x_test = np.zeros(100)
    y_train = np.zeros(100)
    y_test = np.zeros(100)
    return x_train, y_train, x_test, y_test

and an existing keras model like the following

def create_model(x_train, y_train, x_test, y_test):
    model = Sequential()
    model.add(Dense(512, input_shape=(784,)))
    model.add(Activation('relu'))
    model.add(Dropout(0.2))
    model.add(Dense(512))
    model.add(Activation('relu'))
    model.add(Dropout(0.2))
    model.add(Dense(10))
    model.add(Activation('softmax'))

    # ... model fitting

    return model

To do hyper-parameter optimization on this model, just wrap the parameters you want to optimize into double curly brackets and choose a distribution over which to run the algorithm.

In the above example, let's say we want to optimize for the best dropout probability in both dropout layers. Choosing a uniform distribution over the interval [0,1], this translates into the following definition. Note that before returning the model, to optimize, we also have to define which evaluation metric of the model is important to us. For example, in the following, we optimize for accuracy.

Note: In the following code we use 'loss': -accuracy, i.e. the negative of accuracy. That's because under the hood hyperopt will always minimize whatever metric you provide. If instead you want to actually want to minimize a metric, say MSE or another loss function, you keep a positive sign (e.g. 'loss': mse).

from hyperas.distributions import uniform

def create_model(x_train, y_train, x_test, y_test):
    model = Sequential()
    model.add(Dense(512, input_shape=(784,)))
    model.add(Activation('relu'))
    model.add(Dropout({{uniform(0, 1)}}))
    model.add(Dense(512))
    model.add(Activation('relu'))
    model.add(Dropout({{uniform(0, 1)}}))
    model.add(Dense(10))
    model.add(Activation('softmax'))

    # ... model fitting

    score = model.evaluate(x_test, y_test, verbose=0)
    accuracy = score[1]
    return {'loss': -accuracy, 'status': STATUS_OK, 'model': model}

The last step is to actually run the optimization, which is done as follows:

best_run = optim.minimize(model=create_model,
                          data=data,
                          algo=tpe.suggest,
                          max_evals=10,
                          trials=Trials())

In this example we use at most 10 evaluation runs and the TPE algorithm from hyperopt for optimization.

Check the "complete example" below for more details.

Complete example

Note: It is important to wrap your data and model into functions as shown below, and then pass them as parameters to the minimizer. data() returns the data the create_model() needs. An extended version of the above example in one script reads as follows. This example shows many potential use cases of hyperas, including:

  • Varying dropout probabilities, sampling from a uniform distribution
  • Different layer output sizes
  • Different optimization algorithms to use
  • Varying choices of activation functions
  • Conditionally adding layers depending on a choice
  • Swapping whole sets of layers
from __future__ import print_function
import numpy as np

from hyperopt import Trials, STATUS_OK, tpe
from keras.datasets import mnist
from keras.layers.core import Dense, Dropout, Activation
from keras.models import Sequential
from keras.utils import np_utils

from hyperas import optim
from hyperas.distributions import choice, uniform


def data():
    """
    Data providing function:

    This function is separated from create_model() so that hyperopt
    won't reload data for each evaluation run.
    """
    (x_train, y_train), (x_test, y_test) = mnist.load_data()
    x_train = x_train.reshape(60000, 784)
    x_test = x_test.reshape(10000, 784)
    x_train = x_train.astype('float32')
    x_test = x_test.astype('float32')
    x_train /= 255
    x_test /= 255
    nb_classes = 10
    y_train = np_utils.to_categorical(y_train, nb_classes)
    y_test = np_utils.to_categorical(y_test, nb_classes)
    return x_train, y_train, x_test, y_test


def create_model(x_train, y_train, x_test, y_test):
    """
    Model providing function:

    Create Keras model with double curly brackets dropped-in as needed.
    Return value has to be a valid python dictionary with two customary keys:
        - loss: Specify a numeric evaluation metric to be minimized
        - status: Just use STATUS_OK and see hyperopt documentation if not feasible
    The last one is optional, though recommended, namely:
        - model: specify the model just created so that we can later use it again.
    """
    model = Sequential()
    model.add(Dense(512, input_shape=(784,)))
    model.add(Activation('relu'))
    model.add(Dropout({{uniform(0, 1)}}))
    model.add(Dense({{choice([256, 512, 1024])}}))
    model.add(Activation({{choice(['relu', 'sigmoid'])}}))
    model.add(Dropout({{uniform(0, 1)}}))

    # If we choose 'four', add an additional fourth layer
    if {{choice(['three', 'four'])}} == 'four':
        model.add(Dense(100))

        # We can also choose between complete sets of layers

        model.add({{choice([Dropout(0.5), Activation('linear')])}})
        model.add(Activation('relu'))

    model.add(Dense(10))
    model.add(Activation('softmax'))

    model.compile(loss='categorical_crossentropy', metrics=['accuracy'],
                  optimizer={{choice(['rmsprop', 'adam', 'sgd'])}})

    result = model.fit(x_train, y_train,
              batch_size={{choice([64, 128])}},
              epochs=2,
              verbose=2,
              validation_split=0.1)
    #get the highest validation accuracy of the training epochs
    validation_acc = np.amax(result.history['val_acc']) 
    print('Best validation acc of epoch:', validation_acc)
    return {'loss': -validation_acc, 'status': STATUS_OK, 'model': model}


if __name__ == '__main__':
    best_run, best_model = optim.minimize(model=create_model,
                                          data=data,
                                          algo=tpe.suggest,
                                          max_evals=5,
                                          trials=Trials())
    X_train, Y_train, X_test, Y_test = data()
    print("Evalutation of best performing model:")
    print(best_model.evaluate(X_test, Y_test))
    print("Best performing model chosen hyper-parameters:")
    print(best_run)

FAQ

Here is a list of a few popular errors

TypeError: require string label

You're probably trying to execute the model creation code, with the templates, directly in python. That fails simply because python cannot run the templating in the braces, e.g. {{uniform..}}. The def create_model(...) function is in fact not a valid python function anymore.

You need to wrap your code in a def create_model(...): ... function, and then call it from optim.minimize(model=create_model,... like in the example.

The reason for this is that hyperas works by doing template replacement of everything in the {{...}} into a separate temporary file, and then running the model with the replaced braces (think jinja templating).

This is the basis of how hyperas simplifies usage of hyperopt by being a "very simple wrapper".

TypeError: 'generator' object is not subscriptable

This is currently a known issue.

Just pip install networkx==1.11

NameError: global name 'X_train' is not defined

Maybe you forgot to return the x_train argument in the def create_model(x_train...) call from the def data(): ... function.

You are not restricted to the same list of arguments as in the example. Any arguments you return from data() will be passed to create_model()

notebook adjustment

If you find error like "No such file or directory" or OSError, Err22, you may need add notebook_name='simple_notebook'(assume your current notebook name is simple_notebook) in optim.minimize function like this:

best_run, best_model = optim.minimize(model=model,
                                      data=data,
                                      algo=tpe.suggest,
                                      max_evals=5,
                                      trials=Trials(),
                                      notebook_name='simple_notebook')

How does hyperas work?

All we do is parse the data and model templates and translate them into proper hyperopt by reconstructing the space object that's then passed to fmin. Most of the relevant code is found in optim.py and utils.py.

How to read the output of a hyperas model?

Hyperas translates your script into hyperopt compliant code, see here for some guidance on how to interpret the result.

How to pass arguments to data?

Suppose you want your data function take an argument, specify it like this using positional arguments only (not keyword arguments):

import pickle
def data(fname):
    with open(fname,'rb') as fh:
        return pickle.load(fh)

Note that your arguments must be implemented such that repr can show them in their entirety (such as strings and numbers). If you want more complex objects, use the passed arguments to build them inside the data function.

And when you run your trials, pass a tuple of arguments to be substituted in as data_args:

best_run, best_model = optim.minimize(
    model=model,
    data=data,
    algo=tpe.suggest,
    max_evals=64,
    trials=Trials(),
    data_args=('my_file.pkl',)
)

What if I need more flexibility loading data and adapting my model?

Hyperas is a convenience wrapper around Hyperopt that has some limitations. If it's not convenient to use in your situation, simply don't use it -- and choose Hyperopt instead. All you can do with Hyperas you can also do with Hyperopt, it's just a different way of defining your model. If you want to squeeze some flexibility out of Hyperas anyway, take a look here.

Running hyperas in parallel?

You can use hyperas to run multiple models in parallel with the use of mongodb (which you'll need to install and setup users for). Here's a short example using MNIST:

  1. Copy and modify examples/mnist_distributed.py (bump up max_evals if you like):

  2. Run python mnist_distributed.py. It will create a temp_model.py file. Copy this file to any machines that will be evaluating models. It will then begin waiting for evaluation results

  3. On your other machines (make sure they have a python installed with all your dependencies, ideally with the same versions) run:

    export PYTHONPATH=/path/to/temp_model.py
    hyperopt-mongo-worker --exp-key='mnist_test' --mongo='mongo://username:[email protected]:27017/jobs'
  4. Once max_evals have been completed, you should get an output with your best model. You can also look through your mongodb and examine the results, to get the best model out and run it, do:

    from pymongo import MongoClient
    from keras.models import load_model
    import tempfile
    c = MongoClient('mongodb://username:[email protected]:27017/jobs')
    best_model = c['jobs']['jobs'].find_one({'exp_key': 'mnist_test'}, sort=[('result.loss', -1)])
    temp_name = tempfile.gettempdir()+'/'+next(tempfile._get_candidate_names()) + '.h5'
    with open(temp_name, 'wb') as outfile:
        outfile.write(best_model['result']['model_serial'])
    model = load_model(temp_name)
Owner
Max Pumperla
Data Science Professor, Data Scientist & Engineer. DL4J core developer, Hyperopt maintainer, Keras contributor. Author of "Deep Learning and the Game of Go"
Max Pumperla
Comments
  • Unable to run example code - TypeError('require string label')

    Unable to run example code - TypeError('require string label')

    Hi,

    I am unable to run the first hyperas example. Specifically this line:

    >>> model.add(Dropout({{uniform(0, 1)}}))
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/Users/brohoro/Documents/PyProjects/envs/tensorflow/lib/python2.7/site-packages/hyperopt/pyll_utils.py", line 21, in wrapper
        raise TypeError('require string label')
    

    The example without hyperas runs fine.

    Here is my env

    Python 2.7.13 Name: Keras Version: 2.0.4 Name: hyperas Version: 0.3 Name: hyperopt Version: 0.1

    I have tried this both via Jupyter and via the python interpreter.

    Thanks

  • object of type 'NoneType' has no len()

    object of type 'NoneType' has no len()

    First time try with Hyperas. I get an error: object of type 'NoneType' has no len() at trials=Trials()in the optim.minimize(...) call. Not sure what this means. I am trying to optimize a network that takes three inputs in the form of a list X_train[], so X_train[0] has a set of input data, X_train[1] and X_train_[2] also. Not sure if that could be the cause for this error? Don't know what Hyperas can handle.

    Here's some code:

    def data(): 
        # load VGG16 output data, X_train and X_valid are lists with data channels
        X_train, Y_train, X_valid, Y_valid = load_VGG16_output(data_type='train', num_channels=3)
        return X_train, Y_train, X_valid, Y_valid
    
    def model(X_train, Y_train, X_valid, Y_valid):
    
        in_0 = Input(shape=X_train[0].shape[1:])    
        x = Dense(256, activation='relu')(in_0)
        x = Dropout(0.5)(x)
    
        in_1 = Input(shape=X_train[1].shape[1:])
        y = Dense(64, activation='relu')(in_1) #, W_regularizer=l2(0.01))(in_1)
        y = Dropout(0.5)(y)
    
        in_2 = Input(X_train[2].shape[1:])
        z = Dense(32, activation='relu')(in_2) #, W_regularizer=l2(0.01))(in_2)
        z = Dropout(0.5)(z)
    
        x = merge([x, y, z], mode='concat', concat_axis=1)
        
        predict = Dense(1, activation='sigmoid')(x)
        model = Model(input=[in_0, in_1, in_2], output=predict)
       
        optimizer=Adam(lr=1e-4, decay=0.003)
    
        model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy'])
    
        # load the class weight to balance the input positive / negative scores
        class_weight = load_class_weight()
    
        model.fit(X_train, Y_train,
                  batch_size={{choice([32, 64, 128])}}, # optimize batch size
                  nb_epoch=1, 
                  verbose=2, 
                  class_weight=class_weight,
                  validation_data=(X_valid, Y_valid))
    
        score, acc = model.evaluate(X_valid, Y_valid, verbose=0)
        print('Valid accuracy: ', acc)
        return {'loss': -acc, 'status': STATUS_OK, 'model': model}
    
    if __name__ == '__main__':
        
        X_train, Y_train, X_valid, Y_valid = data()
    
        best_run, best_model = optim.minimize(model=model,
                                              data=data,
                                              algo=tpe.suggest,
                                              max_evals=5,
                                              trials=Trials())
           
        print('Evalutation of best performing model: ')
        print(best_model.evaluate(X_valid, Y_valid))
    
  • autoload_pylab_o problem with hyperopt

    autoload_pylab_o problem with hyperopt

    I have recently installed hyperas, but when I run the example code, it raise an error, I have trying to fix it, but not work yet, anyone can help me ? Below is the error info:

    Unexpected error: <type 'exceptions.NameError'>
    Using Theano backend.
    Traceback (most recent call last):
    
      File "<ipython-input-1-570b570a6029>", line 1, in <module>
        runfile('C:/SciSoft/my_code/test_hyperas.py', wdir='C:/SciSoft/my_code')
    
      File "C:\SciSoft\WinPython-64bit-2.7.9.4\python-2.7.9.amd64\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 682, in runfile
        execfile(filename, namespace)
    
      File "C:\SciSoft\WinPython-64bit-2.7.9.4\python-2.7.9.amd64\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 71, in execfile
        exec(compile(scripttext, filename, 'exec'), glob, loc)
    
      File "C:/SciSoft/my_code/test_hyperas.py", line 211, in <module>
        trials=Trials())
    
      File "c:\scisoft\winpython-64bit-2.7.9.4\python-2.7.9.amd64\hyperas-\hyperas\optim.py", line 31, in minimize
        best_run = base_minimizer(model, data, algo, max_evals, trials, rseed)
    
      File "c:\scisoft\winpython-64bit-2.7.9.4\python-2.7.9.amd64\hyperas-\hyperas\optim.py", line 96, in base_minimizer
        from temp_model import keras_fmin_fnct, get_space
    
      File "temp_model.py", line 8, in <module>
        spy_cfg.IPKernelApp.pylab_import_all = autoload_pylab_o
    
    NameError: name 'autoload_pylab_o' is not defined
    
  • hyperas on LSTM

    hyperas on LSTM

    Hi,

    I am using keras 0.3.2 and trying a simple LSTM model with hyperas. But I get the following error. Anything wrong ?

    Exception: Layer is not connected. Did you forget to set "input_shape"?

    model = Sequential()
    model.add(LSTM(input_dim=388, output_dim=300, return_sequences=False)) model.add(Dropout({{uniform(0, 1)}})) model.add(Dense(output_dim=1))
    model.add(Activation("linear"))
    model.compile(loss="mean_squared_error", optimizer="rmsprop")

  • Run Hyperas with Siamese network

    Run Hyperas with Siamese network

    I need to run the hyperas for optimizing my siamese network, But I am unable to do so. Following is my set up codes, and the error. BRANCHES OF SIAMESE:

    def create_base_network(input_shape): random_seed = 7

    model = Input(shape=input_shape)
    x = Conv2D(16, (3, 3), padding="same",activation='relu', data_format='channels_first',
               kernel_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed),
              bias_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed))(model)
    
    x = Conv2D(16, (3, 3), padding="same", activation='relu',
               kernel_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed),
              bias_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed))(x)
    x = MaxPooling2D(pool_size=(2, 2),strides=(2, 2))(x)
    
    x = Conv2D(32, (3, 3), padding="same", activation='relu',
               kernel_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed),
              bias_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed))(x)
    
    x = Conv2D(32, (3, 3), padding="same", activation='relu',
               kernel_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed),
              bias_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed))(x)
    
    x = MaxPooling2D(pool_size=(2, 2),strides=(2, 2))(x)
    x = Flatten()(x)
    x = Dense({{choice([512,96,1024,2048,4096])}}, kernel_initializer=keras.initializers.he_normal(seed=random_seed),activation='relu')(x)
    Dropout({{uniform(0, 1)}})
    x = Dense({{choice([512,96,1024,2048,4096])}}, kernel_initializer=keras.initializers.he_normal(seed=random_seed),activation='relu')(x)
    Dropout({{uniform(0, 1)}})
    x = Dense({{choice([512,96,1024,2048,4096])}}, kernel_initializer=keras.initializers.he_normal(seed=random_seed),activation='relu')(x)
    
    return Model(model,x)
    

    HOW I CREATE THE MODEL def create_model(x_train1, x_train2, y_train,x_val1, x_val2, y_val, x_test1, x_test2, y_test): epochs = 30 input_shape = (1,96,96) patience_ = 1

    base_network = create_base_network(input_shape)
    
    input_a = Input(shape=input_shape)
    input_b = Input(shape=input_shape)
    
    processed_a = base_network(input_a)
    processed_b = base_network(input_b)
    
    opt = Adam(lr={{choice([1e-3,1e-4,1e-5])}})
    distance = Lambda(euclidean_distance,
                      output_shape=eucl_dist_output_shape)([processed_a, processed_b])
    model = Model([input_a, input_b], distance)
    model.compile(loss=contrastive_loss_altered, optimizer=opt, metrics=["accuracy"])
    
    
    es = EarlyStopping(monitor='val_acc', patience=patience_,verbose=1)
    checkpointer = ModelCheckpoint(filepath='keras_weights.hdf5',
                                   verbose=1,
                                   save_best_only=True)
    model.fit([x_train1, x_train2],y_train,
          batch_size={{choice([32,64,128])}},
          epochs=epochs,
          validation_data=([x_val1,x_val2], y_val),
          callbacks=[es],
          verbose=0)
    
    score, acc = model.evaluate([x_test1, x_test2], y_test)
    

    MAIN:

    if name == 'main':

    x_train1, x_train2, y_train,x_val1, x_val2, y_val, x_test1, x_test2, y_test = data()
    
    best_run, best_model = optim.minimize(model=create_model, data=data,
    functions = [process_data,create_base_network,euclidean_distance,eucl_dist_output_shape,contrastive_loss_alter$
    algo=tpe.suggest,max_evals=1,trials=Trials())
    

    THEN THE ERROR I AM GETTING: Error:

    Using TensorFlow backend. Traceback (most recent call last): File "hyperas_contrastive_loss.py", line 152, in algo=tpe.suggest,max_evals=1,trials=Trials()) File "/home/amalli2s/anaconda3/envs/iwin/lib/python3.6/site-packages/hyperas/optim.py", line 67, in minimize verbose=verbose) File "/home/amalli2s/anaconda3/envs/iwin/lib/python3.6/site-packages/hyperas/optim.py", line 133, in base_minimizer return_argmin=True), File "/home/amalli2s/anaconda3/envs/iwin/lib/python3.6/site-packages/hyperopt/fmin.py", line 307, in fmin return_argmin=return_argmin, File "/home/amalli2s/anaconda3/envs/iwin/lib/python3.6/site-packages/hyperopt/base.py", line 635, in fmin return_argmin=return_argmin) File "/home/amalli2s/anaconda3/envs/iwin/lib/python3.6/site-packages/hyperopt/fmin.py", line 320, in fmin rval.exhaust() File "/home/amalli2s/anaconda3/envs/iwin/lib/python3.6/site-packages/hyperopt/fmin.py", line 199, in exhaust self.run(self.max_evals - n_done, block_until_done=self.async) File "/home/amalli2s/anaconda3/envs/iwin/lib/python3.6/site-packages/hyperopt/fmin.py", line 173, in run self.serial_evaluate() File "/home/amalli2s/anaconda3/envs/iwin/lib/python3.6/site-packages/hyperopt/fmin.py", line 92, in serial_evaluate result = self.domain.evaluate(spec, ctrl) File "/home/amalli2s/anaconda3/envs/iwin/lib/python3.6/site-packages/hyperopt/base.py", line 840, in evaluate rval = self.fn(pyll_rval) File "/home/amalli2s/thesis/keras/temp_model.py", line 222, in keras_fmin_fnct File "/home/amalli2s/thesis/keras/temp_model.py", line 192, in create_base_network File "/home/amalli2s/anaconda3/envs/iwin/lib/python3.6/site-packages/hyperopt/pyll_utils.py", line 21, in wrapper raise TypeError('require string label') TypeError: require string label

    This error seems to be popular, but i don't see a specific solution or guide line to fix it yet. Can any one help me with this issue?

  • OSError while executing sample code

    OSError while executing sample code

    The sample code given on hyperas homepage ends up in the following error stack trace:

    OSError                                   Traceback (most recent call last)
    <ipython-input-15-871ae3941e46> in <module>()
         76                                           algo=tpe.suggest,
         77                                           max_evals=5,
    ---> 78                                           trials=Trials())
         79     X_train, Y_train, X_test, Y_test = data()
         80     print("Evalutation of best performing model:")
    
    G:\Programs\anaconda3\envs\py35\lib\site-packages\hyperas\optim.py in minimize(model, data, algo, max_evals, trials, functions, rseed, notebook_name, verbose, eval_space, return_space)
         65                                      full_model_string=None,
         66                                      notebook_name=notebook_name,
    ---> 67                                      verbose=verbose)
         68 
         69     best_model = None
    
    G:\Programs\anaconda3\envs\py35\lib\site-packages\hyperas\optim.py in base_minimizer(model, data, functions, algo, max_evals, trials, rseed, full_model_string, notebook_name, verbose, stack)
         94         model_str = full_model_string
         95     else:
    ---> 96         model_str = get_hyperopt_model_string(model, data, functions, notebook_name, verbose, stack)
         97     temp_file = './temp_model.py'
         98     write_temp_files(model_str, temp_file)
    
    G:\Programs\anaconda3\envs\py35\lib\site-packages\hyperas\optim.py in get_hyperopt_model_string(model, data, functions, notebook_name, verbose, stack)
        176     else:
        177         calling_script_file = os.path.abspath(inspect.stack()[stack][1])
    --> 178         with open(calling_script_file, 'r') as f:
        179             source = f.read()
        180 
    
    OSError: [Errno 22] Invalid argument: 'path\\to\\working\\directory\\<ipython-input-15-871ae3941e46>'
    

    I am now able to understand the cause of the error. I am using Python 3.5.3 (Anaconda virtual environment) on Windows 10. Please help me resolve it. If I am creating a redundant issue, I apologise for the same.

  • IndentationError: unexpected indent

    IndentationError: unexpected indent

    Hey, I just installed on python 3.4.2 and I'm using the PyCharm IDE. I've opened the example file in my ide and attempted to run it. The following error occurs:

    Using Theano backend.
    Traceback (most recent call last):
      File "D:/Dropbox/Dropbox/PhD/My Work/Code/Paper 2/src/temp.py", line 77, in <module>
        trials=Trials())
      File "C:\Anaconda3\envs\py34\lib\site-packages\hyperas\optim.py", line 42, in minimize
        notebook_name=notebook_name, verbose=verbose)
      File "C:\Anaconda3\envs\py34\lib\site-packages\hyperas\optim.py", line 62, in base_minimizer
        model_str = get_hyperopt_model_string(model, data, notebook_name, verbose, stack)
      File "C:\Anaconda3\envs\py34\lib\site-packages\hyperas\optim.py", line 132, in get_hyperopt_model_string
        imports = extract_imports(cleaned_source, verbose)
      File "C:\Anaconda3\envs\py34\lib\site-packages\hyperas\utils.py", line 31, in extract_imports
        tree = ast.parse(source)
      File "C:\Anaconda3\envs\py34\lib\ast.py", line 35, in parse
        return compile(source, filename, mode, PyCF_ONLY_AST)
      File "<unknown>", line 35
        if conditional({{choice(['three', 'four'])}}) == 'four':
        ^
    IndentationError: unexpected indent
    

    I understand that this may be because there is a mixing of tabs and spaces? I will be attempting to resolve it under that understanding next and update here if that is the case.

  • Hyperas in Keras (Older Version)

    Hyperas in Keras (Older Version)

    I am using an older version of keras - last cloned around January/February. I am trying to integrate hyperas with keras, and get an error as follows when building my keras model with dropout.

    model.add(Dropout({{uniform(0, 1)}}))

    File "/Library/Python/2.7/site-packages/hyperopt/pyll_utils.py", line 48, in hp_uniform raise TypeError('require string label') TypeError: require string label

    Can someone please help me as to why I get an error like this? I am simply initially following the examples as in the hyperas examples.

  • Avoid source manip with decorators

    Avoid source manip with decorators

    Opening this per @maxpumperla suggestion on twitter. A little pie in the sky suggestion, forgive me. I noticed the template mechanism and I went to look at the implementation, getsource and related. How robust is that manipulation? Can I pass a variable or a complex expression in between double curly brackets? Can't it be confused with a set of sets (which is not allowed in python, but still that's what a {{x}} tries to evaluate to. My point is that you don't really need a language extension, if I get it right. You need to call model.fit many times with different values for the parameters. This is also done in randomized testing and in python in particular in the package hypothesis. If you have a function f(x,y) you want to run on random inputs you just need to write

    from hypothesis import given
    from hypothesis.strategies import integers, booleans
    @given(x = integers(), y = booleans())
    def f(x,y):
    

    will make f into a 0 non-default argument function that can just be called as f() and will run f(x,y) multiple times with random inputs. So I thought why not apply the same approach to hyperas to write something like

    @optimize(dropout  = uniform(0,1), shape = choice([256, 512, 1024]))
    def opt(dropout, shape):
    return Sequential(
     ...
     Dropout(dropout = dropout))
    

    The decorator just samples the parameters from the distribution, creates and fits as many models as necessary and computes summary stats. It may be a little more verbose then the {{}} notation and I am all for brevity, but doesn't tamper with Python's evaluation mechanism. In the spirit of least surprise and least effort, why not use a decorator? I may be missing the fundamental reason to use {{}}, but then it's good to have it written down for posterity. Thanks

  • Have an issue on pure pycharm file

    Have an issue on pure pycharm file

    @maxpumperla I am still confused how I should implement my code in pure pycharm? it's my pure pycharm file:

    `from future import print_function from hyperopt import Trials, STATUS_OK, tpe from hyperas import optim from hyperas.distributions import choice, uniform

    from keras.models import Sequential from keras.layers.core import Dense, Dropout, Activation from keras.optimizers import RMSprop

    from keras.datasets import mnist from keras.utils import np_utils

    def data(): """ Data providing function:

    This function is separated from model() so that hyperopt
    won't reload data for each evaluation run.
    """
    (X_train, y_train), (X_test, y_test) = mnist.load_data()
    X_train = X_train.reshape(60000, 784)
    X_test = X_test.reshape(10000, 784)
    X_train = X_train.astype('float32')
    X_test = X_test.astype('float32')
    X_train /= 255
    X_test /= 255
    nb_classes = 10
    Y_train = np_utils.to_categorical(y_train, nb_classes)
    Y_test = np_utils.to_categorical(y_test, nb_classes)
    return X_train, Y_train, X_test, Y_test
    

    def model(X_train, Y_train, X_test, Y_test): """ Model providing function:

    Create Keras model with double curly brackets dropped-in as needed.
    Return value has to be a valid python dictionary with two customary keys:
        - loss: Specify a numeric evaluation metric to be minimized
        - status: Just use STATUS_OK and see hyperopt documentation if not feasible
    The last one is optional, though recommended, namely:
        - model: specify the model just created so that we can later use it again.
    """
    model = Sequential()
    model.add(Dense(512, input_shape=(784,)))
    model.add(Activation('relu'))
    model.add(Dropout({{uniform(0, 1)}}))
    model.add(Dense({{choice([256, 512, 1024])}}))
    model.add(Activation('relu'))
    model.add(Dropout({{uniform(0, 1)}}))
    model.add(Dense(10))
    model.add(Activation('softmax'))
    
    rms = RMSprop()
    model.compile(loss='categorical_crossentropy', optimizer=rms, metrics=['accuracy'])
    
    model.fit(X_train, Y_train,
              batch_size={{choice([64, 128])}},
              nb_epoch=1,
              verbose=2,
              validation_data=(X_test, Y_test))
    score, acc = model.evaluate(X_test, Y_test, verbose=0)
    print('Test accuracy:', acc)
    return {'loss': -acc, 'status': STATUS_OK, 'model': model}
    

    X_train, Y_train, X_test, Y_test = data()

    best_run, best_model = optim.minimize(model=model, data=data, algo=tpe.suggest, max_evals=5, trials=Trials(), notebook_name="simple_notebook") best_model.save("best_model.h5") print("Evalutation of best performing model:") print(best_model.evaluate(X_test, Y_test)) ` Should I crate any notebook file?

    1. I create and empty it. then run my code and get the error. "NameError: name 'mnist' is not defined"
    2. then I copy-paste my code into notebook, and run my code. again get the error "NameError: name 'mnist' is not defined".
    3. I run the notebook directly, everything is OK. then run pycharm file and it is OK as well.
    4. empty the notebook, and run pycharm code.. it is still surprisingly working...

    I would be grateful if you could let me know how to utilize hyperas in pure pycharm code...

  • The

    The "show_accuracy" argument is deprecated

    C:\Program Files\Anaconda2\lib\site-packages\keras\models.py:635: UserWarning: he "show_accuracy" argument is deprecated, instead you should pass the "accurac " metric to the model at compile time: model.compile(optimizer, loss, metrics=["accuracy"]) warnings.warn('The "show_accuracy" argument is deprecated, '

    In your example application:

    model.fit(X_train, Y_train,
                  batch_size={{choice([64, 128])}},
                  nb_epoch=1,
                  show_accuracy=True,
                  verbose=2,
                  validation_data=(X_test, Y_test))
    
  • module 'hyperopt.pyll' has no attribute 'base'

    module 'hyperopt.pyll' has no attribute 'base'

    Hi I'm trying to us Hyperas to optimise my Neural Network but I keep receiving this error message:
    module 'hyperopt.pyll' has no attribute 'base'

    It comes from that function:

    def memo_from_config(self, config):
            memo = {}
            for node in pyll.dfs(self.expr):
                if node.name == "hyperopt_param":
                    label = node.arg["label"].obj
                    # -- hack because it's not really garbagecollected
                    #    this does have the desired effect of crashing the
                    #    function if rec_eval actually needs a value that
                    #    the the optimization algorithm thought to be unnecessary
                    memo[node] = config.get(label, pyll.base.GarbageCollected)
            return memo
    

    Someone has an idea how I can fix it ? my script where I used Hyperas: https://gist.github.com/julesangebault/264359cc037dfe3ceee2d8b36b042869

  • global variable issue

    global variable issue

    Helo there, I have the following code. a global variable that num_gpu that indicates the number of gpus availavle. If I can data() or create_mode() separate everything is fine. But when I call optim.minimize and pass the function, it does not see the global variable and num_gpu becomes undfined. Am I doing something wrong? Thanks in advance. ...

    from keras import backend as K
    from livelossplot import PlotLossesKeras
    from hyperas import optim
    from hyperas.distributions import choice, uniform
    from hyperopt import Trials, STATUS_OK, tpe
    
    # Setup (multi) GPU usage with scalable VRAM
    num_gpu = setup_multi_gpus()
    
    
    def data():
        print(num_gpu)
    
        df, train = data_import(possible_columns)
        df = prepare_data(df)
    
        ...
        return X_train, y_train, X_test, y_test
    
    def create_model(X_train, y_train, X_test, y_test):
        print(num_gpu)
    
    
    
    if __name__ == '__main__':
        best_run, best_model = optim.minimize(model=create_model, 
                                              data=data,
                                              algo=tpe.suggest,
                                              max_evals=500,
                                              trials=Trials(),
                                              eval_space=True)
    
        X_train, y_train, X_test, y_test = data()
    
        create_model(X_train, y_train, X_test, y_test )
        print(num_gpu)
    
  • AttributeError: numpy.random.mtrand.RandomState object has no attribute 'integers'

    AttributeError: numpy.random.mtrand.RandomState object has no attribute 'integers'

    The above error indicates to me that the build on the supercomputer I am using to train. It is the same as the one here: https://github.com/maxpumperla/hyperas/issues/284 . I reset hyperas to 0.2.5 like advised in issue, but it hasn't resolved my problem.

    • [ o] Install latest hyperas from GitHub: pip install git+git://github.com/maxpumperla/hyperas.git

    • [ o] Install latest hyperopt from GitHub: pip install git+git://github.com/hyperopt/hyperopt.git

    I could not do the above two steps as I am using a supercomputer, and received error:

    Collecting git+git://github.com/maxpumperla/hyperas.git
      Cloning git://github.com/maxpumperla/hyperas.git to /tmp/pip-req-build-a0z4c6ur
      Running command git clone --quiet git://github.com/maxpumperla/hyperas.git /tmp/pip-req-build-a0z4c6ur
      fatal: remote error:
        The unauthenticated git protocol on port 9418 is no longer supported.
      Please see https://github.blog/2021-09-01-improving-git-protocol-security-github/ for more information.
    
  • Why hyperas is executing Raw Text as code

    Why hyperas is executing Raw Text as code

    I have encountered this issue where the hyper optimisation code tries to execute a block of raw text somewhere else in the notebook, raising a ridiculous Syntax Error. Sometimes it works sometimes it begins to execute markdown or raw text.

    Even when I delete the raw text block the error message still points to that raw text, 'Potential issue:'. What's going on?

    Traceback (most recent call last):
    
      File "/Users/anaconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2963, in run_code
        exec(code_obj, self.user_global_ns, self.user_ns)
    
      File "<ipython-input-13-7ea6376dc3f4>", line 38, in <module>
        eval_space= True, notebook_name= 'MotorIns')
    
      File "/Users/anaconda3/lib/python3.6/site-packages/hyperas/optim.py", line 69, in minimize
        keep_temp=keep_temp)
    
      File "/Users/anaconda3/lib/python3.6/site-packages/hyperas/optim.py", line 98, in base_minimizer
        model_str = get_hyperopt_model_string(model, data, functions, notebook_name, verbose, stack)
    
      File "/Users/anaconda3/lib/python3.6/site-packages/hyperas/optim.py", line 189, in get_hyperopt_model_string
        imports = extract_imports(cleaned_source, verbose)
    
      File "/Users/anaconda3/lib/python3.6/site-packages/hyperas/utils.py", line 40, in extract_imports
        tree = ast.parse(source)
    
      File "/Users/anaconda3/lib/python3.6/ast.py", line 35, in parse
        return compile(source, filename, mode, PyCF_ONLY_AST)
    
      File "<unknown>", line 209
        Potential issue:
                      ^
    SyntaxError: invalid syntax
    
    
    
  • Can Hyperas natively return data regarding all models ran during an optimization cycle?

    Can Hyperas natively return data regarding all models ran during an optimization cycle?

    For example, if I test dropout and l2_regularization through hyperas optimization, is there a way to return the set of data which relates the loss metrics of each NN configuration? Or would I have to hardcode that utility myself, likely using pandas?

Hyperparameter Optimization for TensorFlow, Keras and PyTorch
Hyperparameter Optimization for TensorFlow, Keras and PyTorch

Hyperparameter Optimization for Keras Talos • Key Features • Examples • Install • Support • Docs • Issues • License • Download Talos radically changes

Nov 27, 2022
A very tiny, very simple, and very secure file encryption tool.
A very tiny, very simple, and very secure file encryption tool.

Picocrypt is a very tiny (hence "Pico"), very simple, yet very secure file encryption tool. It uses the modern ChaCha20-Poly1305 cipher suite as well

Nov 29, 2022
Technical Indicators implemented in Python only using Numpy-Pandas as Magic - Very Very Fast! Very tiny! Stock Market Financial Technical Analysis Python library . Quant Trading automation or cryptocoin exchange
Technical Indicators implemented in Python only using Numpy-Pandas as Magic  - Very Very Fast! Very tiny!  Stock Market Financial Technical Analysis Python library .  Quant Trading automation or cryptocoin exchange

MyTT Technical Indicators implemented in Python only using Numpy-Pandas as Magic - Very Very Fast! to Stock Market Financial Technical Analysis Python

Dec 3, 2022
An optimization and data collection toolbox for convenient and fast prototyping of computationally expensive models.
An optimization and data collection toolbox for convenient and fast prototyping of computationally expensive models.

An optimization and data collection toolbox for convenient and fast prototyping of computationally expensive models. Hyperactive: is very easy to lear

Dec 1, 2022
optimization routines for hyperparameter tuning
optimization routines for hyperparameter tuning

Optunity is a library containing various optimizers for hyperparameter tuning. Hyperparameter tuning is a recurrent problem in many machine learning t

Nov 9, 2022
Distributed Asynchronous Hyperparameter Optimization in Python

Hyperopt: Distributed Hyperparameter Optimization Hyperopt is a Python library for serial and parallel optimization over awkward search spaces, which

Nov 30, 2022
Automated Hyperparameter Optimization Competition

QQ浏览器2021AI算法大赛 - 自动超参数优化竞赛 ACM CIKM 2021 AnalyticCup 在信息流推荐业务场景中普遍存在模型或策略效果依赖于“超参数”的问题,而“超参数"的设定往往依赖人工经验调参,不仅效率低下维护成本高,而且难以实现更优效果。因此,本次赛题以超参数优化为主题,从真

Dec 9, 2021
HyperaPy: An automatic hyperparameter optimization framework ⚡🚀
HyperaPy: An automatic hyperparameter optimization framework ⚡🚀

hyperpy HyperPy: An automatic hyperparameter optimization framework Description HyperPy: Library for automatic hyperparameter optimization. Build on t

Sep 6, 2022
A Lightweight Hyperparameter Optimization Tool 🚀
A Lightweight Hyperparameter Optimization Tool 🚀

Lightweight Hyperparameter Optimization ?? The mle-hyperopt package provides a simple and intuitive API for hyperparameter optimization of your Machin

Dec 2, 2022
2021-AIAC-QQ-Browser-Hyperparameter-Optimization-Rank6

2021-AIAC-QQ-Browser-Hyperparameter-Optimization-Rank6

Mar 31, 2022
A very simple tool for situations where optimization with onnx-simplifier would exceed the Protocol Buffers upper file size limit of 2GB, or simply to separate onnx files to any size you want.
A very simple tool for situations where optimization with onnx-simplifier would exceed the Protocol Buffers upper file size limit of 2GB, or simply to separate onnx files to any size you want.

sne4onnx A very simple tool for situations where optimization with onnx-simplifier would exceed the Protocol Buffers upper file size limit of 2GB, or

Aug 30, 2022
This is an implementation of Googles Yogi-Optimizer in Keras (tf.keras)

Yogi-Optimizer_Keras This is an implementation of Googles Yogi-Optimizer in Keras (tf.keras) The NeurIPS-Paper can be found here: http://papers.nips.c

Sep 13, 2022
Keras udrl - Keras implementation of Upside Down Reinforcement Learning

keras_udrl Keras implementation of Upside Down Reinforcement Learning This is me

Jan 24, 2022
Example-custom-ml-block-keras - Custom Keras ML block example for Edge Impulse

Custom Keras ML block example for Edge Impulse This repository is an example on

Nov 2, 2022
Classification models 1D Zoo - Keras and TF.Keras

Classification models 1D Zoo - Keras and TF.Keras This repository contains 1D variants of popular CNN models for classification like ResNets, DenseNet

Nov 23, 2022
Implementation of trRosetta and trDesign for Pytorch, made into a convenient package
Implementation of trRosetta and trDesign for Pytorch, made into a convenient package

trRosetta - Pytorch (wip) Implementation of trRosetta and trDesign for Pytorch, made into a convenient package

Nov 16, 2022
Convenient tool for speeding up the intern/officer review process.

icpc-app-screen Convenient tool for speeding up the intern/officer applicant review process. Eliminates the pain from reading application responses of

Oct 30, 2021
Bib-parser - Convenient script to parse .bib files with the ACM Digital Library like metadata

Bib Parser Convenient script to parse .bib files with the ACM Digital Library li

Jan 26, 2022