Mgiyuk

HDF5 reading and fit_generator multiprocessing error I'm trying to multiprocess the fit_generator. These are the problems that I face. trainable_model.fit_generator(load_random_cached_bottlenecks(BATCH_SIZE, label_map, training_addr_label_map, train_npy_dir, 'h5py', h5py_file_train),epochs = EPOCHS, steps_per_epoch=iterations_per_epoch_t, validation_data = load_random_cached_bottlenecks(BATCH_SIZE, label_map, validation_addr_label_map, val_npy_dir, 'h5py', h5py_file_val), validation_steps=iterations_per_epoch_v, workers = 1, callbacks = callback_list, use_multiprocessing = True, max_queue_size = 32) The main arguments that are causing problem: workers and use_multiprocessing . workers use_multiprocessing When worker=1 , use_multiprocessing=True/False runs with no problem. worker=1 use_multiprocessing=True/False If workers=5 , use_multiprocessing=True its throwing errors. The weird thing is its running, but at some random iteration I'm getting errors like work...

Plateauing loss in neural style transfer I am writing an implementation of style transfer by loading a vgg model from keras and supplying it to a tensorflow model. I am using an adam optimizer. The loss function is reducing but it is very slow and plateaus off at about 10 8 . Also the style loss is huge (order of 10 8 ) whereas content loss is much smaller(order of 10 5 ). This is weird as the paper for style transfer says to scale content loss down by a factor of 100 or 1000 when calculating total loss. I tried increasing the learning rate but that only makes the gradient overshoot. I suspect there must be a bug in my implementation but despite searching endlessly I have been unable to find what's wrong. Here's the code: # coding: utf-8 # In[1]: from keras.applications.vgg16 import VGG16 from keras.models import Model import tensorflow as tf import tensorflow.contrib.eager as tfe import numpy as np import matplotlib.pyplot as plt # In[2]: content_image_path = './skyline.jp...

Restoring checkpoint in distributed tensorflow Using a setup similar to https://github.com/tensorflow/models/tree/master/inception, the chief worker automatically saves a checkpoint file periodically on the node this process is running on. I'm running two ps on two different nodes. Two workers are also running on the two nodes each, with one out of 4 workers being the chief. When restarting training without any modification, the Supervisor automatically tries to restore the last checkpoint file, but ends up giving an error that it could not find the ckpt on the second node (the node other than the chief worker), because the chief never saved the ckpt on the second node. W tensorflow/core/framework/op_kernel.cc:936] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for /home/muneebs/tf_train/model.ckpt-275 If I copy the ckpt directory to the second node, it restores fine. Is it a bug? Should the saver be initialized as sharded=True? If so, is t...