HDF5 reading and fit_generator multiprocessing error


HDF5 reading and fit_generator multiprocessing error



I'm trying to multiprocess the fit_generator.



These are the problems that I face.


trainable_model.fit_generator(load_random_cached_bottlenecks(BATCH_SIZE, label_map, training_addr_label_map, train_npy_dir, 'h5py', h5py_file_train),epochs = EPOCHS, steps_per_epoch=iterations_per_epoch_t, validation_data = load_random_cached_bottlenecks(BATCH_SIZE, label_map, validation_addr_label_map, val_npy_dir, 'h5py', h5py_file_val), validation_steps=iterations_per_epoch_v, workers = 1, callbacks = callback_list, use_multiprocessing = True, max_queue_size = 32)



The main arguments that are causing problem: workers and use_multiprocessing.


workers


use_multiprocessing



When worker=1, use_multiprocessing=True/False runs with no problem.


worker=1


use_multiprocessing=True/False



If workers=5, use_multiprocessing=True its throwing errors. The weird thing is its running, but at some random iteration I'm getting errors like


workers=5


use_multiprocessing=True


KeyError: 'Unable to open object (bad local heap signature)'



or


KeyError: 'Unable to open object (wrong B-tree signature)'



Im using h5py to read the files. I have written custom generator for this purpose.


def load_random_cached_bottlenecks(batch_size, label_map,
addr_label_map, dirs, comp_type = 'h5py', hdf5_file = None):
'''
Parameters
----------
batch_size: Number of bottlenecks to be loaded along with the labels
label_map: The dictionary that maps the class_names and the index
addr_label_map: The dictionary that maps addrs of bottlenecks and the labels
hdf5_file: This is the hdf5 file object with reading enabled.
Returns
-------
batch: (bottlenecks_train, bottlenecks_labels) a batch of them which is equal to batch_size
'''
while True:
chosen_h5py = np.random.choice(dirs, size = batch_size)
# chosen_h5py = [dirs[i] for i in batch_index]
labels_for_chosen_h5py = [label_map[addr_label_map[i]] for i in chosen_h5py]
h5py_data = np.array([hdf5_file[i] for i in chosen_h5py])
h5py_onehot = to_categorical(labels_for_chosen_h5py, num_classes = LABEL_LENGTH)
# print (h5py_data.shape)
yield (h5py_data, h5py_onehot)



I have referred here, but couldn't solve my problem.


Traceback (most recent call last):
File "/opt/anaconda3/lib/python3.6/site-packages/keras/utils/data_utils.py", line 677, in _data_generator_task
generator_output = next(self._generator)
File "general_model.py", line 263, in load_random_cached_bottlenecks
h5py_data = np.array([hdf5_file[i] for i in chosen_h5py])
File "/opt/anaconda3/lib/python3.6/site-packages/keras/utils/data_utils.py", line 677, in _data_generator_task
generator_output = next(self._generator)
File "general_model.py", line 263, in load_random_cached_bottlenecks
h5py_data = np.array([hdf5_file[i] for i in chosen_h5py])
File "general_model.py", line 263, in <listcomp>
h5py_data = np.array([hdf5_file[i] for i in chosen_h5py])
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "general_model.py", line 263, in <listcomp>
h5py_data = np.array([hdf5_file[i] for i in chosen_h5py])
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "/opt/anaconda3/lib/python3.6/site-packages/h5py/_hl/group.py", line 177, in __getitem__
oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5o.pyx", line 190, in h5py.h5o.open
File "/opt/anaconda3/lib/python3.6/site-packages/h5py/_hl/group.py", line 177, in __getitem__
oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
KeyError: 'Unable to open object (wrong B-tree signature)'
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5o.pyx", line 190, in h5py.h5o.open
KeyError: 'Unable to open object (bad symbol table node signature)'
Traceback (most recent call last):
File "general_model.py", line 437, in <module>
train_with_bottlenecks(args, label_map, trainable_model, non_trainable_model, iterations_per_epoch_t, iterations_per_epoch_v)
File "general_model.py", line 326, in train_with_bottlenecks
validation_steps=iterations_per_epoch_v, workers = 4, callbacks = callback_list, use_multiprocessing = True, max_queue_size = 32)
File "/opt/anaconda3/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/opt/anaconda3/lib/python3.6/site-packages/keras/engine/training.py", line 2194, in fit_generator
generator_output = next(output_generator)
File "/opt/anaconda3/lib/python3.6/site-packages/keras/utils/data_utils.py", line 793, in get
six.reraise(value.__class__, value, value.__traceback__)
File "/opt/anaconda3/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
KeyError: 'Unable to open object (wrong B-tree signature)'



Any help is appreciated! Thanks in Advance!









By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Comments

Post a Comment

Popular posts from this blog

paramiko-expect timeout is happening after executing the command

paramiko-expect timeout is happening after executing the command Can someone help me why timeout is happening after executing the command .I am trying to SSH to a machine and execute a command.I want to store the result of the executed command . I have referred paramiko-timeout but i didn't get my expected result. import traceback try: import paramiko except ImportError: raise Exception("Please install paramiko , pip install paramiko") import pexpect from paramiko_expect import SSHClientInteraction def main(): HOSTNAME = "minion5.net" USERNAME = "root" PASSWORD = "help@432" PROMPT = "root@*:~$s+" try: client = paramiko.SSHClient() client.load_system_host_keys() client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) client.connect(hostname=HOSTNAME, username=USERNAME, password=PASSWORD) with SSHClientInteraction(client, timeout=20, display=True) as interact: ...
Read more

Opening a url is failing in Swift

Opening a url is failing in Swift breakpointing inside the callback and logging success shows false. I added itms-apps to my plist: LSApplicationQueriesSchemes and put this in my app delegate: func application(_ app: UIApplication, open url: URL, options: [UIApplicationOpenURLOptionsKey : Any] = [:]) -> Bool { return true } func moreTapped() { let url = NSURL(string: "itms-apps://itunes.com/developer/quantum-productions/id979315877") if #available(iOS 10.0, *) { UIApplication.shared.open(url! as URL, options: [:], completionHandler: {(success: Bool) in }) } else { UIApplication.shared.openURL(url! as URL) } } Are you testing on a real device? – rmaddy Jun 21 at 23:57 I try on a device, get "Cannot Connect to App Store". Device is online ...
Read more

Export result set on Dbeaver to CSV

Export result set on Dbeaver to CSV Normally I use Dbeaver for windows and always export my result set like this: run my query --> select the result --> export the result set --> select export to clipboard --> done This step by step puts my result set in my clipboard and I can paste it wherever I want to work with it. The problem is that now I am using dbeaver for mac and this guide is not working. I can go on until the moment that I select my result set like in the image below: exporting data set But once I go further in the process, in the last step I get: no query Note that in "source" it was suppose to show the query that originated the result set, but instead it says just "select. As a result it does't select my result or anything (besides being "successful"). Normally my query would show up there automatically and I couldn't find any option that corrects this problem in the menus. Please help me! Thanks. Aft...
Read more