Evaluating Deep Speech 2 on Mac OSX

[karllab41]

Hello! Thanks for posting this. I'm excited to run speech recognition on files! I've been trying to use Deep Speech 2 for evaluating my denoising pipeline. However, I'm having some trouble with the installation, and most of it is from aeon and the data loading.

When I run:

python evaluate.py --manifest val:$TOPDIR/librispeech/test-clean/test-manifest.csv --model_file $TOPDIR/model/librispeech_16_epochs.prm

I get:

Traceback (most recent call last):
  File "evaluate.py", line 21, in <module>
    from aeon.dataloader import DataLoader
ModuleNotFoundError: No module named 'aeon.dataloader'

---

Here's how I did my installation. I installed Neon from scratch via the original github page, which I assumed installed aeon. It did, but dataloader apparently was not in that installation. So, I went to the aeon page. The instructions told me to install aeon via:

git clone https://github.com/NervanaSystems/private-aeon.git aeon

That seemed incorrect (since private-aeon.git seems to no longer be private). So, I just installed

git clone https://github.com/NervanaSystems/aeon.git aeon

I ran into some C++ problems, so I followed Aeon Issue 48, which installed it. However, even after I installed aeon, I still couldn't import aeon.datasetloader.

import aeon.datasetloader
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-2f90034cee08> in <module>()
----> 1 import aeon.datasetloader

ImportError: No module named datasetloader

[karllab41]

Update: I reloaded the aeon package with release version 0.2.7: https://github.com/NervanaSystems/aeon/releases/tag/v0.2.7, which I downloaded from here, and it has datasetloader. It still appears that aeon is the problem because after I type in:

python evalrun.py --manifest val:$TOPDIR/librispeech/test-clean/test-manifest.csv --model_file $TOPDIR/model/librispeech_16_epochs.prm

(The paths are correct; I checked.) The error message looks like:

DISPLAY:neon:mklEngine.so not found; falling back to cpu backend
DISPLAY:neon:mklEngine.so not found; falling back to cpu backend
2017-09-22 19:42:54,373 - neon.backends.nervanacpu - WARNING - Problems inferring BLAS info, CPU performance may be suboptimal
2017-09-22 19:42:54,374 - neon.backends - WARNING - deterministic_update and deterministic args are deprecated in favor of specifying random seed
2017-09-22 19:42:54,379 - neon.backends.nervanacpu - WARNING - Problems inferring BLAS info, CPU performance may be suboptimal
Loading model file: /Users/l41admin/Magnolia/deepspeech/model/librispeech_16_epochs.prm
formats: formats: formats: can't open input file `': No such file or directoryformats: can't open input file `': No such file or directorycan't open input file `': No such file or directory
Unable to readdecode_thread_pool exception: number of frames is negative
can't open input file `': No such file or directory


Unable to readUnable to readUnable to readdecode_thread_pool exception: number of frames is negative
decode_thread_pool exception: number of frames is negative
decode_thread_pool exception: number of frames is negative

The data that I'm using comes from running:

python data/ingest_librispeech.py $TOPDIR/librispeech/test-clean $TOPDIR/librispeech/test-clean/transcripts_dir $TOPDIR/librispeech/test-clean/test-manifest.csv

I don't know what I could be doing wrong?

[wei-v-wang]

@karllab41 You are doing everything right. PR #57 is addressing the initial problem.
It rooted from aeon getting bumped to aeon 1.0.

[karllab41]

Thanks, @wei-v-wang.

While on the Mac, I haven't gotten evaluation on Librispeech to work (with either aeon-0.2.7 or 1.0), Linux has been okay with aeon-0.2.7 (release), neon 2.1, python 2.7 (though I'd like to try 3...)

Not sure if that helps, but just FYI

[SkyKingCoversGroundTiger]

report: not working on Mac for both train and evaluation (with aeon 1.0).
will test with Linux later :(

[SkyKingCoversGroundTiger]

report update: tested on Linux (Ubuntu 1604) with aeon-0.2.7 (release), neon 2.2.0, python 2.7. So far so good.

[wei-v-wang]

Thanks @yangroupaomo neon 2.2.0 should be automatically install aeon 1.0. In neon's virtual environment (. .venv/bin/activate), and do "pip list |grep aeon", it should be aeon-1.0, right?
Please feel free to let us know of issues.

@karllab41 We released neon 2.2.0 which featured our first improvement of DS2 on IA (more improvement to come in future releases). Feel free to try the latest neon as well (git checkout latest) from neon directory. :)

[SkyKingCoversGroundTiger]

Yes, it is *nervana-aeon (1.0.0).* Will keep working on the evaluation part tonight and hopefully it will work. BTW, is there any workaround for the Mac Issue?

…

On Thu, Sep 28, 2017 at 11:18 PM, Wei Wang ***@***.***> wrote: Thanks @yangroupaomo <https://github.com/yangroupaomo> neon 2.2.0 should be automatically install aeon 1.0. In neon's virtual environment (. .venv/bin/activate), and do "pip list |grep aeon", it should be aeon-1.0, right? Please feel free to let us know of issues. @karllab41 <https://github.com/karllab41> We released neon 2.2.0 which featured our first improvement of DS2 on IA (more improvement to come in future releases). Feel free to try the latest neon as well (git checkout latest) from neon directory. :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#56 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AWBRGTrz1iw7LwaisBnAYBERjJQV4eLbks5snIvIgaJpZM4PhGCo> .

[saikishor]

I am getting many errors while installing aeon and try to check for nervana-aeon;
after the step : running setup.py install for nervana-aeon .../

......
In file included from /home/saikishor/Deepspeech/neon/aeon/src/block_loader_source.hpp:22:

In file included from /home/saikishor/Deepspeech/neon/aeon/src/buffer_batch.hpp:23:

In file included from /usr/include/opencv2/core/core.hpp:58:

/usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/cstddef:51:11: error: no member named 'max_align_t' in the global namespace

using ::max_align_t;

    ~~^

In file included from /home/saikishor/Deepspeech/neon/aeon/src/box.cpp:16:

In file included from /home/saikishor/Deepspeech/neon/aeon/src/box.hpp:19:

In file included from /usr/include/opencv2/core/core.hpp:58:

/usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/cstddef:51:11: error: no member named 'max_align_t' in the global namespace

using ::max_align_t;

    ~~^

In file included from /home/saikishor/Deepspeech/neon/aeon/src/block_manager.cpp:18:

In file included from /home/saikishor/Deepspeech/neon/aeon/src/block_manager.hpp:21:

In file included from /home/saikishor/Deepspeech/neon/aeon/src/buffer_batch.hpp:23:

In file included from /usr/include/opencv2/core/core.hpp:58:

/usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/cstddef:51:11: error: no member named 'max_align_t' in the global namespace

using ::max_align_t;

    ~~^

1 error generated.

clang++ -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -fstack-protector -I/usr/include/opencv -I/usr/include/python2.7 -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c /home/saikishor/Deepspeech/neon/aeon/src/boundingbox.cpp -o build/temp.linux-x86_64-2.7/home/saikishor/Deepspeech/neon/aeon/src/boundingbox.o -O3 -std=c++11 -Werror=return-type -Werror=inconsistent-missing-override -Weverything -Wno-c++98-compat -Wno-c++98-compat-pedantic -Wno-padded -Wno-weak-vtables -Wno-global-constructors -Wno-switch-enum -Wno-gnu-zero-variadic-macro-arguments -Wno-undef -Wno-exit-time-destructors -Wno-missing-prototypes -Wno-disabled-macro-expansion -Wno-pedantic -Wno-documentation -Wno-covered-switch-default -Wno-old-style-cast -Wno-unknown-warning-option -Wno-sign-compare -Wno-unused-parameter -Wno-conversion -Wno-float-equal -Wno-duplicate-enum -Wno-used-but-marked-unused -Wno-c++11-compat-deprecated-writable-strings -Wno-deprecated -Wno-double-promotion -DPYTHON_FOUND

error: command 'clang++' failed with exit status 1

1 error generated.

1 error generated.

1 error generated.

1 error generated.

In file included from /home/saikishor/Deepspeech/neon/aeon/src/boundingbox.cpp:17:

In file included from /home/saikishor/Deepspeech/neon/aeon/src/etl_boundingbox.hpp:21:

In file included from /home/saikishor/Deepspeech/neon/aeon/src/interface.hpp:27:

In file included from /home/saikishor/Deepspeech/neon/aeon/src/typemap.hpp:19:

In file included from /usr/include/opencv2/core/core.hpp:58:

/usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/cstddef:51:11: error: no member named 'max_align_t' in the global namespace

using ::max_align_t;

    ~~^

1 error generated.

1 error generated.

1 error generated.

1 error generated.

---

Cleaning up...
Command /usr/bin/python -c "import setuptools, tokenize;file='/tmp/pip-ybSgIX-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-lUO1rj-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip-ybSgIX-build
Storing debug log for failure in /home/saikishor/.pip/pip.log

[wei-v-wang]

It was not quite eye-catching but "http://neon.nervanasys.com/index.html/installation.html" has a suggestion when encountering aeon related issues.
"If you have encountered error messages about failing to install aeon while building neon, please visit aeon page for how to install prerequisites for aeon to enable neon with aeon data loader."

To be more clear, the above seems to be related to clang and could you try following aeon readme to install all pre-requisites? aeon is part of neon but neon does not list (automatically install) all aeon pre-requisites.

https://github.com/NervanaSystems/aeon/blob/master/README.md

[saikishor]

Yes the problem is definitely with clang, but it's very hard on resolve. I am using Ubuntu so the prerequisites installation is only:

apt-get install git clang cmake python-dev python-pip libcurl4-openssl-dev libopencv-dev libsox-dev

Followed by normal installation "cmake" of aeon.

Do you have any idea how to solve it?.
The clang is using gcc++ 4.9

[saikishor]

The version of the clang is 3.4 and it's using gcc and gcc++ of 4.9 as default.

Is there any possible way to resolve it?

[wei-v-wang]

any possibility of upgrading clang from 3.4 to 3.5? Google search on "max_align_t" had suggestions along these lines.

[saikishor]

I tried installing clang to 3.6 after removing 3.4 and tried to run the commands, but ended up with the same at the end. I tried about max_align_t on Google, but there are many solutions with clang, but they only propose to run for individual files and they didn't explain well how to do for a Cmake file.

[wei-v-wang]

FYI: NervanaSystems/neon#375 had some suggestion with using libc++

Can you try the fix in the above 375? That was a Mac system in 375 fix.