mac os wheel broken for 2.2.0rc2

I'm getting an error running tokenize with tensorflow-text==2.2.0rc2 that I can only reproduce on macs.
(same error on rc1, and possibly earlier versions)

Steps to reproduce:

Setup:

python3 -m venv .test_venv 
source .test_venv/bin/activate
pip install --upgrade pip
pip install tensorflow==2.2.0rc3
pip install tensorflow-text==2.2.0rc2

Download vocab.txt into the dir you plan to run the test:
aws s3 cp s3://models.huggingface.co/bert/bert-base-uncased-vocab.txt ./vocab.txt
And then run these 5 lines in python

import tensorflow as tf
from tensorflow_text import BertTokenizer
tokenizer = BertTokenizer('./vocab.txt')
test2 = tf.convert_to_tensor(
    'Hello', dtype=tf.string
)
tokenizer.tokenize(test2)

Works on linux, (returns <tf.RaggedTensor [[[100]]]>)
On Mac, it throws an error. I've run on two separate macs (one with all totally fresh installs)

2020-04-16 13:18:07.892934: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at wordpiece_kernel.cc:204 : Invalid argument: Trying to access resource using the wrong type. Expected N10tensorflow6lookup15LookupInterfaceE got N10tensorflow6lookup15LookupInterfaceE
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/sylvia/Desktop/workspace/.tfvenv/lib/python3.7/site-packages/tensorflow_text/python/ops/bert_tokenizer.py", line 222, in tokenize
    return self._wordpiece_tokenizer.tokenize(tokens)
  File "/Users/sylvia/Desktop/workspace/.tfvenv/lib/python3.7/site-packages/tensorflow_text/python/ops/wordpiece_tokenizer.py", line 100, in tokenize
    subword, _, _ = self.tokenize_with_offsets(input)
  File "/Users/sylvia/Desktop/workspace/.tfvenv/lib/python3.7/site-packages/tensorflow_text/python/ops/wordpiece_tokenizer.py", line 156, in tokenize_with_offsets
    tokens.flat_values)
  File "/Users/sylvia/Desktop/workspace/.tfvenv/lib/python3.7/site-packages/tensorflow_text/python/ops/wordpiece_tokenizer.py", line 182, in tokenize_with_offsets
    **kwargs))
  File "<string>", line 141, in wordpiece_tokenize_with_offsets
  File "/Users/sylvia/Desktop/workspace/.tfvenv/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: Trying to access resource using the wrong type. Expected N10tensorflow6lookup15LookupInterfaceE got N10tensorflow6lookup15LookupInterfaceE [Op:WordpieceTokenizeWithOffsets]

running on python 3.7.6

I'm getting the exact same error in Python 3.6.8 - any ideas how to work around? Here's some more reproducible code:

import tensorflow as tf
import tensorflow_text as tf_text

with open("./vocab.txt", "r") as v:
    vocab = v.read().split("\n")

init = tf.lookup.KeyValueTensorInitializer(
    vocab,
    tf.range(tf.size(vocab, out_type=tf.int64), dtype=tf.int64),
    key_dtype=tf.string,
    value_dtype=tf.int64)

table = tf.lookup.StaticVocabularyTable(init, 1, lookup_key_dtype=tf.string)
tokenizer = tf_text.BertTokenizer(
    table,
    lower_case=False,
    normalization_form=None)

tokenizer.tokenize(
    tf.convert_to_tensor(["this is a rad sentence", "i wish i could tokenize it"], dtype=tf.string)
)

Getting:

Trying to access resource using the wrong type. Expected N10tensorflow6lookup15LookupInterfaceE got N10tensorflow6lookup15LookupInterfaceE
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/anaconda3/envs/ml/lib/python3.6/site-packages/tensorflow_text/python/ops/bert_tokenizer.py", line 222, in tokenize
    return self._wordpiece_tokenizer.tokenize(tokens)
  File "/anaconda3/envs/ml/lib/python3.6/site-packages/tensorflow_text/python/ops/wordpiece_tokenizer.py", line 100, in tokenize
    subword, _, _ = self.tokenize_with_offsets(input)
  File "/anaconda3/envs/ml/lib/python3.6/site-packages/tensorflow_text/python/ops/wordpiece_tokenizer.py", line 156, in tokenize_with_offsets
    tokens.flat_values)
  File "/anaconda3/envs/ml/lib/python3.6/site-packages/tensorflow_text/python/ops/wordpiece_tokenizer.py", line 182, in tokenize_with_offsets
    **kwargs))
  File "<string>", line 141, in wordpiece_tokenize_with_offsets
  File "/anaconda3/envs/ml/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: Trying to access resource using the wrong type. Expected N10tensorflow6lookup15LookupInterfaceE got N10tensorflow6lookup15LookupInterfaceE [Op:WordpieceTokenizeWithOffsets]

The same exact code works on my EC2 instance, the only difference being the python version is 3.6.10:

<tf.RaggedTensor [[[2023], [2003], [1037], [10958, 2094], [6251]], [[1045], [4299], [1045], [2071], [19204, 4697], [2009]]]>

@tgsmith61591

@tgsmith61591 Pretty sure you'll need to run on

tensorflow==2.1.0
tensorflow-text==2.1.1

until they fix the mac os wheel. You can also run on a non mac machine (EC2 works cause it's Linux)

Yeah. Disappointing since the custom code I've written depends on the train_step code present in the tf.keras.Model class, which means I'm stuck on 2.2.0. Hopefully this is fixed soon

I haven't tried tensorflow==2.2.0, tensorflow-text==2.1.1 - there's a pretty good chance that will work

This still appears to be busted in the 2.2.0 release

I have the same issue.

@thuang513

Any word on this @thuang513 ? Any chance we can expect a patch soon?

@broken

@broken can you take a look?

This actually looks like a problem with the validation of LookupResource in core TF. I see a similar bug opened up on their end. tensorflow/tensorflow#39524
I'll see what I can dig up.

@broken

I have this issue when I am using 1.15.3.
I suspect that it is due to abi compatible issues.
FYI @broken , I compiled tensorflow using MacMini of clang version 11.0.0 and our customised op using Apple clang version 11.0.3. Then this issue happened.

Confirmed this happens on my mac with tensorflow 2.2.rc3 running python 3.6.8 on Catalina.

The problem is that TypeIndex::Make is returning different hashes on MacOS for the same type when called from within different shared objects. The likely cause was commit 15275d3a14c77e2244ae1155f93243256f08e3ed which was added in tf 2.2.0.

I'm working with TF to test changes that would prevent these differing hashes from being generated to address this problem.

On a related note, we're looking at adding additional TF.Text CI testing against nightly TF to catch these before they reach release.

Getting the same error with tensorflow==2.2.0, tensorflow-text==2.2.1 and python 3.7.6 on
macOS Catalina 10.15.

We've been continuing to look at this. It is currently a blocker for TF 2.3, so it will be fixed soon.

FYI; the fix has been submitted.
tensorflow/tensorflow@1823f87#diff-991a6b786e16708ba1e6f5c9926cf151

This should now be fixed with the release of 2.3.0-rc1 which uses TF 2.3.0-rc*.

TF-text 2.4.2 Windows seems to have same issue. Need to consider wheel broken.

W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at wordpiece_kernel.cc:204 : Invalid argument: Trying to access resource using the wrong type. Expected class tensorflow::lookup::LookupInterface got class tensorflow::lookup::LookupInterface
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument:  Trying to access resource using the wrong type. Expected class tensorflow::lookup::LookupInterface got class tensorflow::lookup::LookupInterface
	 [[{{node StatefulPartitionedCall/StatefulPartitionedCall/bert_tokenizer/StatefulPartitionedCall/WordpieceTokenizeWithOffsets/WordpieceTokenizeWithOffsets/WordpieceTokenizeWithOffsets}}]]
  (1) Invalid argument:  Trying to access resource using the wrong type. Expected class tensorflow::lookup::LookupInterface got class tensorflow::lookup::LookupInterface
	 [[{{node StatefulPartitionedCall/StatefulPartitionedCall/bert_tokenizer/StatefulPartitionedCall/WordpieceTokenizeWithOffsets/WordpieceTokenizeWithOffsets/WordpieceTokenizeWithOffsets}}]]
	 [[StatefulPartitionedCall/StatefulPartitionedCall/bert_pack_inputs/PartitionedCall/map/RaggedToVariant/RaggedTensorToVariant/_31]]
0 successful operations.
0 derived errors ignored. [Op:__inference_restored_function_body_7285]

I will check Linux.

The same issue exist on Windows wheel 2.4.0.
Seems like only Linux version works now...

Navigation Menu

Repository navigation

mac os wheel broken for 2.2.0rc2 #272

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

mac os wheel broken for 2.2.0rc2 #272

Description

Activity

tgsmith61591 commented on Apr 23, 2020

sylviawhoa commented on Apr 23, 2020

tgsmith61591 commented on Apr 24, 2020

sylviawhoa commented on Apr 24, 2020

tgsmith61591 commented on May 14, 2020

johnmoberg commented on May 14, 2020

tgsmith61591 commented on May 21, 2020

thuang513 commented on May 25, 2020

broken commented on May 28, 2020

liyinhgqw commented on Jun 4, 2020

r-wheeler commented on Jun 4, 2020

broken commented on Jun 6, 2020

tabergma commented on Jun 9, 2020

broken commented on Jun 11, 2020

broken commented on Jun 22, 2020

broken commented on Jul 15, 2020

xidchen commented on Dec 23, 2020

T-Troll commented on Feb 8, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions