So you’ve finished training your model, and it’s time to get some insights as towhat it has learned. You decide which tensor should be interesting, and go lookfor it in your code — to find out what its name is. Then it hits you — youforgot to give it a name. You also forgot to wrap the logical code block with anamed scope. It means you’ll have a hard time getting a reference to the tensor.It holds for python scripts as well as TensorBoard:
Can you see that small red circle lost in the sea of tensors? Finding it ishard...
That’s a bummer! It would have been much better if it looked more like this:
That’s more like it! Each set of tensors which form a logical unit is wrappedinside a named scope.
Why can’t the graph be automatically constructed in a way that resembles yourcode? I mean, most chances are you didn’t construct the model using a singlefunction, did you? Your code base contains multiple functions — each forms alogical unit which deserves its own named scope!
Let’s say you have a tensor x which was defined by the function f, which inturn was called by g. It means that while you were writing the code, you hadthis logical structure in mind: g -> f -> x. Wouldn’t it be great if themodel would automatically be constructed in a way that the name of the tensorwould be g/f/x?
Come to think of it, it’s pretty simple to do. All you have to do is go over allyour functions and add a single line of code:
def f(): with tensorflow.name_scope(‘f’): # define tensorsSo what’s wrong with that approach?
- The name of the function
f appears twice — both in the function declarationand as an argument to tensorflow.name_scope. Maybe next week you’ll change thename of the function to something more meaningful, let’s say foo.Unfortunately, you might forget to update the name of the scope!You have to apply indentation to the entire body of f. While it’s not thatbad, personally I don’t like having high indentation levels. Let’s say fcontains a for loop which contains an if statement, which contains another forloop. Thanks to calling to tensorflow.name_scope, we’re already at anindentation level of 4!We can bypass these disadvantages using simple metaprogramming — Python’sdecorators to the rescue!
import redef name_scope(f): def func(*args, **kwargs): name = f.__name__[re.search(r’[^_]’, f.__name__).start():] with tensorflow.name_scope(name): return f(*args, **kwargs) return func@name_scopedef foo(): # define tensorsHow does it work? The @ is a syntactic sugar. It’s equivalent to thefollowing:
def foo(): # define tensorsfoo = name_scope(foo)name_scope gets a function as an argument (f) and returns a new function(func). func creates a named scope, and then calls f.
The result? All the tensors that are defined by f will be created inside anamed scope. The name of the scope will be the name of the original function(“foo”) — thanks to f.__name__.
One small problem is that while function names might start with “_”, tensorflowscope names can’t. This is why we have to use re.
Why is it that important?
The challenge of writing clean tensorflow code is negligible compared to theresearch challenge of actually making the model any good.
Thus, it’s easy to be tempted to just focus on the research aspects of your job.However, in the long run, it’s important not to neglect the maintainability andreadability of your code, including those of your graph.
The decorator approach make my job a little easier, and I hope you’ll benefitfrom it too. Do you have other tips you’d like to share? Drop a line in thecomments!
Originally published by me atengineering.taboola.com.
