Given a binary 3D array, for each layer, cyclically rotate up each of its columns as many steps as indicated by the binary encoding of the columns of the layer above it, and then cyclically rotate left each of its rows as many steps as indicated by the binary encoding of the rows of the layer below it.

There will always be at least three layers. The top layer's columns and the bottom layer's rows should not be rotated.

Walk-through

Lets start with the small 4-layer, 2-row, 3-column array:

[[[1,0,1],

  [1,0,0]],



 [[1,0,1],

  [0,1,1]],



 [[0,1,1],

  [1,1,1]],



 [[1,1,0],

  [1,1,1]]]

The first step is evaluating the numbers encoded in binary by the columns and rows of each layer:

     3 0 2

5 [[[1,0,1],

4   [1,0,0]],



     2 1 3

5  [[1,0,1],

3   [0,1,1]],



     1 3 3

3  [[0,1,1],

7   [1,1,1]],



     3 3 1

6  [[1,1,0],

7   [1,1,1]]]

The first layer, [[1,0,1],[1,0,0]] will not have its columns rotated, but its rows will be cyclically rotated left 5 steps and 3 step respectively, thus becoming [[1,1,0],[1,0,0]].

 The second layer, [[1,0,1],[0,1,1]], will have its columns cyclically rotated up 3, 0, and 2 steps respectively, giving [[0,0,1],[1,1,1]], and then the rows are cyclically rotated left 3 and 7 steps respectively, with no visible change.

 The third layer, [[0,1,1],[1,1,1]] rotated up 2, 1, and 3 steps stays the same, and neither does rotating left 6 and 7 steps do anything.

 Finally, the fourth layer, [[1,1,0],[1,1,1]] rotated up 1, 3, and 3 steps is [[1,1,1],[1,1,0]], but its rows are not rotated afterwards, as it is the last layer.

 Putting all the layers together again, gives us the binary self-rotated 3D array:

[[[1,1,0],

  [1,0,0]],



 [[0,0,1],

  [1,1,1]],



 [[0,1,1],

  [1,1,1]],



 [[1,1,1],

  [1,1,0]]]

Example cases:

[[[1,0,1],[1,0,0]],[[1,0,1],[0,1,1]],[[0,1,1],[1,1,1]],[[1,1,0],[1,1,1]]] gives
[[[1,1,0],[1,0,0]],[[0,0,1],[1,1,1]],[[0,1,1],[1,1,1]],[[1,1,1],[1,1,0]]]

[[[1]],[[1]],[[0]]] gives
[[[1]],[[1]],[[0]]]

[[[1,0,1],[1,0,1],[1,0,1]],[[0,0,1],[0,0,1],[0,0,1]],[[1,0,0],[1,0,1],[0,0,1]]] gives
[[[0,1,1],[0,1,1],[0,1,1]],[[0,1,0],[1,0,0],[0,1,0]],[[1,0,1],[1,0,1],[0,0,0]]]

05AB1E, 41 39 bytes

εNĀiø¹N<èøJC‚øε`._}ø}N¹g<Êi¹N>èJC‚øε`._

This feels way too long.. Can definitely be golfed some more.

Try it online or verify all test cases.

Explanation:

ε                    # Map each layer in the (implicit) input to:

                     # (`N` is the layer-index of this map)

 NĀi                 #  If it is not the first layer:

    ø                #   Zip/transpose the current layer; swapping rows/columns

    ¹N<è             #   Get the `N-1`'th layer of the input

        ø            #   Zip/transpose; swapping rows/columns

         J           #   Join all inner lists (the columns) together

          C          #   And convert it from binary to integer

           ‚         #   Pair it with the current layer's columns we're mapping

            ø        #   Zip/transpose; to pair each integer with a layer's columns

             ε   }   #   Map over these pairs:

              `      #    Push both values of the pair separately to the stack

               ._    #    Rotate the column the integer amount of times

    ø                #   Zip/transpose the rows/columns of the current layer back

   }                 #  Close the if-statement

 N¹g<Êi              #  If this is not the last layer (layer-index-1 != amount_of_layers):

       ¹N>è          #   Get the `N+1`'th layer of the input

           J         #   Join all inner lists (the rows) together

            C        #   And convert it from binary to integer

             ‚       #   Pair it with the current layer's rows we're mapping

              ø      #   Zip/transpose; to pair each integer with a layer's rows

               ε     #   Map over these pairs:

                `    #    Push both values of the pair separately to the stack

                 ._  #    Rotate the row the integer amount of times

                     # (implicitly output the result after the layer-mapping is done)

Jelly,  18  17 bytes

ṙ""Ḅ}

Z€çŻṖ$$Z€çḊ

Try it online!

Code description will be added in five hours or so.

Python 2, 220 211 209 bytes

lambda m:[R(R(l,x),y)for l,x,y in z(m,[[0]]+[map(b,z(*l))for l in m],[map(b,l)for l in m[1:]]+[[0]])]

b=lambda r:int(''.join(map(str,r)),2)%len(r)

R=lambda l,x:[r[i:]+r[:i]for i,r in z(x*len(`l`),z(*l))]

z=zip

Try it online!