I think that this'll work

For mm to the nearest 64th, disregarding whole inches (ie, answer
always <=64)
=ROUND((A1/25.45-INT(A1/25.45))*64,0)

..to include entire length (ie, no limit)
=ROUND((MOD(A1/25.45,64))*64,0)

for in to mm
=ROUND(A1*25.45,0)